How harvardnlp/regulatory-prediction works
1. Project shape
- This is a single-purpose research training repo for genomic regulatory prediction.
- The whole system is centered around one executable entrypoint, train.py, with supporting model, utility, and TensorFlow helper modules.
2. Primary runtime
- The code is a Python 2.7 / TensorFlow 1.0-era project.
- NumPy and SciPy are used as supporting numerical libraries.
- There is no indication of a web service, database, or distributed system; the architecture is a local training pipeline.
3. Top-level flow
- train.py is the orchestration layer.
- It likely loads data, builds the model, runs optimization, and evaluates results.
- Any experiment configuration is probably embedded in the training script or passed via command-line flags.
4. Model subsystem
- models.py is the main model-definition boundary.
- Based on the paper title, it likely contains the neural network architectures for modeling long-range genomic dependencies, probably including dilated convolution variants and task-specific heads.
- This is where the core predictive logic lives.
5. TensorFlow integration
- tf_utils.py appears to hold reusable TensorFlow 1.x helpers: session setup, variable initialization, loss/metric helpers, checkpointing, or graph utilities.
- This layer separates training/model code from low-level TensorFlow boilerplate.
6. Custom operators / low-level kernels
- custom_ops.py suggests there are project-specific TensorFlow ops or wrappers around nonstandard operations.
- Architecturally, this is the “performance/primitive” layer beneath the model definitions, likely used when TensorFlow’s built-ins are insufficient or too slow.
7. General utilities
- util.py is the shared support layer for non-model logic.
- In a repo like this, that usually includes file handling, logging, metrics, sequence preprocessing, batching, and miscellaneous helpers used by train.py and models.py.
8. Data flow
- Input data is not included in the repo; the README says data is “forthcoming.”
- So the practical runtime flow is: external genomic dataset -> preprocessing/utilities -> TensorFlow input pipeline or in-memory batches -> model graph -> training/evaluation.
- Any data schema or dataset-specific parsing is likely implemented in the utility layer rather than as a standalone subsystem.
9. Training as the main control boundary
- There is no separate inference service or deployment layer visible.
- The repo is organized around offline experimentation: build a graph, train it, and report metrics from the same process.
10. Research-specific emphasis
- The architecture reflects a paper implementation rather than a product system.
- The important boundary is between “experiment driver” code and “model/ops” code, not between frontend/backend components.
11. Likely dependency boundaries
- Custom ops and model code depend on TensorFlow primitives.
- Training code depends on both model construction and utility functions.
- Utilities may also depend on NumPy/SciPy for numeric preprocessing and evaluation.
12. What to draw in an architecture diagram
- A good diagram would show:
- External genomic data feeding into train.py
- train.py calling into util.py, tf_utils.py, and models.py
- models.py optionally calling custom_ops.py
- TensorFlow as the execution/runtime layer underneath all of them
- The diagram should emphasize a single training pipeline with supporting libraries, not multiple services.
Main entry points: train.py train.py · models.py models.py · tf_utils.py tf_utils.py · custom_ops.py custom_ops.py
Components 8
Other components
- Genomic data · Training and evaluation inputs come from outside the repository.
- train.py · Main orchestration entrypoint for loading data, building the model, training, and evaluation.
train.py - models.py · Core neural architectures and task-specific prediction heads.
models.py - tf_utils.py · TensorFlow 1.x utilities for graph setup, training support, and reusable boilerplate.
tf_utils.py - custom_ops.py · Project-specific low-level operations or wrappers used by the model code.
custom_ops.py - util.py · General support code for preprocessing, batching, logging, and other shared helpers.
util.py - TensorFlow · Execution framework underlying the model, training, and custom operation layers.
- NumPy/SciPy · Supporting numerical libraries used by utilities and training-time computations.
Connections 11
- Genomic data → train.py: inputs · External genomic datasets flow into the training process.
- train.py → util.py: uses · Training relies on shared utilities for data handling and support logic.
- train.py → tf_utils.py: uses · Training uses TensorFlow helper routines and boilerplate support.
- train.py → models.py: builds · The driver constructs and runs the predictive model graph.
- models.py → custom_ops.py: depends on · Model definitions may call into project-specific low-level operations.
- models.py → TensorFlow: runs on · Model layers are implemented on top of TensorFlow primitives.
- tf_utils.py → TensorFlow: wraps · TensorFlow helper code manages graph/session-level behavior.
- custom_ops.py → TensorFlow: extends · Custom operations extend the TensorFlow execution layer.
- util.py → NumPy/SciPy: uses · Utility code may rely on NumPy and SciPy for preprocessing and metrics.
- train.py → TensorFlow: orchestrates · The training entrypoint coordinates execution within TensorFlow.
- train.py → NumPy/SciPy: uses · Training may use numerical libraries for data preparation and evaluation.