How angxiaoyue/reqflow works
Project type and purpose
• ReQFlow is a research‐grade, open‐source ML pipeline (Python) for protein backbone generation using rectified quaternion flows.
• It integrates data preprocessing (PDB/SCOPe), core flow models, training/inference scripts, and downstream evaluation (ProteinMPNN, ESMFold).
High‐level file‐structure analysis
• Top‐level directories:
• configs: all YAML experiment settings (data/model/inference/training)
• data: parsers, dataset builders, PDB & SCOPe preprocessing
• models: core network definitions (flow_model, feature nets)
• experiments: CLI scripts for train/inference (train_se3_flows.py, inference_se3_flows.py)
• ProteinMPNN: external code dependency for sequence design
• openfold: ESMFold code for evaluation
• analysis, motif_scaffolding: benchmarking & helper scripts
• Key files:
• setup.py / reqflow-env.yml: environment & packaging
• protein_mpnn_run.py: wrapper to call ProteinMPNN
• training/ & inference subfolders: separate pipelines
README insights
• Config‐driven: experiments use -cn <config_name> to launch pipelines.
• Two main workflows:
a) QFlow (unrectified) training & inference
b) ReQFlow (rectified) training & inference
• External evaluation via ProteinMPNN & ESMFold; results stored under inference_outputs.
Main components
A. Entry & Orchestration
• User CLI / Python scripts (experiments/train_se3_flows.py, experiments/inference_se3_flows.py)
• Config Manager (YAML loader)
B. Data Layer
• Raw data (PDB, SCOPe) → Parsers (data/parsers.py, openfold/data/mmcif_parsing.py)
• Transforms & Datasets (data/data_pipeline.py, data/process_*.py)
• DataLoader (data/protein_dataloader.py, openfold/data/data_modules.py)
C. Model Layer
• Core flow model (models/flow_model.py, models/flow_module.py)
• Feature networks (models/node_feature_net.py, models/edge_feature_net.py)
• Utilities (so3/quaternion math in data/so3_utils.py, models/utils.py)
D. Training Pipeline
• Experiment controller → instantiates DataLoader & Model → loss & optimizer setup in training/training.py → checkpointing (openfold/utils/checkpointing.py)
• Logging & metrics (training/plot_training_results.ipynb, WandB integration)
E. Inference Pipeline
• Experiment controller → loads model checkpoint → sampling loop (experiments/inference_se3_flows.py) → produces .pdb outputs + noise/sample pairs
F. Evaluation & Analysis
• ProteinMPNN integration (protein_mpnn_run.py, ProteinMPNN folder) → sequence design
• ESMFold via openfold (inference under self_consistency folder)
• Analysis scripts (analysis/metrics.py, analysis/all_metric_calculation.py)
G. External Dependencies
• ProteinMPNN (sequence design)
• openfold / ESMFold (structure evaluation)
• Deepspeed, PyTorch, torch‐scatter
Relationships & Data flows
• CLI → Config → Data Layer → Model Layer → Training / Inference → Outputs (checkpoints, PDB files)
• Inference Outputs → ProteinMPNN → ESMFold → Analysis → Metrics
• Model Weights & Configs persist in ckpts/ and ca_model_weights/
Architectural patterns & design principles
• Modular separation of concerns: data, model, experiment, analysis
• Config‐driven orchestration (Hydra‐like YAML)
• Plugin‐style integration with ProteinMPNN & openfold
• Reusability: same model code for QFlow & ReQFlow (rectified) via warm‐starts
Technologies & frameworks
• Python / PyTorch / Deepspeed
• YAML configs
• NumPy / custom quaternion math
• Jupyter notebooks for demos & plotting
• External tools: ProteinMPNN, ESMFold (via openfold)