# angxiaoyue/reqflow architecture

> Architecture diagram of the GitHub repository angxiaoyue/reqflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/angxiaoyue/reqflow
- Repository: https://github.com/angxiaoyue/reqflow

## Overview

##### Project type and purpose
- ReQFlow is a research‐grade, open‐source ML pipeline (Python) for protein backbone generation using rectified quaternion flows.
- It integrates data preprocessing (PDB/SCOPe), core flow models, training/inference scripts, and downstream evaluation (ProteinMPNN, ESMFold).

##### High‐level file‐structure analysis
- Top‐level directories:
- configs: all YAML experiment settings (data/model/inference/training)
- data: parsers, dataset builders, PDB & SCOPe preprocessing
- models: core network definitions (flow_model, feature nets)
- experiments: CLI scripts for train/inference (train_se3_flows.py, inference_se3_flows.py)
- ProteinMPNN: external code dependency for sequence design
- openfold: ESMFold code for evaluation
- analysis, motif_scaffolding: benchmarking & helper scripts
- Key files:
- setup.py / reqflow-env.yml: environment & packaging
- protein_mpnn_run.py: wrapper to call ProteinMPNN
- training/ & inference subfolders: separate pipelines

##### README insights
- Config‐driven: experiments use `-cn <config_name>` to launch pipelines.
- Two main workflows:
a) QFlow (unrectified) training & inference
b) ReQFlow (rectified) training & inference
- External evaluation via ProteinMPNN & ESMFold; results stored under `inference_outputs`.

##### Main components
A. Entry & Orchestration
- User CLI / Python scripts (`experiments/train_se3_flows.py`, `experiments/inference_se3_flows.py`)
- Config Manager (YAML loader)
B. Data Layer
- Raw data (PDB, SCOPe) → Parsers (`data/parsers.py`, `openfold/data/mmcif_parsing.py`)
- Transforms & Datasets (`data/data_pipeline.py`, `data/process_*.py`)
- DataLoader (`data/protein_dataloader.py`, `openfold/data/data_modules.py`)
C. Model Layer
- Core flow model (`models/flow_model.py`, `models/flow_module.py`)
- Feature networks (`models/node_feature_net.py`, `models/edge_feature_net.py`)
- Utilities (so3/quaternion math in `data/so3_utils.py`, `models/utils.py`)
D. Training Pipeline
- Experiment controller → instantiates DataLoader & Model → loss & optimizer setup in `training/training.py` → checkpointing (`openfold/utils/checkpointing.py`)
- Logging & metrics (`training/plot_training_results.ipynb`, WandB integration)
E. Inference Pipeline
- Experiment controller → loads model checkpoint → sampling loop (`experiments/inference_se3_flows.py`) → produces `.pdb` outputs + noise/sample pairs
F. Evaluation & Analysis
- ProteinMPNN integration (`protein_mpnn_run.py`, ProteinMPNN folder) → sequence design
- ESMFold via openfold (inference under self_consistency folder)
- Analysis scripts (`analysis/metrics.py`, `analysis/all_metric_calculation.py`)
G. External Dependencies
- ProteinMPNN (sequence design)
- openfold / ESMFold (structure evaluation)
- Deepspeed, PyTorch, torch‐scatter

##### Relationships & Data flows
- CLI → Config → Data Layer → Model Layer → Training / Inference → Outputs (checkpoints, PDB files)
- Inference Outputs → ProteinMPNN → ESMFold → Analysis → Metrics
- Model Weights & Configs persist in ckpts/ and ca_model_weights/

##### Architectural patterns & design principles
- Modular separation of concerns: data, model, experiment, analysis
- Config‐driven orchestration (Hydra‐like YAML)
- Plugin‐style integration with ProteinMPNN & openfold
- Reusability: same model code for QFlow & ReQFlow (rectified) via warm‐starts

##### Technologies & frameworks
- Python / PyTorch / Deepspeed
- YAML configs
- NumPy / custom quaternion math
- Jupyter notebooks for demos & plotting
- External tools: ProteinMPNN, ESMFold (via openfold)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Entry & Orchestration
    subgraph "Entry & Orchestration"
        CLI["User CLI Scripts"]:::orchestration
        ConfigMgr["Config Manager"]:::orchestration
    end

    %% Data Layer
    subgraph "Data Layer"
        Parsers["ReQFlow Parsers"]:::data
        MMParsing["mmCIF Parsing"]:::data
        Processing["PDB/SCOPe Processing"]:::data
        Datasets["Dataset Builders & Transforms"]:::data
        DataLoaders["DataLoaders"]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        FlowModel["Core Flow Models"]:::core
        FlowModule["Flow Module"]:::core
        NodeFeat["Node Feature Net"]:::core
        EdgeFeat["Edge Feature Net"]:::core
        SO3Utils["Quaternion & SO3 Utilities"]:::core
        ModelUtils["Model Utilities"]:::core
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        TrainController["Train Experiment Controller"]:::orchestration
        Checkpoint["Checkpointing"]:::external
        Logging["Logging & Metrics"]:::external
        ModelStore["Model Checkpoints"]:::storage
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        InfController["Inference Experiment Controller"]:::orchestration
        Sampling["Sampling & Output Generation"]:::orchestration
        OutDir["Inference Outputs (.pdb)"]:::storage
    end

    %% Evaluation & Analysis
    subgraph "Evaluation & Analysis"
        MPNNWrapper["ProteinMPNN Integration"]:::external
        MPNNWeights["MPNN Weights"]:::storage
        ESMFold["ESMFold via OpenFold"]:::external
        Analysis["Analysis Scripts"]:::external
    end

    %% External Dependencies
    subgraph "External Dependencies"
        ProtMPNN["ProteinMPNN Codebase"]:::external
        OpenFold["OpenFold (ESMFold)"]:::external
        Env["Env & Packaging"]:::external
    end

    %% Relationships
    CLI -->|launches| ConfigMgr
    ConfigMgr -->|loads configs| Parsers
    Parsers -->|parsed data| Processing
    Processing -->|processed tensors| Datasets
    Datasets -->|datasets| DataLoaders
    DataLoaders -->|batches| FlowModel
    FlowModel --> FlowModule
    FlowModel --> NodeFeat
    FlowModel --> EdgeFeat
    FlowModel --> SO3Utils
    FlowModel --> ModelUtils
    ConfigMgr -->|provides settings| TrainController
    TrainController -->|instantiates| DataLoaders
    TrainController -->|instantiates| FlowModel
    TrainController -->|uses| Checkpoint
    Checkpoint -->|stores| ModelStore
    TrainController -->|logs| Logging
    ModelStore -->|weights| InfController
    ConfigMgr -->|provides settings| InfController
    InfController -->|loads weights| Sampling
    Sampling -->|generates| OutDir
    OutDir -->|to| MPNNWrapper
    MPNNWrapper -->|designs sequences| MPNNWeights
    MPNNWeights -->|input to| ESMFold
    ESMFold -->|evaluation data| Analysis

    %% Click Events
    click CLI "https://github.com/angxiaoyue/reqflow/blob/master/experiments/train_se3_flows.py"
    click InfController "https://github.com/angxiaoyue/reqflow/blob/master/experiments/inference_se3_flows.py"
    click ConfigMgr "https://github.com/angxiaoyue/reqflow/tree/master/configs/"
    click Parsers "https://github.com/angxiaoyue/reqflow/blob/master/data/parsers.py"
    click Parsers "https://github.com/angxiaoyue/reqflow/blob/master/openfold/data/parsers.py"
    click MMParsing "https://github.com/angxiaoyue/reqflow/blob/master/openfold/data/mmcif_parsing.py"
    click Processing "https://github.com/angxiaoyue/reqflow/blob/master/data/process_pdb_files.py"
    click Processing "https://github.com/angxiaoyue/reqflow/blob/master/data/process_pdb_dataset.py"
    click Processing "https://github.com/angxiaoyue/reqflow/blob/master/data/rectify_process_pdb_files.py"
    click Datasets "https://github.com/angxiaoyue/reqflow/blob/master/data/datasets.py"
    click Datasets "https://github.com/angxiaoyue/reqflow/blob/master/data/rectify_datasets.py"
    click Datasets "https://github.com/angxiaoyue/reqflow/blob/master/openfold/data/data_pipeline.py"
    click DataLoaders "https://github.com/angxiaoyue/reqflow/blob/master/data/pdb_dataloader.py"
    click DataLoaders "https://github.com/angxiaoyue/reqflow/blob/master/data/rectify_protein_dataloader.py"
    click DataLoaders "https://github.com/angxiaoyue/reqflow/blob/master/openfold/data/data_modules.py"
    click FlowModel "https://github.com/angxiaoyue/reqflow/blob/master/models/flow_model.py"
    click FlowModule "https://github.com/angxiaoyue/reqflow/blob/master/models/flow_module.py"
    click NodeFeat "https://github.com/angxiaoyue/reqflow/blob/master/models/node_feature_net.py"
    click EdgeFeat "https://github.com/angxiaoyue/reqflow/blob/master/models/edge_feature_net.py"
    click SO3Utils "https://github.com/angxiaoyue/reqflow/blob/master/data/so3_utils.py"
    click ModelUtils "https://github.com/angxiaoyue/reqflow/blob/master/models/utils.py"
    click TrainController "https://github.com/angxiaoyue/reqflow/blob/master/experiments/train_se3_flows.py"
    click Checkpoint "https://github.com/angxiaoyue/reqflow/blob/master/openfold/utils/checkpointing.py"
    click Logging "https://github.com/angxiaoyue/reqflow/blob/master/analysis/metrics.py"
    click Logging "https://github.com/angxiaoyue/reqflow/blob/master/analysis/all_metric_calculation.py"
    click Logging "https://github.com/angxiaoyue/reqflow/blob/master/ProteinMPNN/training/plot_training_results.ipynb"
    click MPNNWrapper "https://github.com/angxiaoyue/reqflow/blob/master/ProteinMPNN/protein_mpnn_run.py"
    click MPNNWeights "https://github.com/angxiaoyue/reqflow/tree/master/ProteinMPNN/ca_model_weights/"
    click MPNNWeights "https://github.com/angxiaoyue/reqflow/tree/master/ProteinMPNN/vanilla_model_weights/"
    click Analysis "https://github.com/angxiaoyue/reqflow/blob/master/analysis/utils.py"
    click ProtMPNN "https://github.com/angxiaoyue/reqflow/tree/master/ProteinMPNN/"
    click OpenFold "https://github.com/angxiaoyue/reqflow/tree/master/openfold/"
    click Env "https://github.com/angxiaoyue/reqflow/blob/master/setup.py"
    click Env "https://github.com/angxiaoyue/reqflow/blob/master/reqflow-env.yml"

    %% Styles
    classDef core fill:#cfe2f3,stroke:#2b5797,color:#000;
    classDef data fill:#d9ead3,stroke:#6aa84f,color:#000;
    classDef external fill:#f9cb9c,stroke:#e69138,color:#000;
    classDef orchestration fill:#e6e6e6,stroke:#7f7f7f,color:#000;
    classDef storage fill:#fff2cc,stroke:#b45f06,color:#000;
```
