How psp3dcg/lightrosetta works
<explanation>
1. Identify the project type and purpose
- LightRoseTTA is an open‑source, PyTorch‑based protein structure prediction system.
- It provides both an inference pipeline (test_script.py, LightRoseTTA_test.py, data_pipeline.py) and a training pipeline (LightRoseTTA_training_code.zip plus generate/prepare scripts).
- Core functionality: given one or more FASTA sequences, generate multiple‑sequence alignment (MSA) features, build an atom‑level graph, run an SE(3)‑equivariant graph neural network, and output predicted 3D structures in PDB format.
2. Analyze the file structure
Top‑level:
• LightRoseTTA-env.yml — Conda environment spec
• download_datasets.py — fetch large biological databases
• data_pipeline.py, test_script.py, LightRoseTTA_test.py — orchestrate inference
• LightRoseTTA_preprocess_train_data.zip, LightRoseTTA_training_code.zip — separate training resources
• README.md — installation, usage, training/testing instructions
• weights/ — pretrained model files
Key packages:
• msa_feat/ — MSA feature generation (calls BLAST, CS-BLAST, PSIPRED, runs shells)
• atom_graph/ — build amino‑acid graph representations (node labels, bond types)
• model/ — defines LightRoseTTA network modules (embeddings, attention_module, SE3_network, transformer, refine_net, distance_predictor, etc.)
• utils/ & confs/ — shared utilities (parsers, kinematics, PDB generation, logging, profiling)
3. Examine the README for additional insights
- No explicit diagram, but describes a two‑stage flow: data download → MSA generation → graph building → model inference/training.
- Highlights external dependencies (Uniref30, BFD, pdb100 datasets; blast, csblast, psipred).
- Testing and training use separate scripts and data archives.
4. How to create the system design diagram
a. Main components (boxes):
1. External Data Sources
• Uniref30, BFD, pdb100
• Example FASTA inputs (example/Orphan25_fasta)
2. Data Preparation Layer
• download_datasets.py → raw data folders
• msa_feat module (generate_msa.py, parsers, make_msa.sh, run_generate_feat.sh)
• atom_graph module (generate_atom_data.py, generate_protein_graph.py)
3. Model Layer
• build_graph.py → constructs graph from atom_graph outputs
• embeddings.py → initial feature embeddings
• attention_module.py & transformer.py → pairwise attention over residues
• SE3_network.py & sym_cnn.py → equivariant convolutional layers
• distance_predictor.py → inter‑residue distance prediction
• refine_net.py → structure refinement
• tr_fold.py & init_str_generator.py → folding and initial structure generation
4. Orchestration & Scripts
• data_pipeline.py / test_script.py → end‑to‑end inference
• LightRoseTTA_train.py (in training_code) → training orchestration
5. Utilities & Configuration
• confs/util.py, utils/… → logging, profiling, PDB output, kinematics
6. Outputs
• Predicted PDB files (data_write_path/predict_pdb)
7. Pretrained Weights
• weights/LightRoseTTA.pth, LightRoseTTA‑Ab.pth
b. Relationships & interactions (arrows):
• FASTA input → MSA Generation (msa_feat) → MSA feature files
• MSA features → Atom‑level graph builder (atom_graph) → Graph data structures
• Graph data → Model.build_graph → Feature embeddings → Attention & SE3 modules → Distance predictions → Refine & fold → PDB output
• download_datasets → External datasets → MSA Generation steps
• Utilities & config are dependencies flowing into all modules
• Training pipeline: raw+processed data → LightRoseTTA_train.py → model parameters → weights
c. Architectural patterns & design principles:
• Modular Python packages by responsibility (separation of concerns)
• Dataflow pipeline: staged preprocessing → model → postprocessing
• Graph neural network with equivariant layers (SE(3) symmetry)
• Shell‑script wrappers for heavy bioinformatics dependencies
• Clear separation of inference vs training codebases
d. Key technologies & frameworks:
• Python 3.x, PyTorch
• Conda environment management
• Bioinformatics tools: BLAST, CS‑BLAST, PSIPRED
• Graph processing: custom atom_graph, SE3 equivariant modules
• Shell scripts for data generation
5. Tailoring the diagram for this open‑source ML tool
- Emphasize the core pipeline: Input FASTA → MSA → Graph → GNN → PDB.
- Show extension points: user‑provided FASTA, custom training data, plugin MSA tools.
- Highlight where external datasets and third‑party tools integrate.
- Indicate training vs inference flow side‑by‑side or in separate swimlanes.
6. Diagramming guidelines
- Use clear rectangular boxes for each package or script.
- Draw directional arrows to show data flow (e.g., FASTA → MSA module → atom_graph → model).
- Color code layers:
• Blue for Data Preparation
• Green for Model Components
• Orange for Scripts & Orchestration
• Gray for Utilities & Config
• Yellow for External Data & Weights
- Use distinct shapes (e.g., cylinders for dataset stores, parallelograms for scripts).
- Label each arrow with data artifacts (e.g., “MSA features,” “graph JSON,” “.pth weights”).
7. Essential details to capture
- The two‑stage nature: preprocessing vs model execution
- All major sub‑packages and their roles
- External dependencies and where they plug in
- Clear separation of training vs inference
- Data inputs/outputs at each boundary
By following these guidelines, the principal engineer will create a detailed, accurate system diagram that maps LightRoseTTA’s preprocessing pipeline, model internals, data dependencies, and orchestration scripts, enabling clear understanding and facilitating future extensions.
</explanation>