# ml-struct-bio/cryodrgn architecture

> Architecture diagram of the GitHub repository ml-struct-bio/cryodrgn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ml-struct-bio/cryodrgn
- Repository: https://github.com/ml-struct-bio/cryodrgn

## Overview

##### Project type and purpose
- CryoDRGN is an open‑source, Python‑based command‑line toolkit and library for heterogeneous cryo‑EM/-ET reconstruction using deep generative models.
- It exposes a single entry point (`cryodrgn`) with multiple subcommands (e.g., `train_vae`, `analyze`, `downsample`, `parse_pose_star`, `abinit_het`, etc.), each implemented under `cryodrgn/commands`.
- Core functionality includes data parsing (STAR/CS files, MRC stacks), dataset loading, neural network model definition (PyTorch), training/inference, and analysis/visualization pipelines.

##### Key directories and modules
- CLI layer:
- `cryodrgn/command_line.py` — registers subcommands
- `cryodrgn/commands/` — individual scripts implementing CLI entrypoints
- `analysis_scripts/` — standalone analysis utilities for clustering, plotting, embedding
- Core library:
- `cryodrgn/dataset.py`, `ctf.py`, `pose.py`, `mrcfile.py` — data I/O and preprocessing
- `cryodrgn/models.py`, `lie_tools.py`, `fft.py`, `losses.py` — model definitions and operations
- `cryodrgn/train_nn.py`, `train_vae.py`, `abinit_het.py`, `abinit_homo.py` — high‑level workflows
- `cryodrgn/utils.py`, `commands_utils/` — utility functions for filtering, masking, plotting
- `cryodrgn/config.py` — configuration management for training and analysis
- Templates and notebooks:
- `cryodrgn/templates/` — Jupyter notebooks for interactive analysis
- Testing & CI:
- `tests/` and `testing/` — unit and integration tests
- `.github/workflows/` — CI for style, tests, releases

##### External dependencies & services
- PyTorch for neural networks (training/inference on CPU/GPU)
- NumPy/SciPy for numeric operations, FFT routines
- UMAP, scikit‑learn for embedding/clustering
- File formats: RELION (.star), cryoSPARC (.cs), MRC/MRCs, pickle (.pkl)
- Distribution & CI: PyPI, GitHub Actions

##### Main components
A. User Interface
- Jupyter notebooks (templates)
B. Command Dispatcher
- Command‐line registry → routes to subcommand modules
C. Core Library
- Data I/O & preprocessing
- Dataset loader (with optional lazy loading, multithreading)
- Model & training engine (PyTorch)
- Analysis engine (latent embedding, volume generation)
D. Storage
- Input data: `.mrcs`, `.star`, `.cs`, `.txt`
- Intermediate: `pose.pkl`, `ctf.pkl`, `z.pkl`, `config.yaml`, `weights.pkl`
- Output volumes: `.mrc` files, plots, notebooks
E. Compute Resources
- CPU threads for FFT/data parsing
- GPU(s) for mixed‑precision training (`--multigpu`, `--amp`)
F. External Tools & Visualization
- ChimeraX (for volume viewing)
- Hyperparameter tracking via logs

##### Relationships & data flow
- CLI → Command dispatcher → Subcommand module
- Subcommand → calls core library functions:
- Data parsing (reads STAR/CS → `pose.pkl`/`ctf.pkl`)
- Preprocessing (`downsample`, filtering)
- Training (`train_vae` invokes dataset loader → model → optimizer → weights)
- Analysis (`analyze` loads `weights.pkl` & `z.pkl` → UMAP/PCA → volumes via decoder)
- Core library uses PyTorch and NumPy layers
- Outputs saved to file system; notebooks/templates instantiated for interactive use

##### Architectural patterns & principles
- Command pattern for CLI subcommands
- Layered architecture: UI layer → dispatcher → core services → data layer
- Plugin‑style extensibility via `cryodrgn/commands` and `commands_utils`
- Separation of concerns: I/O vs. computation vs. analysis
- Configuration as code (YAML) for reproducibility

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface Layer
    subgraph "User Interface"
        CLI["CLI: cryodrgn"]:::ui
        Templates["Notebook Templates"]:::ui
    end

    %% Command Dispatcher
    subgraph "Command Dispatcher"
        Registry["Command Registry"]:::core
        Subcommands["CLI Subcommands"]:::core
    end

    %% Analysis Scripts
    AnalysisScripts["Standalone Analysis Utilities"]:::core

    %% Core Library Layer
    subgraph "Core Library"
        subgraph "Data I/O & Preprocessing"
            STAR["STAR Parsing"]:::core
            CTF["CTF Handling"]:::core
            Pose["Pose Handling"]:::core
            MRC["MRC I/O"]:::core
            Dataset["Dataset Loader"]:::core
        end
        subgraph "Model & Operations"
            Models["Model Definitions"]:::core
            Lie["Lie-Group Tools"]:::core
            FFT["FFT Routines"]:::core
            Losses["Loss Functions"]:::core
        end
        subgraph "High-Level Workflows"
            VAE["train_vae"]:::core
            NN["train_nn"]:::core
            AbHet["abinit_het"]:::core
            AbHomo["abinit_homo"]:::core
        end
        subgraph "Analysis Commands"
            Analyze["analyze"]:::core
            ALand["analyze_landscape"]:::core
            ALandFull["analyze_landscape_full"]:::core
        end
        CommandsUtils["Command Utilities"]:::core
        Utils["General Utils"]:::core
        Config["Configuration Management"]:::core
    end

    %% Storage Layer
    subgraph "Storage"
        InputFiles["Input Data"]:::storage
        Intermediate["Intermediate Files"]:::storage
        OutputVolumes["Output Volumes"]:::storage
    end

    %% Compute Resources
    subgraph "Compute Resources"
        CPU["CPU Threads"]:::resource
        GPU["GPUs (--multigpu, --amp)"]:::resource
    end

    %% External Dependencies & Tools
    subgraph "External Dependencies"
        PyTorch["PyTorch"]:::external
        NumPy["NumPy/SciPy"]:::external
        UMAP["UMAP/Sklearn"]:::external
        ChimeraX["ChimeraX"]:::external
    end

    %% CI/CD Sidebar
    subgraph "CI/CD"
        Tests["Tests"]:::ci
        CIWorkflows["GitHub Actions"]:::ci
    end

    %% Connections
    CLI --> Registry
    CLI --> Templates
    Registry --> Subcommands
    Subcommands --> STAR
    Subcommands --> CTF
    Subcommands --> Pose
    Subcommands --> MRC
    Subcommands --> Dataset
    Subcommands --> VAE
    Subcommands --> NN
    Subcommands --> AbHet
    Subcommands --> AbHomo
    Subcommands --> Analyze
    Subcommands --> ALand
    Subcommands --> ALandFull
    Subcommands --> CommandsUtils
    Subcommands --> Utils
    Subcommands --> Config

    Analyze --> AnalysisScripts
    ALand --> AnalysisScripts
    ALandFull --> AnalysisScripts

    STAR --> Intermediate
    CTF --> Intermediate
    Pose --> Intermediate
    Dataset --> Intermediate
    VAE --> OutputVolumes
    NN --> OutputVolumes
    AbHet --> OutputVolumes
    AbHomo --> OutputVolumes
    Analyze --> OutputVolumes
    ALand --> OutputVolumes
    ALandFull --> OutputVolumes
    Templates --> OutputVolumes

    VAE --> CPU
    NN --> GPU
    AbHet --> GPU
    AbHomo --> GPU

    Dataset --> PyTorch
    Models --> PyTorch
    Models --> NumPy
    FFT --> NumPy
    Losses --> PyTorch

    OutputVolumes --> ChimeraX

    Tests --> CIWorkflows
    CIWorkflows --> OutputVolumes

    %% Click Events
    click CLI "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/command_line.py"
    click Subcommands "https://github.com/ml-struct-bio/cryodrgn/tree/main/cryodrgn/commands/"
    click AnalysisScripts "https://github.com/ml-struct-bio/cryodrgn/tree/main/analysis_scripts/"
    click Templates "https://github.com/ml-struct-bio/cryodrgn/tree/main/cryodrgn/templates/"
    click Config "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/config.py"
    click STAR "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/starfile.py"
    click CTF "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/ctf.py"
    click Pose "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/pose.py"
    click MRC "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/mrcfile.py"
    click Dataset "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/dataset.py"
    click Models "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/models.py"
    click Lie "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/lie_tools.py"
    click FFT "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/fft.py"
    click Losses "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/losses.py"
    click VAE "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/commands/train_vae.py"
    click NN "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/commands/train_nn.py"
    click AbHet "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/commands/abinit_het.py"
    click AbHomo "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/commands/abinit_homo.py"
    click Analyze "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/commands/analyze.py"
    click ALand "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/commands/analyze_landscape.py"
    click ALandFull "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/commands/analyze_landscape_full.py"
    click CommandsUtils "https://github.com/ml-struct-bio/cryodrgn/tree/main/cryodrgn/commands_utils/"
    click Utils "https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/utils.py"
    click Tests "https://github.com/ml-struct-bio/cryodrgn/tree/main/tests/"
    click CIWorkflows "https://github.com/ml-struct-bio/cryodrgn/tree/main/.github/workflows/"

    %% Styles
    classDef ui fill:#AED6F1,stroke:#1F618D,color:#1F618D;
    classDef core fill:#ABEBC6,stroke:#1D8348,color:#1D8348;
    classDef storage fill:#D5DBDB,stroke:#626567,color:#626567;
    classDef resource fill:#F9E79F,stroke:#B7950B,color:#B7950B;
    classDef external fill:#F5CBA7,stroke:#B9770E,color:#B9770E;
    classDef ci fill:#E8DAEF,stroke:#7D3C98,color:#7D3C98;
```
