How ml-struct-bio/cryodrgn works
Project type and purpose
• CryoDRGN is an open‑source, Python‑based command‑line toolkit and library for heterogeneous cryo‑EM/-ET reconstruction using deep generative models.
• It exposes a single entry point (cryodrgn) with multiple subcommands (e.g., train_vae, analyze, downsample, parse_pose_star, abinit_het, etc.), each implemented under cryodrgn/commands.
• Core functionality includes data parsing (STAR/CS files, MRC stacks), dataset loading, neural network model definition (PyTorch), training/inference, and analysis/visualization pipelines.
Key directories and modules
• CLI layer:
• cryodrgn/command_line.py — registers subcommands
• cryodrgn/commands/ — individual scripts implementing CLI entrypoints
• analysis_scripts/ — standalone analysis utilities for clustering, plotting, embedding
• Core library:
• cryodrgn/dataset.py, ctf.py, pose.py, mrcfile.py — data I/O and preprocessing
• cryodrgn/models.py, lie_tools.py, fft.py, losses.py — model definitions and operations
• cryodrgn/train_nn.py, train_vae.py, abinit_het.py, abinit_homo.py — high‑level workflows
• cryodrgn/utils.py, commands_utils/ — utility functions for filtering, masking, plotting
• cryodrgn/config.py — configuration management for training and analysis
• Templates and notebooks:
• cryodrgn/templates/ — Jupyter notebooks for interactive analysis
• Testing & CI:
• tests/ and testing/ — unit and integration tests
• .github/workflows/ — CI for style, tests, releases
External dependencies & services
• PyTorch for neural networks (training/inference on CPU/GPU)
• NumPy/SciPy for numeric operations, FFT routines
• UMAP, scikit‑learn for embedding/clustering
• File formats: RELION (.star), cryoSPARC (.cs), MRC/MRCs, pickle (.pkl)
• Distribution & CI: PyPI, GitHub Actions
Main components
A. User Interface
• Jupyter notebooks (templates)
B. Command Dispatcher
• Command‐line registry → routes to subcommand modules
C. Core Library
• Data I/O & preprocessing
• Dataset loader (with optional lazy loading, multithreading)
• Model & training engine (PyTorch)
• Analysis engine (latent embedding, volume generation)
D. Storage
• Input data: .mrcs, .star, .cs, .txt
• Intermediate: pose.pkl, ctf.pkl, z.pkl, config.yaml, weights.pkl
• Output volumes: .mrc files, plots, notebooks
E. Compute Resources
• CPU threads for FFT/data parsing
• GPU(s) for mixed‑precision training (--multigpu, --amp)
F. External Tools & Visualization
• ChimeraX (for volume viewing)
• Hyperparameter tracking via logs
Relationships & data flow
• CLI → Command dispatcher → Subcommand module
• Subcommand → calls core library functions:
• Data parsing (reads STAR/CS → pose.pkl/ctf.pkl)
• Preprocessing (downsample, filtering)
• Training (train_vae invokes dataset loader → model → optimizer → weights)
• Analysis (analyze loads weights.pkl & z.pkl → UMAP/PCA → volumes via decoder)
• Core library uses PyTorch and NumPy layers
• Outputs saved to file system; notebooks/templates instantiated for interactive use
Architectural patterns & principles
• Command pattern for CLI subcommands
• Layered architecture: UI layer → dispatcher → core services → data layer
• Plugin‑style extensibility via cryodrgn/commands and commands_utils
• Separation of concerns: I/O vs. computation vs. analysis
• Configuration as code (YAML) for reproducibility