How cbg-ethz/sr2silo works
Project Type and Purpose
• The project is a command-line tool for processing nucleotide alignment data (from BAM/SAM files) and converting it into a cleartext JSON format suitable for importing into a SILO database. It handles the extraction of insertions/deletions, translation/alignment using tools like Diamond/BlastX, and can optionally upload results to a backend service.
Analyzing the File Structure
• Notice the clear separation into several top-level directories:
• “src/sr2silo”: Holds the core Python code including modules for configuration, logging, processing (with submodules like convert, merge, translate_align), as well as domain-specific logic (silo and vpipe).
• “tests”: Contains tests mapped to each functional area (process, storage, vpipe, etc.) ensuring unit and integration testing.
• “resources”: Provides external data including genome reference files, primers, and graphics (for branding).
• “environments” and “conda-recipe”: Define multiple environments (core, development, workflow) used for development, testing, and production.
• “workflow”: Contains a Snakemake workflow setup that integrates with the main processing engine.
• Scripts like “scripts/run.sh” and “scripts/submit.sh” which serve as entry points for execution.
• Additional files like the Makefile, pyproject.toml, and configuration YAMLs provide hints on how environments and dependencies are managed.
Main Components to Highlight
a. CLI/Entry Point:
• The sr2silo command with its subcommands (“import-to-loculus” being the primary one) which accepts various parameters.
b. Processing Pipeline:
• The core module “src/sr2silo/main.py” serving as the entry point for processing.
• The “process” submodules (convert.py, merge.py, translate_align.py, etc.) which encapsulate distinct stages (conversion from BAM to JSON, handling merge, and translating alignments).
• The “silo” submodule that likely interfaces with the LAPIS-SILO backend.
• The “vpipe” module used for metadata and sample handling for downstream analysis.
c. Configuration and Logging:
• Modules such as “config.py” and “logger.py” which manage system settings and runtime logging.
d. Storage and External Integration:
• The “storage” module (e.g., s3_storage.py) responsible for interactions with external storage (S3).
• Connections to external services like LAPIS-SILO, and the integration with Diamond/BlastX for translation and alignment.
e. Environments & Workflow:
• The separate Conda environment setups (core, dev, workflow) as defined in the “environments” directory.
• The Snakemake “workflow” which ties in with the processing pipeline for batch/cluster execution.
Relationships and Interactions
• That after processing, the output (NDJSON on disk) might be optionally uploaded (via the storage module) and passed to an external database (LAPIS-SILO).
• Configuration and logging modules as cross-cutting services supporting most components.
• Integration points for tests and CI/CD (reflecting the presence of GitHub workflows for testing, code style checking, etc.).
Architectural Patterns and Technologies
• The design follows a modular pattern where each distinct functionality is encapsulated (e.g., processing, storage, configuration).
• It leverages standard Python packaging, uses Poetry for dependency management, and Conda for environment isolation.
• The design also provides separation between batch workflow handling (via Snakemake) and interactive command-line operations.
• External tools (Diamond/BlastX) are plugged into the processing pipeline, so mark these integrations clearly.