# cbg-ethz/sr2silo architecture

> Architecture diagram of the GitHub repository cbg-ethz/sr2silo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cbg-ethz/sr2silo
- Repository: https://github.com/cbg-ethz/sr2silo

## Overview

##### Project Type and Purpose
- The project is a command-line tool for processing nucleotide alignment data (from BAM/SAM files) and converting it into a cleartext JSON format suitable for importing into a SILO database. It handles the extraction of insertions/deletions, translation/alignment using tools like Diamond/BlastX, and can optionally upload results to a backend service.

##### Analyzing the File Structure
- Notice the clear separation into several top-level directories:
- “src/sr2silo”: Holds the core Python code including modules for configuration, logging, processing (with submodules like convert, merge, translate_align), as well as domain-specific logic (silo and vpipe).
- “tests”: Contains tests mapped to each functional area (process, storage, vpipe, etc.) ensuring unit and integration testing.
- “resources”: Provides external data including genome reference files, primers, and graphics (for branding).
- “environments” and “conda-recipe”: Define multiple environments (core, development, workflow) used for development, testing, and production.
- “workflow”: Contains a Snakemake workflow setup that integrates with the main processing engine.
- Scripts like “scripts/run.sh” and “scripts/submit.sh” which serve as entry points for execution.
- Additional files like the Makefile, pyproject.toml, and configuration YAMLs provide hints on how environments and dependencies are managed.

##### Main Components to Highlight
a. CLI/Entry Point:
- The sr2silo command with its subcommands (“import-to-loculus” being the primary one) which accepts various parameters.
b. Processing Pipeline:
- The core module “src/sr2silo/main.py” serving as the entry point for processing.
- The “process” submodules (convert.py, merge.py, translate_align.py, etc.) which encapsulate distinct stages (conversion from BAM to JSON, handling merge, and translating alignments).
- The “silo” submodule that likely interfaces with the LAPIS-SILO backend.
- The “vpipe” module used for metadata and sample handling for downstream analysis.
c. Configuration and Logging:
- Modules such as “config.py” and “logger.py” which manage system settings and runtime logging.
d. Storage and External Integration:
- The “storage” module (e.g., s3_storage.py) responsible for interactions with external storage (S3).
- Connections to external services like LAPIS-SILO, and the integration with Diamond/BlastX for translation and alignment.
e. Environments & Workflow:
- The separate Conda environment setups (core, dev, workflow) as defined in the “environments” directory.
- The Snakemake “workflow” which ties in with the processing pipeline for batch/cluster execution.

##### Relationships and Interactions
- That after processing, the output (NDJSON on disk) might be optionally uploaded (via the storage module) and passed to an external database (LAPIS-SILO).
- Configuration and logging modules as cross-cutting services supporting most components.
- Integration points for tests and CI/CD (reflecting the presence of GitHub workflows for testing, code style checking, etc.).

##### Architectural Patterns and Technologies
- The design follows a modular pattern where each distinct functionality is encapsulated (e.g., processing, storage, configuration).
- It leverages standard Python packaging, uses Poetry for dependency management, and Conda for environment isolation.
- The design also provides separation between batch workflow handling (via Snakemake) and interactive command-line operations.
- External tools (Diamond/BlastX) are plugged into the processing pipeline, so mark these integrations clearly.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI / Entry Point
    CLI1["CLI Interface (import_to_loculus)"]:::cli
    CLI2["Run Script (run.sh)"]:::cli

    %% Main Controller
    MC["Main Controller (main.py)"]:::core

    %% Processing Modules
    Conv["Data Conversion (convert.py)"]:::process
    Merge["Data Merging (merge.py)"]:::process
    Trans["Alignment Translation (translate_align.py)"]:::process
    Interface["Process Interface (interface.py)"]:::process

    %% Silo Integration
    Silo["Silo Connector (lapis.py)"]:::silo
    SiloSchema["Silo Schema (silo_read_schema.py)"]:::silo

    %% Storage & External Integration
    Storage["S3 Storage (s3_storage.py)"]:::storage

    %% Configuration & Logging
    Config["Configuration (config.py)"]:::config
    Log["Logging (logger.py)"]:::config

    %% Vpipe & Metadata Handling
    Meta["Metadata (metadata.py)"]:::vpipe
    Sample["Sample Handler (sample.py)"]:::vpipe

    %% Environments & Workflow
    Env["Environments"]:::env
    Workflow["Workflow (Snakemake)"]:::env

    %% External Tools
    Ext["External Tools (Diamond/BlastX)"]:::external

    %% Relationships: Data Flow
    CLI1 -->|"calls"| MC
    CLI2 -->|"calls"| MC

    MC -->|"triggers"| Conv
    MC -->|"triggers"| Merge
    MC -->|"triggers"| Trans
    MC -->|"triggers"| Interface
    MC -->|"handles"| Meta
    MC -->|"handles"| Sample

    Conv -->|"calls"| Ext
    Trans -->|"calls"| Ext

    Conv -->|"sends"| Silo
    Trans -->|"sends"| Silo

    Merge -->|"uploads"| Storage
    Trans -->|"uploads"| Storage

    %% Support: Configuration & Logging support Main Controller
    Config --> MC
    Log --> MC

    %% Workflow and Environment integration
    Env -->|"defines"| Workflow
    Workflow -->|"executes"| MC

    %% Click Events
    click CLI1 "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/import_to_loculus.py"
    click CLI2 "https://github.com/cbg-ethz/sr2silo/blob/main/scripts/run.sh"
    click MC "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/main.py"
    click Conv "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/process/convert.py"
    click Merge "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/process/merge.py"
    click Trans "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/process/translate_align.py"
    click Interface "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/process/interface.py"
    click Silo "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/silo/lapis.py"
    click SiloSchema "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/silo_read_schema.py"
    click Storage "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/storage/s3_storage.py"
    click Config "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/config.py"
    click Log "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/logger.py"
    click Meta "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/vpipe/metadata.py"
    click Sample "https://github.com/cbg-ethz/sr2silo/blob/main/src/sr2silo/vpipe/sample.py"
    click Env "https://github.com/cbg-ethz/sr2silo/tree/main/environments/"
    click Workflow "https://github.com/cbg-ethz/sr2silo/tree/main/workflow/"

    %% Style Definitions
    classDef cli fill:#f9e79f,stroke:#d4ac0d,stroke-width:2px;
    classDef core fill:#82e0aa,stroke:#27ae60,stroke-width:2px;
    classDef process fill:#aed6f1,stroke:#5499c7,stroke-width:2px;
    classDef silo fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef storage fill:#f5cba7,stroke:#e67e22,stroke-width:2px;
    classDef config fill:#d5f5e3,stroke:#1e8449,stroke-width:2px;
    classDef vpipe fill:#e8daef,stroke:#8e44ad,stroke-width:2px;
    classDef env fill:#f9e79f,stroke:#d4ac0d,stroke-width:2px;
    classDef external fill:#d6eaf8,stroke:#3498db,stroke-width:2px;
```
