# molmed/snpseq_metadata architecture

> Architecture diagram of the GitHub repository molmed/snpseq_metadata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/molmed/snpseq_metadata
- Repository: https://github.com/molmed/snpseq_metadata

## Overview

##### Project Purpose and Type
- The project is a Python-based command-line tool that parses and converts metadata from sequencing runfolders or service data (from snpseq_data) into several output formats (JSON, XML, TSV, manifest files) compatible with SRA/ENA requirements.
- Its core functionality is to read various input metadata, map metadata values from one source (Clarity LIMS) through multiple layered models (lims_models, ngi_models, sra_models) and export them using standardized XML schemas. In short, it’s a metadata processing and conversion pipeline.

##### File Structure and Key Components
- Top-Level Directories:
- snpseq_metadata: Contains the main package, including the primary command-line interface, model definitions, conversion logic, and mapping utilities.
- tests: Contains comprehensive tests (unit tests, integration tests, and sample test data) showing how the system is validated.
- .github and Dockerfile: Indicate CI/CD setup (GitHub Actions) and containerization support.
- Inside the snpseq_metadata package:
- __init__.py and utilities define shared functionality.
- models/ contains subdirectories for each metadata source: lims_models, ngi_models, sra_models. These demonstrate a clear layered design where each “model” embodies metadata in a particular format.
- converter.py in models provides a conversion layer that ties together the various formats.
- ngi_to_sra_mapping.py encapsulates domain-specific logic to translate between NGI and SRA terminologies.
- Scripts: Provide helper tools to generate Python models (using the xsdata library) from XML schemas.
- README: Offers a complete explanation of the commands (extract and export) and depicts how the metadata flows from Clarity LIMS extraction through the conversion pipeline to standardized output formats.

a. Input Sources:
- Runfolder (disk directory, containing raw sequencing data).
- snpseq_data files (JSON formatted data from the external snpseq_data service).
b. Command-Line Interface (CLI):
- The main entry point “snpseq_metadata” which then dispatches subcommands (“extract” and “export”).
c. Processing Layers (Data Models):
- lims_models: Represents initial metadata extraction from LIMS data.
- ngi_models: Intermediate representation used as a stepping stone for conversion.
- sra_models: Final representation that aligns with SRA/ENA XML schema requirements.
d. Conversion / Mapping Components:
- The converter module (converter.py) which orchestrates transitions between the data model layers.
- The NGI to SRA mapping (ngi_to_sra_mapping.py) that interprets and translates terminologies.
e. Output Modules:
- Serializers to convert data models into JSON, XML, manifest, and TSV files.
f. Auxiliary Components:
- Docker container (via Dockerfile) as an envelope for deployments.
- Testing infrastructure (unit tests and integration tests) that validates each stage of the pipeline.
- xsdata-generated models (used in the creation and export of XML).

##### Relationships and Data Flow
- A top-down flow starting from the CLI receiving command-line arguments.
- Under “extract”, split into two flows:
◦ One taking a runfolder as input and mapping it to NGI models then serializing to JSON.
◦ One taking SNPSEQ data (a JSON file) and converting it similarly.
- Merge flows in the “export” command where the tool takes the NGI JSON files and applies the conversion layer:
◦ Using the mapping logic from ngi_models to sra_models.
◦ Finally, using serializers to export results in XML, TSV, or manifest formats.

##### Architectural Patterns / Design Principles
- Layered Architecture: The clear separation into data model layers (lims, ngi, sra) highlights a pipeline design.
- Modular Mapping / Conversion: The dedicated converter and mapping modules emphasize separation of concerns.
- Extensibility: By abstracting each metadata format into its own module, it’s easier to add new sources or output formats.
- Testability: Inclusion of a complete tests directory reflects an emphasis on robust, test-driven development.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["CLI (metadata.py)"]:::cli
    A -->|"extract"| B["Extract Command"]
    A -->|"export"| C["Export Command"]

    subgraph "Extract Command"
        direction TB
        E1["Runfolder Input"]:::input
        E2["SNPSEQ Data (JSON)"]:::input
        E1 -->|"feeds"| E3["LIMS Models"]:::processing
        E2 -->|"feeds"| E3
        E3 -->|"maps_to"| E4["NGI Models"]:::processing
        E4 -->|"serializes_to"| E5["Output Serialization (JSON)"]:::output
    end

    subgraph "Export Command"
        direction TB
        F1["NGI JSON Files"]:::input
        F1 -->|"processed_by"| F2["Conversion Layer (converter.py)"]:::conversion
        F2 -->|"translates_via"| F3["NGI to SRA Mapping"]:::conversion
        F3 -->|"converts_to"| F4["SRA Models"]:::processing
        F4 -->|"exports_as"| F5["Output Serialization (XML/TSV/Manifest)"]:::output
    end

    subgraph "Auxiliary Components"
        direction TB
        G1["Docker Container"]:::auxiliary
        G2["Testing Framework"]:::auxiliary
        G3["XML Model Generator"]:::auxiliary
    end

    A ---|"deployed_with"| G1
    A ---|"validated_by"| G2
    A ---|"generates_models_via"| G3

    %% Click Events
    click A "https://github.com/molmed/snpseq_metadata/blob/main/snpseq_metadata/scripts/metadata.py"
    click E3 "https://github.com/molmed/snpseq_metadata/tree/main/snpseq_metadata/models/lims_models"
    click E4 "https://github.com/molmed/snpseq_metadata/tree/main/snpseq_metadata/models/ngi_models"
    click F2 "https://github.com/molmed/snpseq_metadata/blob/main/snpseq_metadata/models/converter.py"
    click F3 "https://github.com/molmed/snpseq_metadata/blob/main/snpseq_metadata/models/ngi_to_sra_mapping.py"
    click F5 "https://github.com/molmed/snpseq_metadata/blob/main/snpseq_metadata/models/import_export.py"
    click G1 "https://github.com/molmed/snpseq_metadata/tree/main/Dockerfile"
    click G2 "https://github.com/molmed/snpseq_metadata/tree/main/tests"
    click G3 "https://github.com/molmed/snpseq_metadata/blob/main/snpseq_metadata/scripts/generate_python_models.sh"

    %% Styles
    classDef cli fill:#f9c,stroke:#333,stroke-width:2px;
    classDef input fill:#ccf,stroke:#333,stroke-width:2px;
    classDef processing fill:#cfc,stroke:#333,stroke-width:2px;
    classDef conversion fill:#fc0,stroke:#333,stroke-width:2px;
    classDef output fill:#f90,stroke:#333,stroke-width:2px;
    classDef auxiliary fill:#fcc,stroke:#333,stroke-width:2px;
```
