How molmed/snpseq_metadata works
Project Purpose and Type
• The project is a Python-based command-line tool that parses and converts metadata from sequencing runfolders or service data (from snpseq_data) into several output formats (JSON, XML, TSV, manifest files) compatible with SRA/ENA requirements.
• Its core functionality is to read various input metadata, map metadata values from one source (Clarity LIMS) through multiple layered models (lims_models, ngi_models, sra_models) and export them using standardized XML schemas. In short, it’s a metadata processing and conversion pipeline.
File Structure and Key Components
• Top-Level Directories:
• snpseq_metadata: Contains the main package, including the primary command-line interface, model definitions, conversion logic, and mapping utilities.
• tests: Contains comprehensive tests (unit tests, integration tests, and sample test data) showing how the system is validated.
• .github and Dockerfile: Indicate CI/CD setup (GitHub Actions) and containerization support.
• Inside the snpseq_metadata package:
• __init__.py and utilities define shared functionality.
• models/ contains subdirectories for each metadata source: lims_models, ngi_models, sra_models. These demonstrate a clear layered design where each “model” embodies metadata in a particular format.
• converter.py in models provides a conversion layer that ties together the various formats.
• ngi_to_sra_mapping.py encapsulates domain-specific logic to translate between NGI and SRA terminologies.
• Scripts: Provide helper tools to generate Python models (using the xsdata library) from XML schemas.
• README: Offers a complete explanation of the commands (extract and export) and depicts how the metadata flows from Clarity LIMS extraction through the conversion pipeline to standardized output formats.
a. Input Sources:
• Runfolder (disk directory, containing raw sequencing data).
• snpseq_data files (JSON formatted data from the external snpseq_data service).
b. Command-Line Interface (CLI):
• The main entry point “snpseq_metadata” which then dispatches subcommands (“extract” and “export”).
c. Processing Layers (Data Models):
• lims_models: Represents initial metadata extraction from LIMS data.
• ngi_models: Intermediate representation used as a stepping stone for conversion.
• sra_models: Final representation that aligns with SRA/ENA XML schema requirements.
d. Conversion / Mapping Components:
• The converter module (converter.py) which orchestrates transitions between the data model layers.
• The NGI to SRA mapping (ngi_to_sra_mapping.py) that interprets and translates terminologies.
e. Output Modules:
• Serializers to convert data models into JSON, XML, manifest, and TSV files.
f. Auxiliary Components:
• Docker container (via Dockerfile) as an envelope for deployments.
• Testing infrastructure (unit tests and integration tests) that validates each stage of the pipeline.
• xsdata-generated models (used in the creation and export of XML).
Relationships and Data Flow
• A top-down flow starting from the CLI receiving command-line arguments.
• Under “extract”, split into two flows:
◦ One taking a runfolder as input and mapping it to NGI models then serializing to JSON.
◦ One taking SNPSEQ data (a JSON file) and converting it similarly.
• Merge flows in the “export” command where the tool takes the NGI JSON files and applies the conversion layer:
◦ Using the mapping logic from ngi_models to sra_models.
◦ Finally, using serializers to export results in XML, TSV, or manifest formats.
Architectural Patterns / Design Principles
• Layered Architecture: The clear separation into data model layers (lims, ngi, sra) highlights a pipeline design.
• Modular Mapping / Conversion: The dedicated converter and mapping modules emphasize separation of concerns.
• Extensibility: By abstracting each metadata format into its own module, it’s easier to add new sources or output formats.
• Testability: Inclusion of a complete tests directory reflects an emphasis on robust, test-driven development.