How shandley/hecatomb works
Project Type and Purpose
• Hecatomb is a data analysis pipeline intended for viral metagenomic analysis. Its primary goal is to “sacrifice” false-positive viral reads in order to retain true-positive ones.
• The project uses Snakemake to orchestrate multiple steps (preprocessing, assembly, annotation, reporting) and relies on conda environments and external tools (e.g., Trimnami for trimming).
Analyzing the File Structure
• At the top level, notice directories like .github (for CI/CD workflows) and tests for unit testing.
• The main logic lives under the “hecatomb” directory with an __main__.py for launching the CLI. The “snakemake” subfolder is extensive and broken down into:
• config files (define parameters and database file paths)
• profiles (e.g., default and example_slurm for HPC usage),
• workflow definitions (workflow files, rule sets for annotation, assembly, preflight checks, preprocessing, and reports),
• environment YAML files that specify the dependencies for each workflow step.
• This design clearly maps to a modular and layered pipeline where each rule represents a discrete stage of the processing.
Key Components
a. Main CLI and Entry Point:
• “hecatomb CLI” (via __main__.py) as the starting point. This component parses user input and configuration options.
b. Configuration Management:
• A config loader that reads config files (e.g., hecatomb/snakemake/config/*.yaml) to set up parameters for runs, including paths to databases and settings for read-processing.
c. Snakemake Workflow Orchestrator:
• A central “Workflow Engine” block that represents Snakemake. Within this, break down key workflow stages:
• Preflight/Validation: Rules for setting up directories, validating input files.
• Preprocessing: Quality trimming (integrated with Trimnami) and initial filtering rules.
• Assembly and Mapping: Steps for contig assembly (shortreads/longreads support) and mapping using tools like minimap2, samtools, etc.
• Annotation: Rules for read and contig annotation using customized scripts.
• Environment Setup: Blocks showing the dependency environment YAMLs that define external tools.
d. Data Inputs and Outputs:
e. External Dependencies & HPC Specifics:
• An external “Conda/Env Manager” block that handles installation of dependencies.
Relationships and Interactions
• From the Configuration Loader to the Workflow Engine.
• “trigger” from test commands (e.g., hecatomb test) that initiate workflows with specific profiles.
Architectural Patterns and Design Principles
• The pipeline is modular with a clear separation-of-concerns: configuration, processing steps (rules), and reporting.
• It uses a declarative approach with Snakemake where the dependencies between rules are clearly defined, enabling reproducible and scalable workflows.
• The integration with conda highlights portability and ease of dependency management.
• HPC integration via Snakemake profiles indicates that the workflow can be adapted to different computational environments.