# shandley/hecatomb architecture

> Architecture diagram of the GitHub repository shandley/hecatomb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shandley/hecatomb
- Repository: https://github.com/shandley/hecatomb

## Overview

##### Project Type and Purpose
- Hecatomb is a data analysis pipeline intended for viral metagenomic analysis. Its primary goal is to “sacrifice” false-positive viral reads in order to retain true-positive ones.
- The project uses Snakemake to orchestrate multiple steps (preprocessing, assembly, annotation, reporting) and relies on conda environments and external tools (e.g., Trimnami for trimming).

##### Analyzing the File Structure
- At the top level, notice directories like .github (for CI/CD workflows) and tests for unit testing.
- The main logic lives under the “hecatomb” directory with an __main__.py for launching the CLI. The “snakemake” subfolder is extensive and broken down into:
- config files (define parameters and database file paths)
- profiles (e.g., default and example_slurm for HPC usage),
- workflow definitions (workflow files, rule sets for annotation, assembly, preflight checks, preprocessing, and reports),
- environment YAML files that specify the dependencies for each workflow step.
- This design clearly maps to a modular and layered pipeline where each rule represents a discrete stage of the processing.

##### Key Components
a. Main CLI and Entry Point:
- “hecatomb CLI” (via __main__.py) as the starting point. This component parses user input and configuration options.
b. Configuration Management:
- A config loader that reads config files (e.g., hecatomb/snakemake/config/*.yaml) to set up parameters for runs, including paths to databases and settings for read-processing.
c. Snakemake Workflow Orchestrator:
- A central “Workflow Engine” block that represents Snakemake. Within this, break down key workflow stages:
- Preflight/Validation: Rules for setting up directories, validating input files.
- Preprocessing: Quality trimming (integrated with Trimnami) and initial filtering rules.
- Assembly and Mapping: Steps for contig assembly (shortreads/longreads support) and mapping using tools like minimap2, samtools, etc.
- Annotation: Rules for read and contig annotation using customized scripts.
- Environment Setup: Blocks showing the dependency environment YAMLs that define external tools.
d. Data Inputs and Outputs:
e. External Dependencies & HPC Specifics:
- An external “Conda/Env Manager” block that handles installation of dependencies.

##### Relationships and Interactions
- From the Configuration Loader to the Workflow Engine.
- “trigger” from test commands (e.g., hecatomb test) that initiate workflows with specific profiles.

##### Architectural Patterns and Design Principles
- The pipeline is modular with a clear separation-of-concerns: configuration, processing steps (rules), and reporting.
- It uses a declarative approach with Snakemake where the dependencies between rules are clearly defined, enabling reproducible and scalable workflows.
- The integration with conda highlights portability and ease of dependency management.
- HPC integration via Snakemake profiles indicates that the workflow can be adapted to different computational environments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top-level components
    A["Hecatomb CLI"]:::cli
    B["Config Loader"]:::config
    C["Snakemake Workflow Engine"]:::workflow
    D["Conda/Env Manager"]:::env
    E["HPC Profile (example_slurm)"]:::hpc
    F["CI/CD Workflows"]:::cicd
    G["Test Data"]:::data

    %% Subgraph for Workflow Engine stages
    subgraph "Workflow Stages"
        H["Preflight/Validation"]:::stage
        I["Preprocessing (Trimnami)"]:::stage
        J["Assembly & Mapping"]:::stage
        K["Annotation"]:::stage
        L["Reporting"]:::stage
    end

    %% Connections between top-level components
    G -->|"provides input"| A
    F -->|"triggers"| A
    A -->|"reads configuration"| B
    B -->|"initiates"| C
    D -->|"sets environment for"| C
    E -->|"configures HPC for"| C

    %% Flow within Workflow Engine stages
    C --> H
    H --> I
    I --> J
    J --> K
    K --> L

    %% Click events for component mapping
    click A "https://github.com/shandley/hecatomb/blob/main/hecatomb/__main__.py"
    click B "https://github.com/shandley/hecatomb/tree/main/hecatomb/snakemake/config"
    click C "https://github.com/shandley/hecatomb/tree/main/hecatomb/snakemake/workflow"
    click L "https://github.com/shandley/hecatomb/tree/main/hecatomb/snakemake/workflow/report"
    click D "https://github.com/shandley/hecatomb/tree/main/hecatomb/snakemake/workflow/envs"
    click E "https://github.com/shandley/hecatomb/tree/main/hecatomb/snakemake/profile/example_slurm"
    click F "https://github.com/shandley/hecatomb/tree/main/.github/workflows"
    click G "https://github.com/shandley/hecatomb/tree/main/hecatomb/test_data"

    %% Styles and colors
    classDef cli fill:#F9E79F,stroke:#D4AC0D,stroke-width:2px;
    classDef config fill:#AED6F1,stroke:#2E86C1,stroke-width:2px;
    classDef workflow fill:#D5F5E3,stroke:#58D68D,stroke-width:2px;
    classDef env fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef hpc fill:#FAD7A0,stroke:#F39C12,stroke-width:2px;
    classDef cicd fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef data fill:#FDEBD0,stroke:#E67E22,stroke-width:2px;
    classDef stage fill:#C3F6CE,stroke:#27AE60,stroke-width:2px;
```
