# muefab/genie architecture

> Architecture diagram of the GitHub repository muefab/genie, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/muefab/genie
- Repository: https://github.com/muefab/genie

## Overview

##### Primary Components
- Core Codec Engine – Representing the heart of Genie, it contains the main algorithm implementations for entropy coding (e.g. the “entropy” modules), format handling (e.g. the “format” subdirectories for FASTQ, SAM, MGB, MGG, MGREC), and record reconstruction.
- Input/Output Transcoding Applications – These include “transcode-fastq” and “transcode-sam” apps. They transform external genomic file formats (FASTQ, SAM) into an internal representation (MPEG-G records, “mgrec”) and vice versa.
- Run/Encoding Modules – The “run” application (and underlying libraries) handles the encoding process. Different encoding strategies (global assembly, low latency, local assembly, reference-based) are selected based on provided options (e.g. low latency flag, reference files).
- Core Libraries/Modules – Within the “src/genie” directory, there are modules for “core”, “read”, “quality”, “entropy”, “util”, “name” and “module”. Each of these serves a specific function:
- The “core” directory holds the data structures and processing logic (access unit, API interfaces, etc.).
- The “read” modules contain various approaches for reading and compressing data streams.
- The “entropy” modules implement the entropy coding techniques.
- The “quality” modules manage the quality score encoding (with different submodules like “calq” and “paramqv1”).
- The “name” modules and utilities handle tokenizing or encoding of read identifiers.
- External Dependency Integration – Although not a code module per se, the system must interface with components for optional SAM/BAM support (via HTSlib) and other external libraries.

##### Relationships & Data Flow
- Data Input: External genomic data (FASTQ, SAM, FASTA for references) is read via the transcoding apps.
- Internal Data Representation: Data is converted into internal MPEG-G record format (mgrec) by the transcoding modules.
- Encoding Process: The “run” module accepts the internal representation and applies one of several encoding strategies:
- Selection based on input parameters (e.g. a low latency flag or provided reference).
- Decoding Process: For decompression, the “run” module is also used to decode MPEG-G records back to the internal format, from which the transcoding modules revert the data to the original formats.
- Data Output: Final outputs (encoded files in .mgb, decoded files, re-transcoded FASTQ/SAM) are written out.

##### Architectural Patterns & Modularity
- Separation of concerns; each major function (transcoding, encoding, decoding, quality scoring, entropy coding, and I/O) is encapsulated in its own module.
- That some modules are organized using a layered approach: high-level command-line applications (“apps”) delegate to lower-level core libraries.

##### Technologies & Frameworks
- C++ (C++11 standard)
- CMake as the build system
- OpenMP for multithreading
- Optional tools (Doxygen for documentation)
- External libraries: HTSlib, CLI11, nlohmann/json, etc.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        DS["External Genomic Data"]:::external
    end

    %% Transcoding Applications Layer
    subgraph "Transcoding Applications"
        TF["Transcode FASTQ"]:::app
        TS["Transcode SAM"]:::app
        FF["Transcode FASTA"]:::app
    end

    MREC["Internal MPEG-G Records (mgrec)"]:::internal

    %% Run / Encoding Module Layer with Encoding Strategies
    subgraph "Run / Encoding Module"
        RUN["Run Module"]:::run
        GA["Global Assembly"]:::strategy
        LL["Low Latency"]:::strategy
        LA["Local Assembly"]:::strategy
        RB["Reference-Based Encoding"]:::strategy
    end

    %% Core Codec Engine & Libraries Layer
    subgraph "Core Codec Engine & Libraries"
        CORE["Core Processing Logic"]:::core
        ENT["Entropy Modules"]:::core
        FORMT["Format Handling"]:::core
        READ["Read Modules"]:::core
        QL["Quality Modules"]:::core
        NAME["Name Modules"]:::core
        UTIL["Utility Functions"]:::core
        MODULE["Module Management"]:::core
    end

    %% External Dependencies Layer
    subgraph "External Dependencies"
        CLI["CLI11"]:::externalDep
        JSON["nlohmann JSON Parser"]:::externalDep
        OTH["Other Thirdparty Libraries"]:::externalDep
        HTS["HTSlib"]:::externalDep
    end

    %% Build and Documentation Layer
    subgraph "Build & Documentation"
        CMAKE["CMake Build System"]:::build
        DOXY["Doxygen Configuration"]:::build
        CI["CI/CD Pipeline"]:::build
    end

    %% Data Flow Connections
    DS -->|"raw_data"| TF
    DS -->|"raw_data"| TS
    DS -->|"raw_data"| FF

    TF -->|"converts_to"| MREC
    TS -->|"converts_to"| MREC
    FF -->|"converts_to"| MREC

    MREC -->|"input_for"| RUN

    RUN -->|"selects"| GA
    RUN -->|"selects"| LL
    RUN -->|"selects"| LA
    RUN -->|"selects"| RB

    RUN -->|"delegates_to"| CORE
    RUN -->|"delegates_to"| ENT
    RUN -->|"delegates_to"| FORMT
    RUN -->|"delegates_to"| READ
    RUN -->|"delegates_to"| QL
    RUN -->|"delegates_to"| NAME
    RUN -->|"delegates_to"| UTIL
    RUN -->|"delegates_to"| MODULE

    GA -->|"encodes/decodes"| MREC
    LL -->|"encodes/decodes"| MREC
    LA -->|"encodes/decodes"| MREC
    RB -->|"encodes/decodes"| MREC

    %% External Dependencies usage
    CORE --- CLI
    CORE --- JSON
    CORE --- OTH
    TS --- HTS
    TF --- HTS

    %% Build & Documentation connections
    CMAKE --- CORE
    CMAKE --- RUN
    DOXY --- CORE
    CI --- CMAKE

    %% Click Events for Component Mapping
    click CORE "https://github.com/muefab/genie/tree/main/src/genie/core"
    click TF "https://github.com/muefab/genie/tree/main/src/apps/genie/transcode-fastq"
    click TS "https://github.com/muefab/genie/tree/main/src/apps/genie/transcode-sam"
    click FF "https://github.com/muefab/genie/tree/main/src/apps/genie/transcode-fasta"
    click RUN "https://github.com/muefab/genie/tree/main/src/apps/genie/run"
    click ENT "https://github.com/muefab/genie/tree/main/src/genie/entropy"
    click FORMT "https://github.com/muefab/genie/tree/main/src/genie/format"
    click READ "https://github.com/muefab/genie/tree/main/src/genie/read"
    click QL "https://github.com/muefab/genie/tree/main/src/genie/quality"
    click NAME "https://github.com/muefab/genie/tree/main/src/genie/name"
    click UTIL "https://github.com/muefab/genie/tree/main/src/genie/util"
    click MODULE "https://github.com/muefab/genie/tree/main/src/genie/module"
    click CLI "https://github.com/muefab/genie/blob/main/thirdparty/cli11/CLI11.hpp"
    click JSON "https://github.com/muefab/genie/blob/main/thirdparty/nlohmann/json.hpp"
    click CMAKE "https://github.com/muefab/genie/blob/main/CMakeLists.txt"
    click DOXY "https://github.com/muefab/genie/blob/main/cmake/Doxygen.cmake"

    %% Styles
    classDef external fill:#FFCCCC,stroke:#D9534F,stroke-width:2px;
    classDef app fill:#D0E6F8,stroke:#3498DB,stroke-width:2px;
    classDef internal fill:#FFF2CC,stroke:#F1C40F,stroke-width:2px;
    classDef run fill:#DFF0D8,stroke:#27AE60,stroke-width:2px;
    classDef strategy fill:#FCF3CF,stroke:#F4D03F,stroke-width:2px;
    classDef core fill:#D6EAF8,stroke:#5DADE2,stroke-width:2px;
    classDef externalDep fill:#F9E2E2,stroke:#E74C3C,stroke-width:2px;
    classDef build fill:#EDE7F6,stroke:#8E44AD,stroke-width:2px;
```
