# opengene/fastp architecture

> Architecture diagram of the GitHub repository opengene/fastp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opengene/fastp
- Repository: https://github.com/opengene/fastp

## Overview

##### Project type and purpose
- This is a native C++ command-line application (“fastp”) for ultrafast all-in-one preprocessing and quality control of FASTQ sequencing data, plus a companion Python script (`parallel.py`) for batch orchestration and aggregation of reports.

##### File structure
- Top level:
- `parallel.py` – Python batch-processing orchestrator.
- `main.cpp`, `Makefile`, CI config – build and entrypoint.
- `src/` – C++ source and headers, organized by functionality (readers, filters, processors, reporters, thread management, utilities).
- `src/zlib/` – vendored zlib headers for gzip support.
- `testdata/` – sample FASTQ files for tests.
- No frontend/server code; pure CLI.
- CI pipeline via `.github/workflows/ci.yml`.
- Dependencies: zlib, libdeflate, isa-l (external libs).

##### README for additional insights
- Describes all pipeline stages (reading, filtering, trimming, adapter detection, UMI handling, merging, deduplication, reporting).
- Mentions multi-threading (default 3 threads), streaming to STDIN/STDOUT, split outputs for parallel downstream.

a. Main components
1. CLI Frontend (argument parsing via `src/cmdline.h`)
2. I/O Module
- FASTQ/FASTA Reader (`fastqreader`, `fastareader`)
- Gzip/Deflate Interface (vendored zlib, libdeflate)
- STDIN/STDOUT streaming
3. Preprocessing Modules (pipeline stages)
- UMI Processor (`umiprocessor`)
- Global Trimmer (`processor`, `filter`, `trim_*`)
- Quality Cutters (sliding window, `cut_front`, `cut_tail`, `cut_right`)
- Adapter Trimmer (`adaptertrimmer`, `matcher`, `knownadapters`)
- PolyG/PolyX Trimmer
- Base Corrector / Overlap Analysis (`overlapanalysis`, `basecorrector`)
- Merge PE Reads (`peprocessor`, `writerthread`)
- Deduplicator / Duplication Stats (`duplicate`, `stats`)
4. Thread Management
- Reader Pool (`readpool`)
- Worker Threads (`threadconfig`, multiple `processor` instances)
- Writer Threads (`writerthread`)
5. Reporter Module
- JSON Reporter (`jsonreporter`)
- HTML Reporter (`htmlreporter`)
6. Parallel Orchestration (external)
- Python script `parallel.py` for batch invocation
- Aggregator for combined HTML summary

b. Relationships and data flow
- CLI parses arguments → initializes ThreadConfig
- Reader Pool spawns I/O threads: read compressed FASTQ → produce record batches → enqueue to worker queue
- Worker Threads dequeue batches → apply pipeline stages in order → enqueue to writer queue
- Writer Threads dequeue processed batches → write to output files or STDOUT, optionally split into multiple files
- Reporter collects metrics during processing → at end, Reporter Module emits JSON and HTML
- For batch runs, `parallel.py` spawns multiple `fastp` processes in parallel, then parses individual JSONs to produce `overall.html`.

c. Key architectural patterns / design principles
- Pipeline pattern: sequential processing stages on streaming data
- Producer-consumer (multi-threaded) for I/O and processing
- Modular single-responsibility classes (each filter / trimmer / processor is self-contained)
- Pluggable reporters for different output formats
- Vendored dependencies for reproducible builds

d. Technologies and libraries
- C++17 (likely), POSIX threads or std::thread via `threadconfig`
- Vendored zlib (`src/zlib/`) and external libdeflate, isa-l
- Python 3 for batch script

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Frontend"
        direction TB
        CLI["CLI Frontend"]:::cli
    end

    %% Thread Management Layer
    subgraph "Thread Management" 
        direction TB
        TC["ThreadConfig"]:::thread
        RP["Reader Pool"]:::thread
        WT["Worker Threads"]:::thread
        WTR["Writer Threads"]:::thread
    end

    %% I/O Layer
    subgraph "I/O Module"
        direction TB
        FQ["FASTQ Reader"]:::io
        FA["FASTA Reader"]:::io
        ZI["Gzip/Deflate Interface"]:::io
        RS["Streaming & Record Abstraction"]:::io
    end

    %% Processing Layer
    subgraph "Preprocessing Pipeline"
        direction TB
        UMI["UMI Processor"]:::proc
        TRIM["Global Trimmer & Filter"]:::proc
        ADPT["Adapter Trimmer"]:::proc
        POLY["PolyG/PolyX Trimmer"]:::proc
        OVER["Overlap Analysis & Base Correction"]:::proc
        MERGE["Paired-End Merge"]:::proc
        DEDUP["Deduplication & Stats"]:::proc
    end

    %% Reporting Layer
    subgraph "Reporter Module"
        direction TB
        JSON["JSON Reporter"]:::report
        HTML["HTML Reporter"]:::report
    end

    %% External Orchestration
    subgraph "External Orchestration"
        direction TB
        PY["parallel.py"]:::external
        AGG["HTML Aggregator"]:::external
    end

    %% Data Stores
    Input[("Input FASTQ/FASTA")]:::data
    Output[("Output FASTQ/Streams")]:::data
    Metrics[("Metrics Data")]:::data
    Reports[("JSON / HTML Reports")]:::data

    %% Flow
    CLI -->|parses args| TC
    TC --> RP
    RP --> FQ
    RP --> FA
    FQ --> RS
    FA --> RS
    RS --> ZI
    RS -->|"batches"| WT
    WT --> UMI
    UMI --> TRIM
    TRIM --> ADPT
    ADPT --> POLY
    POLY --> OVER
    OVER --> MERGE
    MERGE --> DEDUP
    DEDUP --> WTR
    WTR --> Output
    DEDUP -->|metrics| Metrics
    Metrics --> JSON
    Metrics --> HTML
    JSON --> Reports
    HTML --> Reports

    PY -->|spawns processes| CLI
    PY -->|collects JSON| AGG
    AGG --> Reports

    %% Click Events
    click CLI "https://github.com/opengene/fastp/blob/master/src/main.cpp"
    click CLI "https://github.com/opengene/fastp/blob/master/src/cmdline.h"
    click TC "https://github.com/opengene/fastp/blob/master/src/threadconfig.cpp"
    click TC "https://github.com/opengene/fastp/blob/master/src/threadconfig.h"
    click RP "https://github.com/opengene/fastp/blob/master/src/readpool.cpp"
    click RP "https://github.com/opengene/fastp/blob/master/src/readpool.h"
    click FQ "https://github.com/opengene/fastp/blob/master/src/fastqreader.cpp"
    click FQ "https://github.com/opengene/fastp/blob/master/src/fastqreader.h"
    click FA "https://github.com/opengene/fastp/blob/master/src/fastareader.cpp"
    click FA "https://github.com/opengene/fastp/blob/master/src/fastareader.h"
    click ZI "https://github.com/opengene/fastp/tree/master/src/zlib/"
    click ZI "https://github.com/opengene/fastp/blob/master/src/libdeflate.h"
    click ZI "https://github.com/opengene/fastp/blob/master/src/igzip_lib.h"
    click RS "https://github.com/opengene/fastp/blob/master/src/read.cpp"
    click RS "https://github.com/opengene/fastp/blob/master/src/read.h"
    click RS "https://github.com/opengene/fastp/blob/master/src/sequence.cpp"
    click RS "https://github.com/opengene/fastp/blob/master/src/sequence.h"
    click UMI "https://github.com/opengene/fastp/blob/master/src/umiprocessor.cpp"
    click UMI "https://github.com/opengene/fastp/blob/master/src/umiprocessor.h"
    click TRIM "https://github.com/opengene/fastp/blob/master/src/processor.cpp"
    click TRIM "https://github.com/opengene/fastp/blob/master/src/processor.h"
    click TRIM "https://github.com/opengene/fastp/blob/master/src/filter.cpp"
    click TRIM "https://github.com/opengene/fastp/blob/master/src/filter.h"
    click ADPT "https://github.com/opengene/fastp/blob/master/src/adaptertrimmer.cpp"
    click ADPT "https://github.com/opengene/fastp/blob/master/src/adaptertrimmer.h"
    click ADPT "https://github.com/opengene/fastp/blob/master/src/matcher.cpp"
    click ADPT "https://github.com/opengene/fastp/blob/master/src/matcher.h"
    click ADPT "https://github.com/opengene/fastp/blob/master/src/knownadapters.h"
    click POLY "https://github.com/opengene/fastp/blob/master/src/polyx.cpp"
    click POLY "https://github.com/opengene/fastp/blob/master/src/polyx.h"
    click OVER "https://github.com/opengene/fastp/blob/master/src/overlapanalysis.cpp"
    click OVER "https://github.com/opengene/fastp/blob/master/src/overlapanalysis.h"
    click OVER "https://github.com/opengene/fastp/blob/master/src/basecorrector.cpp"
    click OVER "https://github.com/opengene/fastp/blob/master/src/basecorrector.h"
    click MERGE "https://github.com/opengene/fastp/blob/master/src/peprocessor.cpp"
    click MERGE "https://github.com/opengene/fastp/blob/master/src/peprocessor.h"
    click DEDUP "https://github.com/opengene/fastp/blob/master/src/duplicate.cpp"
    click DEDUP "https://github.com/opengene/fastp/blob/master/src/duplicate.h"
    click DEDUP "https://github.com/opengene/fastp/blob/master/src/stats.cpp"
    click DEDUP "https://github.com/opengene/fastp/blob/master/src/stats.h"
    click WTR "https://github.com/opengene/fastp/blob/master/src/writerthread.cpp"
    click WTR "https://github.com/opengene/fastp/blob/master/src/writerthread.h"
    click WTR "https://github.com/opengene/fastp/blob/master/src/writer.cpp"
    click WTR "https://github.com/opengene/fastp/blob/master/src/writer.h"
    click JSON "https://github.com/opengene/fastp/blob/master/src/jsonreporter.cpp"
    click JSON "https://github.com/opengene/fastp/blob/master/src/jsonreporter.h"
    click HTML "https://github.com/opengene/fastp/blob/master/src/htmlreporter.cpp"
    click HTML "https://github.com/opengene/fastp/blob/master/src/htmlreporter.h"
    click PY "https://github.com/opengene/fastp/blob/master/parallel.py"

    %% Styles
    classDef cli fill:#e5f5e0,stroke:#2d572c,stroke-width:1px
    classDef io fill:#e0f2f1,stroke:#00695c,stroke-width:1px
    classDef proc fill:#e3f2fd,stroke:#1565c0,stroke-width:1px
    classDef thread fill:#fff3e0,stroke:#ef6c00,stroke-width:1px
    classDef report fill:#f3e5f5,stroke:#6a1b9a,stroke-width:1px
    classDef external fill:#eceff1,stroke:#607d8b,stroke-width:1px
    classDef data fill:#ffffff,stroke:#9e9e9e,stroke-width:1px,stroke-dasharray: 5 5
```
