How opengene/fastp works
Project type and purpose
• This is a native C++ command-line application (“fastp”) for ultrafast all-in-one preprocessing and quality control of FASTQ sequencing data, plus a companion Python script (parallel.py) for batch orchestration and aggregation of reports.
File structure
• Top level:
• parallel.py – Python batch-processing orchestrator.
• main.cpp, Makefile, CI config – build and entrypoint.
• src/ – C++ source and headers, organized by functionality (readers, filters, processors, reporters, thread management, utilities).
• src/zlib/ – vendored zlib headers for gzip support.
• testdata/ – sample FASTQ files for tests.
• No frontend/server code; pure CLI.
• CI pipeline via .github/workflows/ci.yml.
• Dependencies: zlib, libdeflate, isa-l (external libs).
README for additional insights
• Describes all pipeline stages (reading, filtering, trimming, adapter detection, UMI handling, merging, deduplication, reporting).
• Mentions multi-threading (default 3 threads), streaming to STDIN/STDOUT, split outputs for parallel downstream.
a. Main components
1. CLI Frontend (argument parsing via src/cmdline.h)
2. I/O Module
• FASTQ/FASTA Reader (fastqreader, fastareader)
• Gzip/Deflate Interface (vendored zlib, libdeflate)
• STDIN/STDOUT streaming
3. Preprocessing Modules (pipeline stages)
• UMI Processor (umiprocessor)
• Global Trimmer (processor, filter, trim_*)
• Quality Cutters (sliding window, cut_front, cut_tail, cut_right)
• Adapter Trimmer (adaptertrimmer, matcher, knownadapters)
• PolyG/PolyX Trimmer
• Base Corrector / Overlap Analysis (overlapanalysis, basecorrector)
• Merge PE Reads (peprocessor, writerthread)
• Deduplicator / Duplication Stats (duplicate, stats)
4. Thread Management
• Reader Pool (readpool)
• Worker Threads (threadconfig, multiple processor instances)
• Writer Threads (writerthread)
5. Reporter Module
• JSON Reporter (jsonreporter)
• HTML Reporter (htmlreporter)
6. Parallel Orchestration (external)
• Python script parallel.py for batch invocation
• Aggregator for combined HTML summary
b. Relationships and data flow
• CLI parses arguments → initializes ThreadConfig
• Reader Pool spawns I/O threads: read compressed FASTQ → produce record batches → enqueue to worker queue
• Worker Threads dequeue batches → apply pipeline stages in order → enqueue to writer queue
• Writer Threads dequeue processed batches → write to output files or STDOUT, optionally split into multiple files
• Reporter collects metrics during processing → at end, Reporter Module emits JSON and HTML
• For batch runs, parallel.py spawns multiple fastp processes in parallel, then parses individual JSONs to produce overall.html.
c. Key architectural patterns / design principles
• Pipeline pattern: sequential processing stages on streaming data
• Producer-consumer (multi-threaded) for I/O and processing
• Modular single-responsibility classes (each filter / trimmer / processor is self-contained)
• Pluggable reporters for different output formats
• Vendored dependencies for reproducible builds
d. Technologies and libraries
• C++17 (likely), POSIX threads or std::thread via threadconfig
• Vendored zlib (src/zlib/) and external libdeflate, isa-l
• Python 3 for batch script