How muefab/genie works
Primary Components
• Core Codec Engine – Representing the heart of Genie, it contains the main algorithm implementations for entropy coding (e.g. the “entropy” modules), format handling (e.g. the “format” subdirectories for FASTQ, SAM, MGB, MGG, MGREC), and record reconstruction.
• Input/Output Transcoding Applications – These include “transcode-fastq” and “transcode-sam” apps. They transform external genomic file formats (FASTQ, SAM) into an internal representation (MPEG-G records, “mgrec”) and vice versa.
• Run/Encoding Modules – The “run” application (and underlying libraries) handles the encoding process. Different encoding strategies (global assembly, low latency, local assembly, reference-based) are selected based on provided options (e.g. low latency flag, reference files).
• Core Libraries/Modules – Within the “src/genie” directory, there are modules for “core”, “read”, “quality”, “entropy”, “util”, “name” and “module”. Each of these serves a specific function:
• The “core” directory holds the data structures and processing logic (access unit, API interfaces, etc.).
• The “read” modules contain various approaches for reading and compressing data streams.
• The “entropy” modules implement the entropy coding techniques.
• The “quality” modules manage the quality score encoding (with different submodules like “calq” and “paramqv1”).
• The “name” modules and utilities handle tokenizing or encoding of read identifiers.
• External Dependency Integration – Although not a code module per se, the system must interface with components for optional SAM/BAM support (via HTSlib) and other external libraries.
Relationships & Data Flow
• Data Input: External genomic data (FASTQ, SAM, FASTA for references) is read via the transcoding apps.
• Internal Data Representation: Data is converted into internal MPEG-G record format (mgrec) by the transcoding modules.
• Encoding Process: The “run” module accepts the internal representation and applies one of several encoding strategies:
• Selection based on input parameters (e.g. a low latency flag or provided reference).
• Decoding Process: For decompression, the “run” module is also used to decode MPEG-G records back to the internal format, from which the transcoding modules revert the data to the original formats.
• Data Output: Final outputs (encoded files in .mgb, decoded files, re-transcoded FASTQ/SAM) are written out.
Architectural Patterns & Modularity
• Separation of concerns; each major function (transcoding, encoding, decoding, quality scoring, entropy coding, and I/O) is encapsulated in its own module.
• That some modules are organized using a layered approach: high-level command-line applications (“apps”) delegate to lower-level core libraries.
Technologies & Frameworks
• C++ (C++11 standard)
• CMake as the build system
• OpenMP for multithreading
• Optional tools (Doxygen for documentation)
• External libraries: HTSlib, CLI11, nlohmann/json, etc.