How jdisset/kmap works
Main Components
• Command Line Interface (CLI) & Input Handling: This component is responsible for reading command line arguments (indicated by flags like –k, –i, –o, –a, etc.), loading configuration and input file lists.
• Sequence File Parser: A module that reads fasta sequence files, performs any preprocessing (like prepending “start” and appending “stop” symbols), and passes the sequences on for kmer analysis.
• Kmer Counter / Matrix Builder: The core logic that identifies each kmer and counts its occurrences. This component builds a matrix where each row corresponds to an input sequence file and each column corresponds to letters (from the chosen alphabet).
• Data Persistence & Database Layer: Integration with RocksDB (built via integrated CMake scripts and triggered on build) manages the storage of intermediate kmer data. It ensures efficient read/write operations, especially for large sequence datasets.
• Conversion / Post-Processing Unit (kconv): Once kmap has computed counts in a sparse format, kconv takes the output data and formats or consolidates it into the final CSV (or another user-specified format).
• Python Integration Layer (pykmap): Via the included pybind11 framework, this module exposes parts of the C++ API to Python, allowing for scripting or integration with Python-based tools.
Out Relationships and Interactions
• The CLI module passes parameters to the Sequence Parser.
• Parsed sequence data flows into the Kmer Counter which uses internal helper functions (likely defined in header files like alphabet.hpp, export.hpp, and utils.hpp).
• As counting progresses, the Database Layer (with RocksDB support, as well as considerations for libBz2 and compression libraries configured via CMake) handles data storage and retrieval.
• Once counting is finished, the Kconv component uses the stored data to generate the final output file.
• Optionally, the Python Integration Layer (pykmap) bridges the core C++ functionality to Python. This module is likely built separately using its own CMakeLists.txt under pykmap/cpp.
• The build process, driven by CMake (using the top-level CMakeLists.txt and additional cmake scripts), orchestrates building all modules and pulling in dependencies (like RocksDB).
Architectural Patterns & Design Principles
• Modular Design: Each processing step (parsing, counting, conversion) is separated into independent source files and modules.
• Extensibility: The separate “pykmap” module and use of pybind11 indicate a clear design intention to expose functionality for further expansion, e.g., to embed in Python workflows.
• Use of External Libraries & Dependency Management: The use of RocksDB and several third-party header-only libraries is managed via CMake, making dependency resolution transparent.
• Pipeline Processing: The workflow follows a clear pipeline (Input -> Processing -> Data Storage -> Output Conversion).