How eric9n/kun-peng works
Project Type and Purpose
• The project is a command-line tool for metagenomic sequence classification. It includes both database construction (building a specialized k2d/block-based database) and sequence classification workflows.
• It uses Rust for performance and reliability and targets efficient resource management for processing large-scale genome databases.
• Key functions include downloading data (via the ncbi_dl tool), building the database, and classifying query sequences using different processing modes (direct, chunked, and step-by-step pipelines).
Analyzing the File Structure
• Top-level directories include:
– .github: Contains CI/CD workflows.
– data: Contains genome reference files, taxonomy files, and libraries used in building the database.
– examples: Provides example scripts to demonstrate building and classifying using the system.
– src: The heart of the code where you have:
▫ Modules for argument parsing (src/args.rs) and utility functions (src/utils.rs).
▫ Multiple binaries under src/bin for different functionalities (annotate, build_k2_db, chunk_db, classify, etc.).
▫ Core library code in src/lib.rs and additional modules such as taxonomy, report, db, and hash table management.
• Presence of configuration/build files (Cargo.toml, cal_memory.sh) imply that the system also needs to manage external binaries and resource estimations.
Key Components
a. Data Retrieval / Input Layer
– External genome and taxonomy data from the NCBI website (downloaded using ncbi_dl).
– Local data storage (data directory), which holds the raw sequence files (e.g., FASTA files in data/) and the reference databases.
b. Database Construction Module
– “build” command: Processes downloaded files, merges genome files, constructs the k2d (or block-based) database.
– Components such as “merge_fna”, “compact_hash”, and “db” modules that combine sequences and generate a hash table (as described by the k-mer and minimizer generation steps).
c. Classification Pipeline
– Input of query sequences (FASTA/FASTQ).
– The processing sub-pipeline:
▫ “splitr”: Splits input sequences into chunked ranges.
▫ “annotate”: Assigns taxonomy annotations based on minimizer lookups.
▫ “resolve”: Interprets taxonomic relationships (using taxonomy data in data/taxonomy) to generate final classification.
– Alternative “direct” mode for high-memory systems.
d. External Utilities and Tools
– “cal_memory.sh”: A shell script for calculating memory requirements.
– Integration of external tools like ncbi_dl for downloading genome data.
e. Output and Reporting
– Generation of standard Kraken-style output files (classification reports, kreport, etc.)
– Reporting modules in src/report.rs, which take processed data and generate human-readable output.
f. Execution Modes
– Modes like Direct Processing, Chunk Processing, and Step-by-Step Processing indicate that the system supports multiple execution pathways depending on memory and speed trade-offs.
Architectural Patterns and Relationships
• The design exhibits a layered structure:
– Data Acquisition Layer → Database Build Layer → Classification Pipeline → Reporting Layer.
• Modules communicate through clearly defined interfaces (e.g., the output of merge/estimate feeds directly into classification routines).
• The system uses a modular command-line pattern (multiple subcommands – each corresponding to a specific binary under src/bin) implemented in a single codebase. This suggests a pattern akin to microservices at the command level or a plugin-like architecture for extending functionality.
Final Touches
• Clearly label every component and sub-component.