# eric9n/kun-peng architecture

> Architecture diagram of the GitHub repository eric9n/kun-peng, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eric9n/kun-peng
- Repository: https://github.com/eric9n/kun-peng

## Overview

##### Project Type and Purpose
- The project is a command-line tool for metagenomic sequence classification. It includes both database construction (building a specialized k2d/block-based database) and sequence classification workflows.
- It uses Rust for performance and reliability and targets efficient resource management for processing large-scale genome databases.
- Key functions include downloading data (via the ncbi_dl tool), building the database, and classifying query sequences using different processing modes (direct, chunked, and step-by-step pipelines).

##### Analyzing the File Structure
- Top-level directories include:
– .github: Contains CI/CD workflows.
– data: Contains genome reference files, taxonomy files, and libraries used in building the database.
– examples: Provides example scripts to demonstrate building and classifying using the system.
– src: The heart of the code where you have:
▫ Modules for argument parsing (src/args.rs) and utility functions (src/utils.rs).
▫ Multiple binaries under src/bin for different functionalities (annotate, build_k2_db, chunk_db, classify, etc.).
▫ Core library code in src/lib.rs and additional modules such as taxonomy, report, db, and hash table management.
- Presence of configuration/build files (Cargo.toml, cal_memory.sh) imply that the system also needs to manage external binaries and resource estimations.

##### Key Components
a. Data Retrieval / Input Layer
– External genome and taxonomy data from the NCBI website (downloaded using ncbi_dl).
– Local data storage (data directory), which holds the raw sequence files (e.g., FASTA files in data/) and the reference databases.
b. Database Construction Module
– “build” command: Processes downloaded files, merges genome files, constructs the k2d (or block-based) database.
– Components such as “merge_fna”, “compact_hash”, and “db” modules that combine sequences and generate a hash table (as described by the k-mer and minimizer generation steps).
c. Classification Pipeline
– Input of query sequences (FASTA/FASTQ).
– The processing sub-pipeline:
▫ “splitr”: Splits input sequences into chunked ranges.
▫ “annotate”: Assigns taxonomy annotations based on minimizer lookups.
▫ “resolve”: Interprets taxonomic relationships (using taxonomy data in data/taxonomy) to generate final classification.
– Alternative “direct” mode for high-memory systems.
d. External Utilities and Tools
– “cal_memory.sh”: A shell script for calculating memory requirements.
– Integration of external tools like ncbi_dl for downloading genome data.
e. Output and Reporting
– Generation of standard Kraken-style output files (classification reports, kreport, etc.)
– Reporting modules in src/report.rs, which take processed data and generate human-readable output.
f. Execution Modes
– Modes like Direct Processing, Chunk Processing, and Step-by-Step Processing indicate that the system supports multiple execution pathways depending on memory and speed trade-offs.

##### Architectural Patterns and Relationships
- The design exhibits a layered structure:
– Data Acquisition Layer → Database Build Layer → Classification Pipeline → Reporting Layer.
- Modules communicate through clearly defined interfaces (e.g., the output of merge/estimate feeds directly into classification routines).
- The system uses a modular command-line pattern (multiple subcommands – each corresponding to a specific binary under src/bin) implemented in a single codebase. This suggests a pattern akin to microservices at the command level or a plugin-like architecture for extending functionality.

##### Final Touches
- Clearly label every component and sub-component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    NC("ncbi_dl"):::external
    DS("Data Download & Storage"):::dataIn
    NC --> DS

    subgraph "Database Build Layer"
        build_k2("Build K2 DB"):::build
        merge_fna("Merge FNA"):::build
        compact_hash("Compact Hash"):::build
        db_module("DB Module"):::build
    end

    DS --> build_k2
    DS --> merge_fna
    build_k2 --> compact_hash
    merge_fna --> compact_hash
    compact_hash --> db_module

    subgraph "Classification Pipeline"
        subgraph "Direct Mode"
            direct_processing("Direct Processing"):::classify
            classify_module("Classify Module"):::classify
        end
        subgraph "Chunk/Step Mode"
            chunk_db("Chunk DB"):::classify
            splitr_module("Splitr Module"):::classify
            annotate_module("Annotate Module"):::classify
            resolve_module("Resolve Module"):::classify
        end
    end

    db_module --> direct_processing
    db_module --> chunk_db

    direct_processing --> classify_module
    chunk_db --> splitr_module
    splitr_module --> annotate_module
    annotate_module --> resolve_module

    subgraph "External Utilities & Tools"
        cal_memory("Memory Estimation"):::external
        estimate_cap("Estimate Capacity"):::external
    end

    cal_memory -.-> compact_hash
    estimate_cap -.-> compact_hash

    report_module("Reporting Module"):::reporting
    classify_module --> report_module
    resolve_module --> report_module

    classDef dataIn fill:#b3e5fc,stroke:#0288d1,stroke-width:2px;
    classDef build fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef classify fill:#fff9c4,stroke:#fbc02d,stroke-width:2px;
    classDef external fill:#ffe0b2,stroke:#fb8c00,stroke-width:2px;
    classDef reporting fill:#d1c4e9,stroke:#7e57c2,stroke-width:2px;

    click DS "https://github.com/eric9n/kun-peng/tree/main/data/"
    click build_k2 "https://github.com/eric9n/kun-peng/blob/main/src/bin/build_k2_db.rs"
    click merge_fna "https://github.com/eric9n/kun-peng/blob/main/src/bin/merge_fna.rs"
    click compact_hash "https://github.com/eric9n/kun-peng/blob/main/src/compact_hash.rs"
    click db_module "https://github.com/eric9n/kun-peng/blob/main/src/db.rs"
    click direct_processing "https://github.com/eric9n/kun-peng/blob/main/src/bin/direct.rs"
    click classify_module "https://github.com/eric9n/kun-peng/blob/main/src/classify.rs"
    click chunk_db "https://github.com/eric9n/kun-peng/blob/main/src/bin/chunk_db.rs"
    click splitr_module "https://github.com/eric9n/kun-peng/blob/main/src/bin/splitr.rs"
    click annotate_module "https://github.com/eric9n/kun-peng/blob/main/src/bin/annotate.rs"
    click resolve_module "https://github.com/eric9n/kun-peng/blob/main/src/bin/resolve.rs"
    click cal_memory "https://github.com/eric9n/kun-peng/blob/main/cal_memory.sh"
    click estimate_cap "https://github.com/eric9n/kun-peng/blob/main/src/bin/estimate_capacity.rs"
    click report_module "https://github.com/eric9n/kun-peng/blob/main/src/report.rs"
```
