# ylab-hi/deepchopper architecture

> Architecture diagram of the GitHub repository ylab-hi/deepchopper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ylab-hi/deepchopper
- Repository: https://github.com/ylab-hi/deepchopper

## Overview

##### Main Components
- User Interface/CLI Layer:
- This includes the Python CLI (deepchopper/cli.py), the web interface (deepchopper/ui/main.py), and the Python library interface.
- How users kick off tasks (encode, predict, chop) via command line and the web UI (accessible through Hugging Face Spaces).
- Core Processing & Model Engine:
- Deep learning model handling (found under deepchopper/models, including submodules for CNN, LLM, and downstream callbacks).
- Data Handling & Preprocessing:
- Components under deepchopper/data (modules for data ingestion, preparation, and encoding of FASTQ data).
- Rust modules in src/ (such as src/bin/ files and modules under src/fq_encode and src/output) which deal with efficient encoding, file format conversions, and other high-performance tasks.
- Utility & Infrastructure:
- Shared components like deepchopper/utils (for logging, instantiation, and other utilities), configuration management (including hydra and logger configurations), and testing modules.
- External Integrations & Dependencies:
- How the system leverages external Python libraries (for example, support for various loggers as configured under configs/logger such as mlflow, wandb, tensorboard etc.)
- That it supports multi-GPU, distributed training, and integration with other sequencing workflows.

##### Relationships and Data Flow
- Data ingestion modules (deepchopper/data and the Rust modules under src for encoding operations).
- Next, illustrate the control flow towards the “Core Model Engine” where the pre-trained models (from deepchopper/models, including LLM adaptations) process incoming data.
- Configuration and logging components so it’s clear that behavior is controlled via the YAML configuration files (configs/…) and distributed logging is managed by various logger adapters.
- For the Rust integration seen in the src/ directory, note that they implement performance-critical utilities (encoding, file manipulation) which are called by either CLI commands or Python bindings (e.g., via py.typed in deepchopper).

##### Denote Architectural Patterns & Design Principles
- Clearly separate the architecture into layers – Interface (CLI/Web/UI), Application Core (model processing, command handling), and Infrastructure (data processing, utility functions, external tool integrations).
- Modular design: each major function (encoding, prediction, chopping) is split out into separate modules; configuration is externalized to YAML files, enabling flexible experimentation.
- Co-existence of micro-components (small executables under src/bin, model components in Python) and highlight the integration points between the Python-based orchestration and the high-performance Rust utilities.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Interface Layer
    subgraph "Interface Layer"
        UI_CLI["Python CLI"]:::interface
        UI_Web["Web UI"]:::interface
    end

    %% Data Preprocessing (Python/Rust)
    subgraph "Data Preprocessing (Python/Rust)"
        D_encode["encode_fq.py"]:::data
        D_datamodule["fq_datamodule.py"]:::data
        D_RustEncode["encode.rs"]:::data
        D_fq["Rust: fq_encode"]:::data
        D_output["Rust: output"]:::data
    end

    %% Core Model Engine
    subgraph "Core Model Engine"
        M_CNN["CNN Module"]:::core
        M_LLM["LLM Modules"]:::core
        M_Train["Train Module"]:::core
    end

    %% Configuration & Logging
    subgraph "Configuration & Logging"
        C_Model["Model Config"]:::config
        C_Exp["Experiment Config"]:::config
        C_Trainer["Trainer Config"]:::config
        C_Mlflow["MLflow Logger"]:::config
        C_Wandb["WandB Logger"]:::config
    end

    %% Utility & Infrastructure
    subgraph "Utility & Infrastructure"
        U_Logging["Logging Utils"]:::utility
        U_Instantiators["Instantiators"]:::utility
    end

    %% Data Flow Connections
    %% Interface -> Data Preprocessing: Users initiate with CLI/Web UI; input flows to data layer.
    UI_CLI -->|"input"| D_encode
    UI_CLI -->|"input"| D_datamodule
    UI_Web -->|"input"| D_encode
    UI_Web -->|"input"| D_datamodule
    UI_CLI -->|"trigger"| D_RustEncode
    UI_Web -->|"trigger"| D_RustEncode
    D_encode --> D_fq
    D_datamodule --> D_fq
    D_RustEncode --> D_output

    %% Data Preprocessing -> Core Model Engine: Processed data flows into modeling
    D_fq -->|"preprocessed data"| M_CNN
    D_output -->|"preprocessed data"| M_LLM

    %% Within Core Model Engine: Modules lead to training/chopping functionality
    M_CNN --> M_Train
    M_LLM --> M_Train

    %% Core Model Engine -> Output: Model outcomes feed back to CLI for operations like chop
    M_Train -->|"results"| UI_CLI

    %% Configuration & Logging integration: Configurations support model engine
    C_Model --> M_CNN
    C_Exp --> M_LLM
    C_Trainer --> M_Train
    C_Mlflow --- M_Train
    C_Wandb --- M_Train

    %% Utility integration: Utility components support model engine and data processing
    U_Logging --- M_Train
    U_Instantiators --- M_CNN

    %% Click Events
    click UI_CLI "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/cli.py"
    click UI_Web "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/ui/main.py"
    click D_encode "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/data/encode_fq.py"
    click D_datamodule "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/data/fq_datamodule.py"
    click D_RustEncode "https://github.com/ylab-hi/deepchopper/blob/main/src/bin/encode.rs"
    click D_fq "https://github.com/ylab-hi/deepchopper/tree/main/src/fq_encode/"
    click D_output "https://github.com/ylab-hi/deepchopper/tree/main/src/output/"
    click M_CNN "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/models/cnn.py"
    click M_LLM "https://github.com/ylab-hi/deepchopper/tree/main/deepchopper/models/llm/"
    click M_Train "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/train.py"
    click C_Model "https://github.com/ylab-hi/deepchopper/tree/main/configs/model"
    click C_Exp "https://github.com/ylab-hi/deepchopper/tree/main/configs/experiment"
    click C_Trainer "https://github.com/ylab-hi/deepchopper/tree/main/configs/trainer"
    click C_Mlflow "https://github.com/ylab-hi/deepchopper/blob/main/configs/logger/mlflow.yaml"
    click C_Wandb "https://github.com/ylab-hi/deepchopper/blob/main/configs/logger/wandb.yaml"
    click U_Logging "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/utils/logging_utils.py"
    click U_Instantiators "https://github.com/ylab-hi/deepchopper/blob/main/deepchopper/utils/instantiators.py"

    %% Styling
    classDef interface fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef data fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef core fill:#a9dfbf,stroke:#28b463,stroke-width:2px;
    classDef config fill:#f9c5d1,stroke:#c0392b,stroke-width:2px;
    classDef utility fill:#d7bde2,stroke:#8e44ad,stroke-width:2px;
```
