# datacanvasio/hypergbm architecture

> Architecture diagram of the GitHub repository datacanvasio/hypergbm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datacanvasio/hypergbm
- Repository: https://github.com/datacanvasio/hypergbm

## Overview

##### Project type and purpose
- HyperGBM is a Python-based AutoML toolkit for tabular data. It provides both a Python API (`make_experiment`) and a CLI (`hypergbm train|evaluate|predict`).
- It orchestrates a full ML pipeline (data cleaning → feature generation/selection → model selection → hyperparameter tuning) using reinforcement learning, Monte Carlo tree search, evolutionary algorithms, and meta-learning.
- It supports CPU, GPU (via cuML), and distributed execution (via Dask).

##### File structure
- Top-level:
- `hypergbm/`: core Python package subdivided into:
– `cuml/`: GPU-accelerated pipelines
– `dask/`: distributed pipelines on Dask clusters
– `sklearn/`: CPU pipelines using scikit-learn
– `experiment_callbacks/`, `gbm_callbacks/`: callback hooks for training lifecycle
– `utils/cli.py`: CLI entry point; `utils/detect_estimator.py`: runtime capability detection
- `docs/`: Sphinx documentation, examples and how-to guides
- `hypergbm/examples/`: Jupyter notebooks demonstrating key workflows
- `hypergbm/tests/`: unit/integration tests per subpackage
- Config files: `Dockerfile*`, `.github/workflows/`, `setup.py`, `requirements-*.txt`.

##### README for additional insights
- Full pipeline AutoML using multiple GBM backends (XGBoost, LightGBM, CatBoost).
- Optional integration with Hypernets framework (for advanced cleaning, ensemble, search-space representation).
- GPU/Dask toggles at runtime enable acceleration or distributed execution.
- Provides both Python and CLI interfaces.

##### Main components and their interactions
a. Frontends:
- Python API (`hypergbm.make_experiment`)
- CLI (`hypergbm` command)
b. Orchestrator / Experiment Manager:
- Parses user inputs, dispatches to appropriate “engine” (sklearn / cuml / dask)
- Builds a SearchSpace object, configures callbacks
c. Pipeline Steps:
1. DataCleanStep
2. FeatureGeneration + FeatureSelection
3. ModelSelection + HyperparameterOptimization
4. Ensemble construction (GreedyEnsemble)
d. Search algorithms:
- Reinforcement learning controller
- Monte Carlo Tree Search
- Evolutionary Algorithm + Meta-learner
e. GBM Engines:
- CPU: XGBoost, LightGBM, CatBoost (via sklearn wrappers)
- GPU: cuDF/cuML versions
- Dask: distributed wrappers for feature ops and estimators
f. External services/dependencies:
- Hypernets (optional)
- Dask scheduler + workers
- HyperBoard (for visualization)

##### Architectural patterns & design principles
- Modular subpackages for each runtime environment (CPU/GPU/Dask)
- Plugin-style callbacks for extensibility (`experiment_callbacks/*`, `gbm_callbacks/*`)
- Separation of concerns: search-space definition, search algorithm, pipeline execution
- Factory/strategy pattern to select estimator backend at runtime
- CLI vs. API share the same orchestrator core

##### Elements
- Labels: clearly name each module (e.g., `hypergbm.utils.cli`, `hypergbm.dask._estimators`)
– User input → Orchestrator
– Pipeline Steps → Search Algorithm (bidirectional, for iterative trial)
– Search Algorithm → GBM Engines (spawn model training jobs)
– GBM Engines → Model Artifact (output)
– Optional → HyperBoard, Dask Scheduler
- Config flags (`--enable-gpu`, `--enable-dask`) branching logic to select engine subpackage

##### Customizing to HyperGBM
- “full-pipeline” nature: data cleaning and feature engineering are first-class stages.
- Multi-algorithm search (RL, MCTS, EA) interacting with the SearchSpace.
- Optional GPU/Dask branches in the engine selection path.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        direction TB
        API["Python API\nmake_experiment"]:::frontend
        CLI["CLI Interface\nhypergbm train/..."]:::frontend
    end

    %% Orchestrator Layer
    subgraph "Orchestrator Layer"
        direction TB
        ExperimentMgr["Experiment Manager\n(hyper_gbm.py & experiment.py)"]:::orchestration
        SearchSpaceBuilder["SearchSpace Builder\n(search_space.py)"]:::orchestration
        Detector["Runtime Capability Detector\n(detect_estimator.py)"]:::orchestration
        CallbackMgrExp["Experiment Callback Manager\n(experiment_callbacks/)"]:::orchestration
        CallbackMgrGBM["GBM Callback Manager\n(gbm_callbacks/)"]:::orchestration
    end

    %% Execution Pipeline Layer
    subgraph "Execution Layer"
        direction TB
        DataClean["DataCleanStep"]:::pipeline
        FeatureGenSel["FeatureGeneration\n& FeatureSelection"]:::pipeline
        ModelSelectOpt["ModelSelection\n& HyperparameterOptimization"]:::pipeline
        Ensemble["GreedyEnsemble Construction"]:::pipeline
    end

    %% Search Algorithm Layer
    subgraph "Search Algorithms"
        direction TB
        RL["Reinforcement Learning"]:::orchestration
        MCTS["Monte Carlo Tree Search"]:::orchestration
        EA["Evolutionary Algorithm + Meta-learner"]:::orchestration
    end

    %% Engine Layer
    subgraph "Engine Layer"
        direction TB

        subgraph "CPU Engine" 
            direction TB
            CPUOps["sklearn Ops & Estimators\n(sklearn_ops.py)"]:::backend
        end

        subgraph "GPU Engine" 
            direction TB
            GPU_Est["cuML Estimators\n(_estimators.py)"]:::backend
            GPU_SS["cuML SearchSpace\n(_search_space.py)"]:::backend
            GPU_Ops["cuML Ops\n(_ops.py)"]:::backend
        end

        subgraph "Dask Engine"
            direction TB
            Dask_Est["Distributed Estimators\n(_estimators.py)"]:::backend
            Dask_Ops["Distributed Ops\n(dask_ops.py)"]:::backend
            Dask_Trans["Distributed Transformers\n(dask_transformers.py)"]:::backend
            Dask_SS["Distributed SearchSpace\n(search_space.py)"]:::backend
        end
    end

    %% External Services
    subgraph "External Layer"
        direction TB
        Hypernets["Hypernets Framework"]:::external
        DaskCluster["Dask Scheduler & Workers"]:::external
        HyperBoard["HyperBoard Visualization"]:::external
    end

    %% Model Artifact
    ModelArtifact["Model Artifact"]:::data

    %% Connections
    API -->|"invoke"| ExperimentMgr
    CLI -->|"invoke"| ExperimentMgr

    ExperimentMgr --> SearchSpaceBuilder
    ExperimentMgr --> Detector
    ExperimentMgr --> CallbackMgrExp
    ExperimentMgr --> CallbackMgrGBM

    ExperimentMgr --> DataClean
    DataClean --> FeatureGenSel
    FeatureGenSel --> ModelSelectOpt
    ModelSelectOpt --> Ensemble

    ModelSelectOpt <--> RL
    ModelSelectOpt <--> MCTS
    ModelSelectOpt <--> EA

    RL --> CPUOps
    RL --> GPU_Est
    RL --> Dask_Est
    MCTS --> CPUOps
    MCTS --> GPU_Est
    MCTS --> Dask_Est
    EA --> CPUOps
    EA --> GPU_Est
    EA --> Dask_Est

    CPUOps --> ModelArtifact
    GPU_Est --> ModelArtifact
    Dask_Est --> ModelArtifact

    ExperimentMgr -->|"--enable-gpu"| GPU_Est
    ExperimentMgr -->|"--enable-dask"| Dask_Est

    Hypernets -. optional .-> SearchSpaceBuilder
    HyperBoard -. optional .-> ModelSelectOpt
    DaskCluster -. optional .-> Dask_Est

    %% Click Events
    click API "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/experiment.py"
    click CLI "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/utils/cli.py"
    click ExperimentMgr "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/hyper_gbm.py"
    click ExperimentMgr "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/experiment.py"
    click SearchSpaceBuilder "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/search_space.py"
    click Detector "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/utils/detect_estimator.py"
    click CallbackMgrExp "https://github.com/datacanvasio/hypergbm/tree/main/hypergbm/experiment_callbacks/"
    click CallbackMgrGBM "https://github.com/datacanvasio/hypergbm/tree/main/hypergbm/gbm_callbacks/"
    click CPUOps "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/sklearn/sklearn_ops.py"
    click GPU_Est "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/cuml/_estimators.py"
    click GPU_SS "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/cuml/_search_space.py"
    click GPU_Ops "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/cuml/_ops.py"
    click Dask_Est "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/dask/_estimators.py"
    click Dask_Ops "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/dask/dask_ops.py"
    click Dask_Trans "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/dask/dask_transformers.py"
    click Dask_SS "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/dask/search_space.py"
    click ModelArtifact "https://github.com/datacanvasio/hypergbm/blob/main/hypergbm/objectives.py"

    %% Styles
    classDef frontend fill:#f0f8ff,stroke:#333,stroke-width:1px
    classDef orchestration fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef pipeline fill:#c8e6c9,stroke:#333,stroke-width:1px
    classDef backend fill:#ffcc80,stroke:#333,stroke-width:1px
    classDef external fill:#eeeeee,stroke:#333,stroke-width:1px
    classDef data fill:#fff9c4,stroke:#333,stroke-width:1px
```
