How datacanvasio/hypergbm works
Project type and purpose
• HyperGBM is a Python-based AutoML toolkit for tabular data. It provides both a Python API (make_experiment) and a CLI (hypergbm train|evaluate|predict).
• It orchestrates a full ML pipeline (data cleaning → feature generation/selection → model selection → hyperparameter tuning) using reinforcement learning, Monte Carlo tree search, evolutionary algorithms, and meta-learning.
• It supports CPU, GPU (via cuML), and distributed execution (via Dask).
File structure
• Top-level:
• hypergbm/: core Python package subdivided into:
– cuml/: GPU-accelerated pipelines
– dask/: distributed pipelines on Dask clusters
– sklearn/: CPU pipelines using scikit-learn
– experiment_callbacks/, gbm_callbacks/: callback hooks for training lifecycle
– utils/cli.py: CLI entry point; utils/detect_estimator.py: runtime capability detection
• docs/: Sphinx documentation, examples and how-to guides
• hypergbm/examples/: Jupyter notebooks demonstrating key workflows
• hypergbm/tests/: unit/integration tests per subpackage
• Config files: Dockerfile*, .github/workflows/, setup.py, requirements-*.txt.
README for additional insights
• Full pipeline AutoML using multiple GBM backends (XGBoost, LightGBM, CatBoost).
• Optional integration with Hypernets framework (for advanced cleaning, ensemble, search-space representation).
• GPU/Dask toggles at runtime enable acceleration or distributed execution.
• Provides both Python and CLI interfaces.
Main components and their interactions
a. Frontends:
• Python API (hypergbm.make_experiment)
• CLI (hypergbm command)
b. Orchestrator / Experiment Manager:
• Parses user inputs, dispatches to appropriate “engine” (sklearn / cuml / dask)
• Builds a SearchSpace object, configures callbacks
c. Pipeline Steps:
1. DataCleanStep
2. FeatureGeneration + FeatureSelection
3. ModelSelection + HyperparameterOptimization
4. Ensemble construction (GreedyEnsemble)
d. Search algorithms:
• Reinforcement learning controller
• Monte Carlo Tree Search
• Evolutionary Algorithm + Meta-learner
e. GBM Engines:
• CPU: XGBoost, LightGBM, CatBoost (via sklearn wrappers)
• GPU: cuDF/cuML versions
• Dask: distributed wrappers for feature ops and estimators
f. External services/dependencies:
• Hypernets (optional)
• Dask scheduler + workers
• HyperBoard (for visualization)
Architectural patterns & design principles
• Modular subpackages for each runtime environment (CPU/GPU/Dask)
• Plugin-style callbacks for extensibility (experiment_callbacks/*, gbm_callbacks/*)
• Separation of concerns: search-space definition, search algorithm, pipeline execution
• Factory/strategy pattern to select estimator backend at runtime
• CLI vs. API share the same orchestrator core
Elements
• Labels: clearly name each module (e.g., hypergbm.utils.cli, hypergbm.dask._estimators)
– User input → Orchestrator
– Pipeline Steps → Search Algorithm (bidirectional, for iterative trial)
– Search Algorithm → GBM Engines (spawn model training jobs)
– GBM Engines → Model Artifact (output)
– Optional → HyperBoard, Dask Scheduler
• Config flags (--enable-gpu, --enable-dask) branching logic to select engine subpackage
Customizing to HyperGBM
• “full-pipeline” nature: data cleaning and feature engineering are first-class stages.
• Multi-algorithm search (RL, MCTS, EA) interacting with the SearchSpace.
• Optional GPU/Dask branches in the engine selection path.