How autogluon/autogluon works
Project type and purpose
• AutoGluon is an open-source AutoML library (monorepo of Python packages) that enables fast, accurate ML model training and deployment on tabular, multimodal (image/text), and time‐series data.
• Key use cases:
• TabularPredictor for structured data
• MultiModalPredictor for combined text/image/tabular
• TimeSeriesPredictor for probabilistic forecasting
• EDA toolkit for automated exploratory analysis
High-level file structure
• Top-level modules/packages (each with its own setup.py, src, tests):
• common – shared utilities (I/O, serialization, feature metadata)
• core – orchestration (HPO, model trial, scheduler, ensemble)
• tabular – tabular ML workflows (feature engineering, model families, predictor)
• multimodal – deep‐learning pipelines (data modules, learners, models, optim)
• timeseries – forecasting workflows (dataset, models, trainer, metrics)
• eda – automated exploratory data analysis (analysis + visualization)
• features – standalone feature generator library
• Supporting directories:
• docs & tutorials – system documentation, how to install, run examples
• CI & GitHub/actions – build/test pipelines, Dockerfiles, benchmarks
• examples – end-to-end sample scripts
Core architectural components
a. Client / API layer
• Predictor classes (TabularPredictor, MultiModalPredictor, TimeSeriesPredictor, EDA CLI)
• User invokes .fit(), .predict(), .analyze()
b. Data ingestion & preprocessing
• common.loaders & savers (load_pandas, load_s3, etc.)
• per-package data modules (tabular/data, multimodal/data, timeseries/dataset)
• feature generators (features/generators)
c. Orchestration & scheduling
• core.hpo (search space, RayTune executor, scheduler)
• core.scheduler (sequential or parallel job scheduling)
• core.callback & early stopping
d. Model training & ensembling
• core.models.abstract + concrete implementations (bagged ensemble, stackers)
• tabular.models (XGBoost, CatBoost, random forest, neural nets, etc.)
• multimodal.learners & models (transformers, fusion layers)
• timeseries models (AutogluonTabular wrappers, Chronos pipelines, ensemble)
e. Utilities & infrastructure
• common.utils (logging, resource management, context)
• core.ray integration (resource calculation, distributed jobs)
• checkpointing & caching (multimodal/utils/cache, common/savers)
f. Extension & registry mechanisms
• autogluon.*.registry for model/component registration
• config & preset YAML files (multimodal/configs, tabular/configs, timeseries/configs)
Interactions & data flows
• Data flow:
User data → API layer (.fit) → Data loaders → Feature generation → HPO search loop → Model training → Evaluation → Ensemble building → Predictor artifact
• Control flow:
Predictor invokes scheduler → each trial uses trainer/learner → callbacks report to scheduler/HPO → final model selection
• External integrations:
• Ray cluster for parallel HPO
• S3 / local filesystem for data & model storage
• HuggingFace model hub mirror (CI/hf_mirror)
• Cloud services via tutorials (SageMaker, AWS Lambda)
Architectural patterns & design principles
• Modular, layered design: core engine decoupled from domain packages
• Plugin/registry pattern for adding new model families or data handlers
• Configuration‐driven workflows via YAML presets
• Separation of concerns: data, model, training, HPO, orchestration
• Reuse of common utilities across packages
Essential details to capture
• Predictor → core.hpo & core.scheduler → per-package learner/trainer
• Data modules feeding feature generators & model inputs
• Model registry & ensemble building in core.models
• Common utilities (logging, resource management) underpinning all workflows
• Config & preset files driving default hyperparameters and pipelines