How microsoft/flaml works
Project Type and Purpose
• This is a Python library (not a deployed service) that provides:
• Automated Machine Learning (AutoML) with low compute footprint (“automl” module).
• Hyperparameter tuning as a standalone tool (“tune” module).
• An LLM‐based multi‐agent conversation framework (“autogen” module).
• Online learning support (“onlineml” module).
• Defaults & presets for common learners (“default” folder).
• Integrations with MLflow, AzureML, Spark, HuggingFace, XGBoost, LightGBM, scikit‐learn, etc.
File‐Structure Analysis
• Top‐level “flaml/” folder: source code, subdivided by functionality:
• flaml/automl – core AutoML engine (task definitions, model selection, search loop, state).
• flaml/tune – general-purpose hyperparameter tuning library (searchers, schedulers, trials).
• flaml/autogen – multi-agent orchestration (Agent classes, chain-of-thought, LLM wrappers).
• flaml/onlineml – streaming/online learning support.
• flaml/default – default HP configurations for various learners.
• flaml/fabric – MLflow adapter.
• flaml/config, logger, utils, version.
• Other top‐level dirs:
• test/ – comprehensive unit tests.
• notebook/ and tutorials/ – Jupyter examples.
• website/ – Docusaurus‐based documentation and blog.
• CI/CD config under .github/workflows, Dockerfiles, pyproject.toml, setup.py.
Insights from README
• The library exposes three main APIs:
a. AutoML as sklearn‐style estimator: “from flaml import AutoML; automl.fit(...)”
b. Hyperparameter tuning: “from flaml import tune; tune.run(...)”
c. Multi‐agent LLM orchestration: “from flaml import autogen; autogen.AssistantAgent(...)”
• Integrations: OpenAI API, ChatCompletion, MLflow, Spark, AzureML, HuggingFace.
Main Components
a. API Layer (User Interface)
– AutoML API
– Tune API
– Autogen API
– OnlineML API
b. Core Engines
– flaml.automl.AutomlEngine (task factory, search loop, model manager)
– flaml.tune.Tuner (trial runner, scheduler, searchers)
– flaml.autogen.AgentFramework (agents, conversation manager)
– flaml.onlineml.OnlineTrialRunner
c. Extension/Plugin Modules
– Searchers: blendsearch, CFO, Flow2, online_searcher
– Schedulers: trial_scheduler, online_scheduler
– Default configurations: JSON presets under flaml/default
– Third‐party wrapper modules: spark.utils, nlp/huggingface, fabric/mlflow
d. External Services and Libraries
– LLM providers: OpenAI API
– ML frameworks: scikit‐learn, XGBoost, LightGBM, SparkML, HuggingFace Transformers
– MLOps: MLflow, AzureML pipelines
– Data storage/log: local file system, object_store, MLflow Tracking
e. Configuration & Orchestration
– flaml/config
– CLI entry points (setup.py console_scripts if any)
– CI/CD pipelines (.github/workflows)
Interactions and Data Flow
• User code calls API Layer → Core Engine starts search or chat →
• For AutoML/Tune: Core Engine selects configs → TrialScheduler dispatches trials →
– Each trial uses specific estimator (XGB, LGBM, RF, custom) or user-defined function
– Records metrics via flaml/logger or MLflow adapter → Scheduler/Searcher uses metrics to guide next configs
• For Autogen: Conversation Manager sends messages to Agent classes →
– Agents wrap OpenAI.Completion/ChatCompletion → optional caching, tuning, templating → returns responses → conversation continues
• OnlineML: streams data, updates model incrementally.
• Default module supplies search space presets loaded by engines.
• Config module reads user options; logging streams to console or MLflow.
Architectural Patterns and Principles
• Modular “Engine + Plugin” design: engines orchestrate workflows, plugins (searchers, schedulers, estimators) are registered dynamically.
• Separation of concerns:
• API Layer for user interface
• Core logic for search orchestration
• Plugins for algorithmic strategies
• Integrations/adapters for external systems
• Extensibility via config‐driven defaults and Python extension points.
• Observer pattern for logging/tracking (logger & fabric/mlflow).
Detailed Elements
• Module names and high‐level class names (AutomlEngine, Tuner, AgentFramework)
• Plugin subfolders (searcher, scheduler, default)
• External adapters (fabric/mlflow.py, nlp/huggingface)
Final Checklist
• Clear, consistent labels for each component
• Extensibility points (where new searchers/schedulers/models can be plugged in)
• Keep it at a single page, focusing on major modules and their interactions