How muzammilck/metatune works
1. What this repo is
MetaTune is a single-process Python ML orchestration system for hyperparameter optimization. Its core architecture is a pipeline that takes a CSV dataset, analyzes its statistical characteristics, chooses a model family, tunes hyperparameters, trains a model, packages the result, and stores the run outcome for future learning.
2. Primary entry points
There are two user-facing fronts: app.py for the Streamlit dashboard and pipeline.py for headless CLI execution. Both drive the same core modules, so the UI and batch path are separate shells around a shared optimization/training core.
3. Data ingestion and dataset “DNA”
data_analyzer.py is the first real subsystem in the workflow. It loads a CSV, detects the target column, classifies the task as classification or regression, and computes the dataset “DNA” meta-features described in the README: size, feature mix, missingness, sparsity, skew/kurtosis, correlations, cardinality, entropy, imbalance, and a composite difficulty score. This meta-feature vector is the main handoff artifact for downstream components.
4. Meta-learning brain
brain.py is the learned decision layer. It maps dataset DNA to hyperparameters using prior run history stored in knowledge_base.csv. The architecture is explicitly stateful: cold-start heuristics are used when history is sparse, then trained predictions take over once enough elite runs exist. The brain also appends new experiences back into the knowledge base.
5. Algorithm selection boundary
algorithm_recommender.py sits between dataset analysis and training. It turns DNA plus hyperparameter context into a ranked algorithm recommendation and a trial budget. This module is important because it defines the contract for what the engines must support; the tests enforce that every algorithm ID produced here is recognized by the execution layer.
6. Training engines are split by runtime target
There are two execution engines: sklearn_engine.py for deployable classical ML packages and engine.py / engine_stream.py for PyTorch training. The sklearn path is designed to return a serializable package, typically for .joblib export, while the PyTorch path is used for training dynamics and live monitoring. engine_stream.py is the streaming/generator-based variant used when the UI needs epoch-by-epoch updates.
7. Adaptive training loop
engine_stream.py contains the real-time training monitor logic. During training it can detect patterns such as plateauing, overfitting, or vanishing gradients and adjust the training process on the fly. Architecturally, this makes the training engine not just a dumb executor but a feedback-controlled subsystem that influences the optimization loop.
8. Bilevel optimization layer
bilevel.py orchestrates the outer optimization loop. It evaluates candidate configurations, perturbs winners, and feeds results back into the system. The README frames this as a state machine with phases like initialize, tune, and use best. This layer is the bridge between “predict a good config once” and “iteratively improve across generations.”
9. Vizier-compatible local search abstraction
vizier_stub.py and designer_brain.py form the search-management boundary. The stub provides local replacements for concepts like Study, Trial, Measurement, and SearchSpace, while designer_brain.py adapts the meta-learner into a designer/suggester interface. This gives the repo a Vizier-like control plane without depending on Google Cloud Vizier.
10. Persistent feedback loop
knowledge_base.csv is the main persistence artifact and acts as the system’s memory. Every run can append dataset DNA, chosen parameters, and outcome metrics. That history is then reused by the brain for later predictions, making the architecture an online-learning loop rather than a stateless optimizer.
11. Packaging and output artifacts
The system produces deployable model bundles as part of the training path. The sklearn route emits serialized package artifacts, while the PyTorch route works with state dictionaries. The CLI workflow also produces run reports and charts, which makes the repo useful both as an optimizer and as a training/reporting tool.
12. Infrastructure and external dependencies
The main runtime is Python 3.9+ with Streamlit, PyTorch, scikit-learn, and optional XGBoost. W&B is mentioned for experiment tracking, but it does not appear to be a hard architectural dependency in the file tree. The knowledge base is file-based rather than database-backed, so the repo is intentionally low-infrastructure and session-persistent through CSV files.
13. Testing and contract boundaries
The tests/ directory is not incidental; it enforces the repo’s architecture. test_integration.py protects the recommender-to-engine contract, test_vizier_stub.py validates the local study/trial abstraction, test_brain.py checks meta-learner outputs, and test_pipeline.py exercises the end-to-end orchestration. This test structure mirrors the system boundaries and is a good guide for diagramming the architecture accurately.
14. Supporting utilities and dataset generators
car.py and generate_audit_data.py are not core runtime modules, but they support validation and demos by generating realistic and stress-test datasets. verify_metatune.py and verify_evolution.py are verification scripts for system-wide and evolutionary behavior. These sit outside the main optimization pipeline but are useful to show as auxiliary tooling around the core system.
15. Recommended diagram framing
A clear architecture diagram for this repo should show four main layers: input/UI, dataset analysis and recommendation, optimization/training execution, and persistent feedback memory. The most important data artifact flowing through the system is the dataset DNA dictionary, and the most important long-lived boundary is the knowledge_base.csv feedback store that closes the learning loop.
Main entry points: algorithm_recommender.py Recommender · data_analyzer.py Data DNA · bilevel.py Bilevel · brain.py Brain
Components 17
Entry points
User-facing shells for interactive and batch runs.
Analysis & recommendation
Dataset characterization and algorithm selection.
- Data DNA · Loads CSVs, detects target/task, and computes meta-features.
data_analyzer.py - Recommender · Ranks algorithms and sets trial budget from dataset DNA.
algorithm_recommender.py
Search & meta-learning
Stateful decision layers and outer-loop orchestration.
- Brain · Maps dataset DNA to hyperparameters using prior run history.
brain.py - Designer · Adapts the meta-learner to a Vizier-like suggester interface.
designer_brain.py - Bilevel · Coordinates candidate evaluation, mutation, and best-config reuse.
bilevel.py - Vizier stub · Local Study/Trial/SearchSpace abstraction for search control.
vizier_stub.py
Training engines
Classical and PyTorch execution paths.
- Torch engine · PyTorch training runtime for model fitting.
engine.py - Stream engine · Generator-based training with epoch-by-epoch monitoring and adaptation.
engine_stream.py - Sklearn engine · Classical ML execution path that returns serializable artifacts.
sklearn_engine.py
Persistent memory
Run history used for online learning.
- Knowledge base · Persistent CSV memory of dataset DNA, params, and outcomes.
knowledge_base.csv
Tooling & checks
Auxiliary generators and verification scripts.
- Tests · Integration and unit checks that protect subsystem boundaries.
tests - CAR data · Auxiliary dataset/support script for demos and validation.
car.py - Audit data · Creates realistic and stress-test datasets.
generate_audit_data.py - Verify core · System-level verification for end-to-end behavior.
verify_metatune.py - Verify evolution · Checks evolutionary and iterative optimization behavior.
verify_evolution.py
Connections 31
- App → Data DNA: load/analyze · Interactive runs begin with dataset inspection.
- App → Bilevel: optimize · The UI drives the shared optimization loop.
- App → Stream engine: live train · Streamlit consumes epoch-level updates from the streaming engine.
- Pipeline → Data DNA: load/analyze · Batch runs begin with the same dataset analysis stage.
- Pipeline → Bilevel: optimize · CLI execution invokes the outer optimization loop.
- Pipeline → Torch engine: train · Headless runs can use the non-streaming torch trainer.
- Data DNA → Recommender: DNA · Meta-features are handed to algorithm ranking.
- Data DNA → Brain: meta-features · Dataset DNA also informs hyperparameter priors.
- Data DNA → Bilevel: task context · Outer-loop orchestration uses the task and difficulty context.
- Recommender → Designer: suggest · Recommendations are adapted into search suggestions.
- Recommender → Sklearn engine: classical path · Ranks candidates supported by the sklearn execution layer.
- Recommender → Torch engine: torch path · Ranks candidates supported by the torch execution layer.
- Recommender → Stream engine: live path · Streaming training is one of the executable options.
- Designer → Vizier stub: study/trial · The designer uses the local Vizier-like search abstractions.
- Designer → Brain: delegate · Suggestion logic ultimately relies on the meta-learner.
- Bilevel → Recommender: select · The outer loop asks for ranked candidates and budgets.
- Bilevel → Stream engine: evaluate · Candidates are evaluated with adaptive live training.
- Bilevel → Brain: feedback · Evaluation outcomes feed back into the learned decision layer.
- Stream engine → Brain: metrics · Training dynamics and outcomes update meta-learning.
- Torch engine → Knowledge base: results · Training outcomes ultimately become persistent run history.
- Sklearn engine → Knowledge base: package/run · Classical runs contribute results to the persistent memory loop.
- Brain → Knowledge base: append/history · The brain reads and appends to the CSV knowledge base.
- Tests → Data DNA: check · Tests cover analysis behavior and metadata extraction.
- Tests → Recommender: contract · Tests ensure recommender outputs are executable downstream.
- Tests → Brain: check · Tests validate meta-learner predictions and updates.
- Tests → Vizier stub: check · Tests validate the study/trial abstraction boundary.
- Tests → Pipeline: end-to-end · Integration tests exercise the orchestration path.
- CAR data → Audit data: support · Auxiliary dataset support is used for generation and demos.
- Audit data → Data DNA: sample data · Generated data can be analyzed by the core pipeline.
- Verify core → Pipeline: verify · Core verification script exercises the batch workflow.
- Verify evolution → Bilevel: verify · Evolution checks focus on the outer optimization loop.