How aditya-7-7/sales-forecasting-mlops-pipeline works
Project Type & Purpose
• This is a full-stack MLOps platform for Sales Forecasting. It uses Apache Airflow (via Astronomer) to orchestrate an end-to-end ML pipeline, MLflow for experiment tracking, MinIO for artifact storage, ensemble models (XGBoost + LightGBM) for prediction, and a Streamlit-based UI for real-time inference.
File Structure Highlights & Architectural Clues
• Top-Level
• Dockerfile, docker-compose.override.yml → containerized services
• .astro/ & dags/ → Airflow DAG definitions
• / → reusable Python modules (data validation, feature engineering, ML model logic, utilities)
• ui/ → Streamlit inference app + helper scripts
• tests/ → unit tests for DAGs and core logic
• Patterns
• Modular code under include/ suggests clear separation of pipeline stages
• Container per service (Airflow Scheduler/Webserver, MLflow, MinIO, Streamlit)
• CI/testing focused on DAG integrity and core modules
Core Components
A. Orchestration Layer
• Astronomer/Airflow Service (Scheduler, Webserver, Workers)
• DAG: sales_forecasting_training.py
B. Pipeline Modules (Python Tasks invoked by Airflow)
1. Data Generation & Validation
• include/utils/data_generator.py
• include/data_validation/validators.py
2. Feature Engineering
• include/feature_engineering/feature_pipeline.py
3. Model Training & Ensemble
• include/ml_models/train_models.py
• include/ml_models/ensemble_model.py
• include/ml_models/model_visualization.py
4. Experiment Tracking & Artifact Storage
• MLflow Server (tracking, registry)
• MinIO S3 (artifact bucket)
C. Inference & Visualization
• Streamlit App (ui/inference_app.py)
• Model-loading utilities (ui/utils/*.py)
D. Infrastructure & DevOps
• Docker Compose network defining Airflow, MLflow, MinIO, Streamlit
• Configuration files (include/config/ml_config.yaml, .astro/config.yaml)
Interactions & Data Flow
1. Airflow triggers DAG on schedule or manually.
2. Task A uses Data Generator → writes synthetic data.
3. Task B runs Validators → ensures data quality.
4. Task C performs Feature Engineering → outputs feature store.
5. Task D trains XGBoost & LightGBM → logs metrics/artifacts to MLflow & uploads models to MinIO.
6. Task E runs ensemble logic → final model artifact.
7. MLflow UI available for browsing experiments.
8. MinIO Console for raw artifacts.
9. Streamlit UI calls MLflow/MinIO to load models, runs include/feature_engineering logic on input data, shows predictions & plots.
Architectural Patterns & Principles
• Pipeline-as-code: each stage is a standalone Python module invoked by Airflow
• Modular layering: separation of concerns (data, features, training, inference)
• Service isolation via containers (Airflow, MLflow, MinIO, UI)
• Experiment reproducibility: configuration driven, MLflow versioning, S3 storage