How hdegen/mlops-end-to-end works
Project type and purpose
• This is a full-stack MLOps template for building, tracking, and deploying machine-learning pipelines on AWS.
• It standardizes project structure, orchestrates data→feature→model steps via MLflow, DVC/MLflow data versioning, containerizes with Docker, and schedules via Apache Airflow on cloud compute.
File structure
• Top-level:
• dag.py – example Airflow DAG
• Dockerfile, build/run scripts – containerization
• data/, notebooks/ – raw data placeholders and exploratory notebooks
• src/ – source code subdivided into data, features, models, lib, examples, plus MLproject, conda.yaml, main.py
• src/data: data ingestion and cleaning (SQL queries, S3 reads)
• src/features: feature engineering logic
• src/models: model training and prediction entry points
• src/lib: utility functions and AWS/MLflow helper code
• src/examples: CLI usage examples
• src/main.py: orchestrates MLflow steps using Click for parameter parsing
Extract key architectural insights from the README
• Uses MLflow to define a reproducible pipeline (MLproject file) with steps: make_data → build_features → train_model → predict
• Tracks parameters, metrics, and artifacts (including versioned data) via MLflow backed by an AWS-hosted database and S3 artifact store
• Containerization: builds a conda-based Docker image, pushes to AWS ECR via CI/CD (e.g., Jenkins)
• Scheduling: defines an Airflow DAG to run each pipeline step on cloud resources (EC2/EKS, CPU/GPU)
• Optional DVC for data versioning—though MLflow is used here to store data snapshots in S3
1. Local Development Workstation (IDE, Conda env)
2. CI/CD Pipeline (Jenkins/GitHub Actions)
3. Docker Registry (AWS ECR)
4. Orchestration Layer (Apache Airflow on EKS or EC2)
5. Compute Tasks (container-based jobs for data, features, training, predict)
6. MLflow Tracking Server (backed by RDS/PostgreSQL)
7. Artifact Store (AWS S3 buckets for raw data, processed data, models)
8. Metadata Store (MLflow metadata database)
9. External Services (AWS S3, Athena, other AWS APIs)
• Developer → commit code → CI/CD → build Docker image → push to ECR
• Airflow scheduler → pull latest image from ECR → spin up tasks for each pipeline step
• Pipeline tasks → read raw data from S3 (or query Athena) → write cleaned data/features back to S3
• Training task → log params/metrics/models to MLflow Tracking Server → store artifacts in S3
• Predict task → retrieve model from MLflow registry → write predictions to S3 or downstream system
c. Architectural patterns and principles:
• Modular pipeline steps (decoupled via MLproject entry points)
• Containerization for reproducibility and portability
• Orchestration via DAG for flexible scheduling and scaling
• Separation of concerns: code (src/) vs. data (S3) vs. compute (containers) vs. metadata (RDS)
• Infrastructure as code/scripts: Dockerfile, build/run scripts, conda.yaml
d. Technologies, frameworks, libraries:
• Python, Click for CLI, MLflow for experiment tracking and model registry
• Docker + AWS ECR for container images
• Apache Airflow on AWS (EC2/EKS) for scheduling
• AWS S3 for data and model artifact storage
• AWS RDS/PostgreSQL for MLflow backend store
• (Optional) DVC for data versioning