How hurryingauto3/ece-gy-9183-mlops-project works
Project type and purpose
• This is a cloud-native, full-stack MLOps system (“AgriYield”) for end-to-end crop yield prediction.
• It ingests large satellite/weather datasets (CropNet), runs ETL, trains deep learning models (LSTM+TCN) on GPU nodes, tracks experiments in MLflow, serves features and predictions via FastAPI microservices, and monitors system and model health with Prometheus/Grafana and Locust. Infrastructure is managed via Terraform, Ansible, K3s (lightweight Kubernetes), and GitOps (ArgoCD).
File structure
• Top-level directories:
• chameleon_devops—Terraform, Ansible, K8s manifests, Helm charts, ArgoCD apps (infrastructure as code + GitOps)
• etl_pipeline—Dockerized extract & transform jobs
• feature_serving—FastAPI service to retrieve pre-computed features
• model_training—PyTorch training scripts, Jupyter notebooks, Ray support
• model_serving—FastAPI microservice for inference with MLflow fallback
• dashboard & tutorials—User-facing Streamlit/Gradio dashboards and example notebooks
• monitoring—Prometheus/Grafana configs, Locust load-test scripts
• mlflow—Compose file for MLflow server and MinIO object store
• Key inf rastructure files: Terraform .tf under chameleon_devops/terraform; Ansible playbooks & roles under chameleon_devops/ansible; K8s manifests under chameleon_devops/k8s; Docker-compose files at root and in subdirs.
Pull insights from the README
• Core components: Data Lake (persistent storage + MinIO), ETL pipeline, Feature Service, Training Platform (GPU node, Ray, PyTorch, MLflow), Model Serving API, Dashboards, Monitoring stack, CI/CD/GitOps.
• Infrastructure: Two VMs on Chameleon Cloud (services_node, gpu_node) joined in a K3s cluster. Floating IPs, block storage, GPU reservation.
• Deployment & automation: Terraform + Ansible for infra; ArgoCD for continuous deployment; Docker images built via CI; canary/staging/production overlays.
Main components & interactions
• Cloud/IaaS layer: Two VMs (services_node, gpu_node) with Attached Block Storage (CropNet data, MLflow artifacts)
• Kubernetes (K3s) cluster: Master + worker nodes on those VMs
• Data services:
• MinIO (S3-compatible) for object storage
• Data Lake (persistent volumes hosting raw CropNet files)
• ETL Pipeline: Dockerized Extract and Transform jobs (etl_pipeline/extract and etl_pipeline/transform)
• Feature Service: FastAPI (feature_serving)
• Model Training:
• PyTorch LSTM+TCN containers (model_training)
• Ray cluster for distributed jobs (optional)
• MLflow server & UI (mlflow/docker-compose)
• Model Registry: MLflow Model Registry
• Model Serving: FastAPI microservice (model_serving) with MLflow loader & dummy fallback
• Dashboard/UI: Streamlit/Gradio (dashboard/app.py)
• CI/CD/GitOps: GitHub → ArgoCD → K8s deployments (chameleon_devops/k8s, charts, manifests)
• Monitoring & Testing:
• Prometheus (scraping K3s, services)
• Grafana dashboards (monitoring/grafana, provisioning files)
• Locust load tests (monitoring/locustfile.py)
1. Offline: CropNet files → ETL extract container → transformed features → Data Lake / MinIO
2. Feature Service queries Data Lake / MinIO → returns feature vectors
3. Model Training containers pull features via Feature Service or directly from Data Lake → run training on GPU node → log metrics/artifacts to MLflow/MinIO
4. Upon promotion (via promote_model.py or CI), trained model version moves to “Production” in MLflow registry
5. Model Serving API pulls production model from MLflow registry on startup/inference → serves /predict endpoints
6. Dashboard UI calls Feature Service and Model Serving API to display predictions vs. truth
7. Monitoring: Prometheus scrapes metrics from ETL pods, Feature Service, Model Serving, Ray, MLflow → Grafana visualizes; Locust periodically sends traffic to /predict and /features
8. CI/CD: GitHub updates → ArgoCD syncs to K8s cluster → Kubernetes applies new Docker images / manifests in staging → canary → production
c. Architectural patterns & design principles:
• Microservices (ETL, Feature, Training, Serving, Dashboard each in its own container)
• Containerization + Kubernetes for orchestration (K3s)
• GitOps (ArgoCD) + IaC (Terraform, Ansible)
• Model registry and versioning (MLflow)
• Infrastructure layering: IaaS → Kubernetes → Services
• Resiliency with dummy fallback for model serving; separate staging/canary/production overlays
d. Technologies & framework callouts:
• Terraform, Ansible, K3s, ArgoCD, Helm
• Docker / Docker-Compose
• Python, FastAPI, PyTorch, Ray
• MLflow, MinIO
• Prometheus, Grafana, Locust
• Streamlit/Gradio for dashboard
Detail level
• Major microservices and their interactions
• Infrastructure provisioning paths (Terraform/Ansible → VMs → K3s cluster)
• MLflow registry transitions (None → Staging → Production)
• Canary/staging/production overlays in GitOps loop
• Don’t clutter with every file—aggregate by functional directory