How arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments works
Project type and purpose
• This is an end-to-end machine learning application for chicken disease classification.
• It includes: an MLOps pipeline (built with DVC), experiment tracking (MLflow + DagsHub), model training/inference code (Python modules under src/cnnClassifier), a simple Flask-based web UI (app.py + templates/index.html), and deployment automation (GitHub Actions → AWS ECR → EC2).
File structure
• Top‐level:
• config/ (YAML files for pipeline & params)
• dvc.yaml + params.yaml (defines DVC pipeline)
• main.py (CLI entry point to orchestrate pipelines)
• app.py + templates/index.html (Flask app for serving predictions)
• .github/workflows/cicd.yaml (CI/CD for building/pushing Docker & deploying to EC2)
• src/cnnClassifier/… (core components: data_ingestion, prepare_base_model, model_trainer, evaluation; configuration manager, entities, logger, pipelines)
• Research notebooks under research/ (exploratory steps, not part of prod system)
• setup.py, requirements.txt (packaging)
README for insights
• Describes the 4 major MLOps tools: DVC (for pipelines + data versioning), MLflow (experiment tracking), DAGsHub remote, GitHub Actions + AWS (CICD and deployment).
• Outlines the workflow: update configs → implement components → run dvc repro → monitor MLflow → push to production via GitHub Actions → EC2 self-hosted runner.
a. Main components:
1. Data Layer
• Raw data repository (e.g., local file system or remote storage tracked by DVC)
• DVC pipeline stages: data_ingestion → prepare_base_model → model_trainer → evaluation
2. Model Registry & Experiment Tracking
• MLflow server (remote MLFLOW_TRACKING_URI on DagsHub)
• MLflow UI (for metrics, parameters, artifacts)
3. Application Layer
• Flask Web App (app.py + index.html) exposing a /predict API
• Prediction pipeline (src/cnnClassifier/pipeline/predict.py) loading the trained model artifact and returning inference
4. CI/CD & Deployment
• GitHub Actions workflow (cicd.yaml)
• Docker build & push to AWS ECR
• EC2 instance (self-hosted runner) pulling the Docker image and running the container
b. Relationships & interactions:
• Developer commits → GitHub Actions triggers → builds Docker image → pushes to ECR → EC2 pulls image → runs container → exposes web app.
• During development, running “dvc repro” executes sequential pipeline stages; each stage logs metrics/artifacts to MLflow.
• Flask app loads the latest model artifact (from local volume or DVC cache) to serve predictions.
c. Architectural patterns & principles:
• Pipeline orchestration (DVC as a lightweight DAG orchestrator)
• Configuration-driven design (YAML + config manager)
• Separation of concerns: data ingestion, model prep, training, evaluation, inference
• MLOps integration: experiment tracking decoupled from compute; CI/CD automates build & deploy
d. Key technologies & libraries:
• Python 3.x, DVC, MLflow, Flask, GitHub Actions, AWS ECR/EC2, PyYAML, TensorFlow/Keras or PyTorch (implied by transfer learning), Docker