How codewithkaran-21/mlops-vehicle-insurance works
Project Type and Purpose
• This is a full-stack, end-to-end MLOps application for Vehicle Insurance Claim Prediction.
• It orchestrates data ingestion from MongoDB Atlas, ML pipeline steps (ingestion, validation, transformation, training, evaluation, pushing), model registry on AWS S3, and deployment via Docker on EC2 with CI/CD.
File Structure
• Top-level scripts: app.py (Flask API), demo.py, notebook folder (exploratory work).
• config/ model.yaml & schema.yaml define pipeline settings and data schemas.
• src/ contains:
• cloud_storage (AWS interactions)
• components (pipeline stages)
• configuration (connections to AWS & MongoDB)
• data_access (raw data fetchers)
• entity (data structures and artifacts)
• pipeline (orchestrators: training_pipeline, prediction_pipeline)
• utils, logger, exception for shared helpers and logging.
• Dockerfile, .github/workflows/aws.yaml, and setup files support containerization and CI/CD.
Insights from README
• Five logical layers: Data Layer, Pipeline Layer, Entity Layer, Utility & Logging Layer, Application Layer.
• Uses MongoDB Atlas, AWS S3, EC2/ECR, GitHub Actions, Docker, Flask.
• Emphasizes modular, scalable, production-grade MLOps design.
a. Main Components
• Data Layer: MongoDB Atlas cluster → Data Access Connector (mongo_db_connection.py)
• Pipeline Layer:
– Data Ingestion (json→csv, train/test split)
– Data Validation (schema checks)
– Data Transformation (encoding, scaling)
– Model Training & Selection
– Model Evaluation & Promotion
– Model Pusher (uploads to S3)
• Entity Layer: config/entity classes for artifacts and estimators
• Utility & Logging Layer: shared utilities, logger, exception
• Cloud Storage: AWS S3 bucket for model registry
• Application Layer: Flask service (app.py) exposing /train and /predict
• CI/CD & Deployment: GitHub Actions ➔ build Docker image ➔ push to ECR ➔ deploy to EC2
b. Relationships & Interactions
• Flask API invoking Prediction Pipeline, reading model from S3/local.
c. Architectural Patterns & Principles
• Layered architecture (Data, Pipeline, Entity, Utility, Application).
• Modular pipeline components with artifact hand-off.
• Separation of concerns: configuration, data access, business logic, infrastructure.
• Infrastructure as Code: Dockerfile and GitHub Actions workflow.
d. Key Technologies
• Python 3.10, Flask
• MongoDB Atlas
• Scikit-learn, Pandas, NumPy
• AWS S3, EC2, ECR, IAM
• Docker
• GitHub Actions CI/CD
Essential Elements
• Component labels matching code modules (e.g., src/components/data_validation).
• Artifact storage (local filesystem) between pipeline stages.
• External connections (environment variables for MongoDB URL, AWS credentials).