How saad1912/vehicle_insurance_project works
Project Type & Purpose
• The project is an end-to-end machine learning pipeline focused on vehicle insurance data. It covers data ingestion from MongoDB Atlas, data validation & transformation, model training/evaluation, AWS integration for model management (S3), and even a prediction web interface. Additionally, there’s comprehensive CI/CD automation using Docker and GitHub Actions.
• Key technologies include Python, Docker, MongoDB Atlas, AWS (S3, IAM and possibly other services), and GitHub Actions.
Analyzing the File Structure
• Top-level files such as app.py, Dockerfile, README.md, and setup.py indicate an application entry point and containerization.
• The “src” directory is organized into subdirectories that clearly segment the architecture:
• “cloud_storage” for AWS interactions,
• “components” for pipeline steps (e.g., data_ingestion, data_transformation, model_trainer, model_evaluation, model_pusher),
• “configuration” for defining cloud and database connections,
• “data_access” for handling database connectivity (MongoDB),
• “entity” for classes representing configurations, artifacts, and model descriptors,
• “pipeline” for orchestrating the prediction and training pipelines,
• “utils” for miscellaneous helper functions.
• There are separate directories for static files and HTML templates, indicating the existence of a frontend part for the prediction pipeline.
Key Components
a. Data Engineering & Storage:
• Data Access Layer: Component (src/data_access) that connects the application to MongoDB.
b. Machine Learning Pipeline:
• Data Ingestion Component: Responsible for fetching data from MongoDB into the pipeline.
• Data Validation & Transformation Components: Handle schema validation (based on config/schema.yaml) and transforming data before training.
• Model Trainer: Uses the transformed data to create a machine learning model.
• Model Evaluation: Evaluates performance against a threshold to check if the new model supersedes the current one.
• Model Pusher: Uploads the approved, trained model to storage – specifically interacting with AWS S3.
c. AWS Integration:
• AWS S3 Storage: Used for model registry and management.
• AWS Connection/Configuration: Modules that manage AWS credentials and S3 bucket interactions (e.g., in src/cloud_storage/aws_storage.py and entity/s3_estimator.py).
d. Prediction Pipeline & Frontend:
• Web Application (app.py): Acts as an entry point for prediction requests.
• Static Assets & Templates: “static” folder for CSS and “templates” folder for HTML views.
e. DevOps & CI/CD:
• Docker: Containerization elements include Dockerfile and .dockerignore to package the application.
• GitHub Actions: CI/CD pipelines (under .github/workflows) automatically build, test, and deploy the application.
• EC2 Deployment & GitHub Runner: Deployment components for the live environment.
f. Utilities & Support:
• Logging & Exception Handling: Custom logger modules and exception processing (in src/logger and src/exception).
• Configuration Files: YAML files and configuration modules that centralize settings for the pipeline, DB, and AWS.
Relationships & Data Flow
• Data then flows from Ingestion to Validation and Transformation.
• From there, the pipeline splits into training and evaluation stages.
• After evaluation, a successful model is pushed to AWS S3 via the Model Pusher.
• On the front end, the Prediction Pipeline (app.py) retrieves models and handles user requests for predictions.
• An outer loop for CI/CD shows automated build and deployment (Docker + GitHub Actions) that touches the entire pipeline.
• Additionally, auxiliary elements like logging and exception handling loosely tie into every component for error monitoring.
Architectural Patterns & Design Principles
• Layered/Microservices-Like Decomposition: The project is modularized heavily into pipeline stages.
• Separation of Concerns: Each directory (components, configuration, entity, etc.) encapsulates a specific responsibility.
• Cloud-Native & Containerized: With Docker and AWS integration, the system design supports scalable and deployable microservices.
• CI/CD Integration: Ensures automated testing and deployment practices are integrated into the architecture.