How vucongtuanduong/heart-disease-prediction-mlops works
Project Type & Purpose
• The project is a data science and machine learning application focused on predicting heart disease. It involves data ingestion, EDA, model training and evaluation, and both batch and web-service model deployments.
• The README emphasizes a full ML lifecycle including data preprocessing, model training (using libraries like scikit-learn, xgboost, catboost), deployment (batch and real-time), orchestration with Docker and MLflow, and monitoring using Evidently.
Analyzing the File Structure
• Top-level directories indicate clear segregation of concerns:
• "data": Contains data files and scripts for generating random test data.
• "best-practices": Contains code for testing, building, and automated scripts with Makefiles.
• "deployment": Separates the batch deployment and web-service (Flask/Gunicorn) deployment, each with their own Dockerfiles and configuration (Pipfile, requirements.txt) along with serialized model artifacts.
• "model": Stores the serialized artifacts (rf_model.pkl, scaler.pkl, dict_vectorizer.pkl) produced by the training process.
• "notebooks": Provides an interactive environment for EDA and model development.
• "orchestration": Contains scripts, Docker configuration, and orchestrates workflows for data ingestion, transformation, and model training using pipelines, MLFlow integration, and custom scripts.
• "monitoring": Contains everything related to monitoring the deployed model's performance, including Docker-compose files and notebooks for Evidently reports.
• Additional directories and files (e.g., .github/workflows, code best practices tests, and orchestration charts) confirm a comprehensive, multi-component architecture.
Main Components to Identify
a. Data & Preprocessing:
• Data ingestion scripts (in orchestration, data_loaders/ingest.py, and data/create_random_test_data.py).
• Notebooks for exploratory data analysis.
b. Model Training & Lifecycle Management:
• Components in "best-practices/code" and "orchestration/heart-disease-prediction" (for training, pipelines, and transformers).
• Use of MLflow (check orchestration/mlflow and MLFlow-related files) to manage experiments, tracking and model registry.
c. Model Artifact Storage:
• The "model" directory holds the final model, scaler, and vectorizer artifacts.
d. Model Deployment:
• Batch deployment: Under "deployment/batch" where a Docker container is used to perform batch predictions.
• Real-time web service: Under "deployment/web-service" with a Flask/Gunicorn setup to serve predictions.
e. Workflow Orchestration:
• The "orchestration" directory contains Docker-compose, custom scripts, and pipelines that orchestrate model training and data transformation.
f. Model Monitoring:
• The "monitoring" directory shows the integration of Evidently (or similar monitoring tools) to review performance metrics of the deployed model.
g. CI/CD and Best Practices:
• GitHub workflows for CI (in .github/workflows) and testing scripts within best-practices code.
Relationships & Data Flow
• Data originates from the "data" directory then flows into preprocessing (notebooks and orchestration/data_loaders), where it may be transformed by the pipelines (orchestration/heart-disease-prediction/pipelines).
• The transformed data is used in model training (best-practices and orchestration training scripts). The trained model and related artifacts are stored in the "model" directory.
• Deployment components (batch and web-service) load these artifacts to serve predictions.
• The orchestration layer integrates model retraining with the ML lifecycle tooling (MLflow) and passes new or updated models to the deployment workflows.
• Monitoring components receive inputs from the deployed services, analyze predictions, and generate performance reports.
Architectural Patterns & Technologies
• Microservices/Containerized Architecture: Each deployment scenario (batch, web-service, orchestration, monitoring) has its own Docker environment and configurations.
• Pipeline/Workflow Orchestration: Clear use of pipelines (data ingestion, preparation, and transformation), as seen in orchestration/heart-disease-prediction.
• ML Lifecycle Management: Integration with MLflow for experiment tracking and model registry.
• CI/CD Pipeline: GitHub actions configured under .github/workflows for continuous integration and testing.
Specific Elements
• Data Source (data directory and Kaggle dataset)
• EDA/Preprocessing (notebooks, orchestration/data_loaders)
• Model Training (best-practices, orchestration/train scripts)
• Model Artifact Repository (model directory)
• Batch Deployment Service (deployment/batch) and associated Docker container
• Real-Time Prediction Service (deployment/web-service) with Flask/Gunicorn
• Orchestration Framework (orchestration directory’s components, pipelines, MLflow integrated container)
• Monitoring Dashboard (monitoring directory and reports)
• Data flow from ingestion → preprocessing → training
• Model artifact movement from training to deployment
• Feedback loop from monitoring back to retraining or model update processes
• Optionally, note external integrations like Docker, MLflow, and Evidently in a separate annotation block.
Happy diagramming!