How shivamshinde123/battery-features-predictions works
Main Components
a. Data Storage & Versioning
• Raw data (“Data/” directory, managed by DVC and remote storage)
• Processed datasets in DVC cache (".dvc/")
b. Pipeline Orchestration
• DVC pipeline (dvc.yaml, dvc.lock, params.yaml)
• Steps: loader → validation → preprocessing → model training → evaluation
c. Core ML Modules (src/)
• loader.py (data ingestion)
• validation.py (data sanity checks)
• preprocessing.py (feature engineering)
• model_training_xgboost.py & model_training_nn.py
• utils.py (shared helpers)
d. Experiment Tracking & Logging
• MLflow (mlruns/ directory)
• Logs/ (training logs)
• Metrics/ (JSON metrics files)
e. Model Artifacts
• Models/ (serialized model files)
f. Serving Layer
• src/frontend/webapp.py (Flask/FastAPI app exposing prediction API)
g. Notebooks & Exploratory Analysis
• Notebooks/ (Jupyter notebooks)
h. Continuous Integration
• GitHub Actions CI (".github/workflows/ci.yml")
• pytest tests (src/tests/)
Relationships & Data Flow
• Raw Data → DVC fetch → loader → validation → preprocessing → model training → evaluation
• model_training_xgboost & model_training_nn output → Models/
• Evaluator → writes performance metrics → Metrics/ → MLflow logging
• MLflow client in scripts logs parameters & metrics → mlruns/
• Frontend app loads latest model from Models/ to serve predictions
• CI pipeline runs:
• dvc repro (validates pipeline reproducibility)
• pytest src/tests (ensures code correctness)
Architectural Patterns & Design Principles
• Pipeline-as-Code: DVC defines a reproducible DAG of data-processing steps
• Modular Codebase: Single-responsibility modules (loader, validation, preprocessing, training)
• MLOps Best Practices: Experiment tracking (MLflow), version control for data (DVC), automated testing
• Separation of Concerns: Data pipeline vs. model serving (frontend)
Key Technologies & Frameworks
• Python 3.10
• DVC (pipeline orchestration & data versioning)
• MLflow (experiment tracking)
• XGBoost, PyTorch/Keras (neural nets)
• Flask or FastAPI (webapp)
• pytest (testing)
• GitHub Actions (CI)
Details
• Params.yaml as input to both training modules