How vinayakmane47/nasa-turbofan-ml-project-aiops works
Project Type and Purpose
• Full-stack MLOps application.
• Purpose: Ingest turbofan engine data, validate and transform it, train multiple ML models, select the best, persist it, and expose prediction and monitoring via a Flask web UI.
Main Architectural Layers and Components
A. Presentation Layer (Flask Web Front-end)
• app.py: Flask routes/controllers
• templates/*.html: Jinja2 views (index, train, predict, logs, experiment history)
• Static assets (if any)
B. Application/MLOps Layer
• Turbofan.pipeline.Pipeline: Orchestrates ingestion → validation → transformation → training → evaluation → model pusher
• Turbofan.component.*:
• data_ingestion.py
• data_validation.py
• data_transformation.py
• model_trainer.py
• model_evaluation.py
• model_pusher.py
• Turbofan.entity: Configuration and artifact definitions (config_entity, artifact_entity, experiment, model_factory, Turbofan_predictor)
• Turbofan.config.configuration: Loads YAML configs (config/schema.yaml, config/model.yaml, config/config.yaml)
• Turbofan.util.util: Shared utility functions
• Turbofan.logger: Central logging
C. Data Layer
• Raw/Processed Data: Proccessed_data/*.csv
• Artifacts (auto-generated during pipeline runs): feature stores, train/test datasets
• saved_models/<timestamp>/model.pkl: Persisted best model
D. Infrastructure/DevOps
• Dockerfile / .dockerignore: Containerization
• .github/workflows/main.yaml: CI/CD (test, build, deploy)
• setup.py / requirements.txt: Packaging and dependencies
• Heroku (production hosting)
Data and Control Flow
1. User (browser) → HTTP request → Flask app (app.py).
2. For “Train” action, Flask invokes Turbofan.pipeline.Pipeline.run_pipeline().
3. Pipeline stages execute in order:
a. Data Ingestion: Read CSVs → raw artifact directory
b. Data Validation: Schema checks (schema.yaml)
c. Data Transformation: Feature engineering → feature store
d. Model Trainer: Fit multiple models, use model_factory for instantiation
e. Model Evaluation: Compare metrics (r2_score_train/test)
f. Model Pusher: Save best model to saved_models/<timestamp>
4. Pipeline writes logs and experiment metadata (experiments.csv or DB)
5. For “Predict” action, Flask loads Turbofan_predictor (entity) which loads saved model, processes user input, returns RUL
6. Flask renders templates showing results or experiment history
Architectural Patterns and Principles
• Layered (Presentation / Application / Data).
• Modular MLOps pipeline: each component has single responsibility.
• Configuration-driven design: behavior controlled via YAML files.
• Entity-driven: clear separation of data structures (entities) vs. logic (components).
• Log-and-monitor: Central logger, experiment history UI.
Technologies and Frameworks
• Python 3.7+, Flask (app.py)
• Jinja2 templates
• Pandas, NumPy, scikit-learn (ML tasks)
• YAML (configuration)
• Docker, GitHub Actions (CI/CD)
• Heroku (hosting)