How pankaj901/sensor-fault-detection-1 works
Project Type and Purpose
• This is a Python-based ML back-end service that trains and serves a binary classification model for APS (Air Pressure System) fault detection.
• It exposes HTTP endpoints via FastAPI for triggering training (/train) and serving predictions (/predict).
• It persists raw data, model artifacts, and metadata to AWS S3 and MongoDB Atlas, and is containerized and deployed on AWS EC2/ECR via Docker, with infra managed by Terraform and CI/CD via GitHub Actions.
Main Components
a. Data Sources
• APS failure CSV files (local & S3).
• External AWS S3 buckets for raw data and model artifacts.
b. ML Pipeline Modules (under sensor/components)
• Data Ingestion
• Data Validation
• Data Transformation
• Model Trainer
• Model Evaluator
• Model Pusher
c. Data Access & Storage
• sensor/data_access for reading raw data.
• MongoDB Atlas (via sensor/configuration/mongo_db_connection) for logging pipeline metadata (experiments, metrics).
• AWS S3 (via sensor/cloud_storage/s3_syncer) for storing/retrieving dataset and artifacts.
d. API Layer
• FastAPI app (main.py/app.py) serving two endpoints: /train and /predict.
• Internally invokes the training pipeline or model loading for inference.
e. DevOps & Deployment
• Dockerfile for containerizing the app.
• AWS ECR for Docker registry.
• AWS EC2 for running the container.
• GitHub Actions workflows (/.github/workflows/main.yml) for CI/CD.
• Terraform scripts (not shown) for infra provisioning.
Relationships and Data Flows
• Raw data ingestion:
APS CSV (local or S3) → Data Ingestion → Data Validation → Data Transformation.
• Training:
Transformed data → Model Trainer → Model Evaluator → Model Pusher → persist model to S3 + log metadata to MongoDB.
• Inference:
Client → FastAPI /predict → load model from S3 or local cache → return prediction.
• Control plane:
Client → FastAPI /train → kicks off pipeline modules in sequence.
• DevOps:
Commit → GitHub Actions → build Docker image → push to ECR → Terraform deploy EC2 + run container.
Architectural Patterns and Principles
• Modular “pipeline” design: each step is a separate component, enabling reusability and testability.
• Clean separation of concerns:
• sensor/components (business logic)
• sensor/data_access (I/O)
• sensor/configuration (env and connections)
• sensor/utils & sensor/constant (shared utilities and constants)
• Infrastructure as code (Terraform) and CI/CD (GitHub Actions) for reproducible deployments.
• Twelve-factor app principles: config via env vars, stateless containers, backing services as attached resources.
Final Checklist
• Clear, human-readable labels on every component.
• Annotations for important environment variables and external dependencies.