# pankaj901/sensor-fault-detection-1 architecture

> Architecture diagram of the GitHub repository pankaj901/sensor-fault-detection-1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pankaj901/sensor-fault-detection-1
- Repository: https://github.com/pankaj901/sensor-fault-detection-1

## Overview

##### Project Type and Purpose
- This is a Python-based ML back-end service that trains and serves a binary classification model for APS (Air Pressure System) fault detection.
- It exposes HTTP endpoints via FastAPI for triggering training (/train) and serving predictions (/predict).
- It persists raw data, model artifacts, and metadata to AWS S3 and MongoDB Atlas, and is containerized and deployed on AWS EC2/ECR via Docker, with infra managed by Terraform and CI/CD via GitHub Actions.

##### Main Components
a. Data Sources
- APS failure CSV files (local & S3).
- External AWS S3 buckets for raw data and model artifacts.
b. ML Pipeline Modules (under sensor/components)
- Data Ingestion
- Data Validation
- Data Transformation
- Model Trainer
- Model Evaluator
- Model Pusher
c. Data Access & Storage
- sensor/data_access for reading raw data.
- MongoDB Atlas (via sensor/configuration/mongo_db_connection) for logging pipeline metadata (experiments, metrics).
- AWS S3 (via sensor/cloud_storage/s3_syncer) for storing/retrieving dataset and artifacts.
d. API Layer
- FastAPI app (main.py/app.py) serving two endpoints: /train and /predict.
- Internally invokes the training pipeline or model loading for inference.
e. DevOps & Deployment
- Dockerfile for containerizing the app.
- AWS ECR for Docker registry.
- AWS EC2 for running the container.
- GitHub Actions workflows (/.github/workflows/main.yml) for CI/CD.
- Terraform scripts (not shown) for infra provisioning.

##### Relationships and Data Flows
- Raw data ingestion:
APS CSV (local or S3) → Data Ingestion → Data Validation → Data Transformation.
- Training:
Transformed data → Model Trainer → Model Evaluator → Model Pusher → persist model to S3 + log metadata to MongoDB.
- Inference:
Client → FastAPI /predict → load model from S3 or local cache → return prediction.
- Control plane:
Client → FastAPI /train → kicks off pipeline modules in sequence.
- DevOps:
Commit → GitHub Actions → build Docker image → push to ECR → Terraform deploy EC2 + run container.

##### Architectural Patterns and Principles
- Modular “pipeline” design: each step is a separate component, enabling reusability and testability.
- Clean separation of concerns:
- sensor/components (business logic)
- sensor/data_access (I/O)
- sensor/configuration (env and connections)
- sensor/utils & sensor/constant (shared utilities and constants)
- Infrastructure as code (Terraform) and CI/CD (GitHub Actions) for reproducible deployments.
- Twelve-factor app principles: config via env vars, stateless containers, backing services as attached resources.

##### Final Checklist
- Clear, human-readable labels on every component.
- Annotations for important environment variables and external dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Title
    title["Sensor Fault Detection System Architecture"]

    %% External Data Sources
    subgraph "Data Sources" 
        direction TB
        LocalCSV["Local CSV\naps_failure_training_set1.csv"]:::external
        S3RawBucket["AWS S3 Raw Data"]:::external
    end

    %% Docker Container Boundary
    subgraph "Docker Container" 
        direction TB
        subgraph "ML Pipeline Modules" 
            direction TB
            DataIngestion["Data Ingestion"]:::compute
            DataValidation["Data Validation"]:::compute
            DataTransformation["Data Transformation"]:::compute
            ModelTrainer["Model Trainer"]:::compute
            ModelEvaluator["Model Evaluator"]:::compute
            ModelPusher["Model Pusher"]:::compute
        end
        TrainingPipeline["Training Pipeline\n(sensor/pipeline/training_pipeline.py)"]:::compute
        API["FastAPI Service\n(main.py)"]:::compute
        Utils["Utilities\n(sensor/utils/main_utils.py)"]:::compute
        Constants["Environment & Application Constants"]:::storage
    end

    %% Data Access & Config
    DataAccess["Raw Data I/O\n(sensor/data_access/sensor_data.py)"]:::compute
    MongoConfig["MongoDB Connection\n(sensor/configuration/mongo_db_connection.py)"]:::storage
    S3Syncer["S3 Syncer\n(sensor/cloud_storage/s3_syncer.py)"]:::storage
    S3BucketConst["S3 Bucket Constants\n(sensor/constant/s3_bucket.py)"]:::storage
    EnvConsts["Env & DB Constants"]:::storage

    %% Storage Systems
    subgraph "Storage Systems"
        direction TB
        S3Models["AWS S3 Models"]:::storage
        MongoDBAtlas["MongoDB Atlas"]:::storage
    end

    %% DevOps & Infra
    subgraph "DevOps & Deployment"
        direction TB
        GitHubActions["GitHub Actions\n(.github/workflows/main.yml)"]:::infra
        Docker["Dockerfile"]:::infra
        ECR["AWS ECR"]:::infra
        Terraform["Terraform"]:::infra
        EC2["AWS EC2"]:::infra
        Requirements["requirements.txt\nsetup.py"]:::infra
        Schema["Config Schema\n(config/schema.yaml)"]:::infra
    end

    %% Legends
    subgraph "Legend"
        direction TB
        compute["Compute Modules"]:::compute
        storage["Storage & Config"]:::storage
        infra["Infra & DevOps"]:::infra
        external["External Sources"]:::external
    end

    %% Data Flow Relationships
    LocalCSV -->|upload raw CSV| DataIngestion
    S3RawBucket -->|fetch raw CSV| DataIngestion
    DataIngestion -->|produce raw data| DataValidation
    DataValidation -->|produce validated data| DataTransformation
    DataTransformation -->|produce features| TrainingPipeline
    TrainingPipeline -->|invoke Trainer| ModelTrainer
    ModelTrainer -->|train model| ModelEvaluator
    ModelEvaluator -->|evaluate metrics| ModelPusher
    ModelPusher -->|push model| S3Models
    ModelPusher -->|log metadata| MongoDBAtlas

    %% API Flow
    Client["Client"]:::external
    Client -->|HTTP POST /train| API
    API -->|call Training Pipeline| TrainingPipeline
    Client -->|HTTP POST /predict| API
    API -->|load model| S3Models
    API -->|return prediction| Client

    %% Data Access & Config usage
    DataIngestion -->|read/write data| DataAccess
    DataIngestion -->|sync data| S3Syncer
    ModelPusher -->|use bucket constants| S3BucketConst
    TrainingPipeline -->|use utils| Utils
    TrainingPipeline -->|use constants| Constants
    TrainingPipeline -->|log to| MongoConfig

    %% DevOps Flow
    GitHubActions -->|CI build| Docker
    GitHubActions -->|run tests| Requirements
    GitHubActions -->|provision infra| Terraform
    Docker -->|build image| ECR
    Terraform -->|deploy instance| EC2
    EC2 -->|run container| Docker

    %% Click Events
    click LocalCSV "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/aps_failure_training_set1.csv"
    click S3Syncer "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/cloud_storage/s3_syncer.py"
    click DataIngestion "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/components/data_ingestion.py"
    click DataValidation "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/components/data_validation.py"
    click DataTransformation "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/components/data_transformation.py"
    click ModelTrainer "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/components/model_trainer.py"
    click ModelEvaluator "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/components/model_evaluation.py"
    click ModelPusher "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/components/model_pusher.py"
    click DataAccess "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/data_access/sensor_data.py"
    click MongoConfig "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/configuration/mongo_db_connection.py"
    click S3BucketConst "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/constant/s3_bucket.py"
    click EnvConsts "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/constant/env_variable.py"
    click EnvConsts "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/constant/application.py"
    click EnvConsts "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/constant/database.py"
    click TrainingPipeline "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/pipeline/training_pipeline.py"
    click Utils "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/sensor/utils/main_utils.py"
    click API "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/main.py"
    click Docker "https://github.com/pankaj901/sensor-fault-detection-1/tree/main/Dockerfile"
    click GitHubActions "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/.github/workflows/main.yml"
    click Requirements "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/requirements.txt"
    click Requirements "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/setup.py"
    click Schema "https://github.com/pankaj901/sensor-fault-detection-1/blob/main/config/schema.yaml"

    %% Styles
    classDef compute fill:#DAE8FC,stroke:#6C8EBF,color:#000;
    classDef storage fill:#D5E8D4,stroke:#82B366,color:#000;
    classDef infra fill:#FFE6CC,stroke:#D79B00,color:#000;
    classDef external fill:#F2F2F2,stroke:#A6A6A6,color:#000;
```
