# ishanpandey15/datascienceproject architecture

> Architecture diagram of the GitHub repository ishanpandey15/datascienceproject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ishanpandey15/datascienceproject
- Repository: https://github.com/ishanpandey15/datascienceproject

## Overview

##### Project type and purpose
- This is an end-to-end data science application with two main “faces”:
- A modular ML pipeline (data ingestion → validation → transformation → training → evaluation → prediction)
- A lightweight web UI (Flask) that serves model predictions
- The README confirms that it’s a full-stack data science project with notebooks, config files, and an app entrypoint.

##### File structure
- Top-level:
- config/config.yaml, params.yaml, schema.yaml → central configuration and schema definitions
- research/*.ipynb → exploratory work, pipeline prototyping
- src/datascience/components → individual processing stages
- src/datascience/pipeline → orchestrated pipelines per stage plus prediction
- src/datascience/config, entity, constants, utils → support modules (config loader, DTOs, constants, helpers)
- main.py → command-line orchestrator for the full ML pipeline
- app.py & templates/ → Flask-based web front end for live predictions
- Dockerfile, setup.py, requirements.txt → packaging and containerization
- .github/workflows → CI/CD definitions (likely auto-run tests, lint, build, deploy)

##### README for additional insights
- Clear 5-stage workflow: ingestion, validation, transformation, trainer, evaluation
- MLflow & DagsHub mentioned for experiment tracking
- Steps emphasize updating config, schema, params, entities, components, pipeline, and main.py

1. Data Sources & Config
- Raw data (CSV, database, external API)
- config/config.yaml, params.yaml, schema.yaml
2. Configuration Manager
- src/datascience/config/configuration.py → reads YAMLs and provides settings
3. Pipeline Orchestrator
- main.py → kicks off the full end-to-end workflow
4. Component Modules
- Data Ingestion
- Data Validation
- Data Transformation
- Model Trainer
- Model Evaluator
- Prediction Pipeline (used by app.py)
5. Model Artifact Store & Tracking
- MLflow server (or local folder)
- DagsHub remote repo
6. Web UI
- Flask app (app.py)
- Jinja2 templates (templates/index.html, templates/results.html)
7. External Services
- MLflow Tracking Server
- DagsHub
b. Relationships and interactions
- Trainer writes model artifacts + metrics to MLflow / DagsHub
- Evaluator reads artifacts, logs evaluation to tracking server
- Prediction Pipeline reads the final model artifact
- Web UI sends input to Prediction Pipeline → returns results to user
- All components load parameters via Configuration Manager
c. Architectural patterns / design principles
- Modular, component-based pipeline (single responsibility per component)
- Config-driven (YAML-first approach)
- Layered directories: config, entity (DTOs), components, pipeline, utils
- Experiment tracking integration (MLflow / DagsHub)
- Separation of concerns: ML pipeline vs. serving layer
d. Key technologies / frameworks / libraries
- Python 3.x
- Flask (app.py + Jinja2 templates)
- PyYAML for config
- Pandas, scikit-learn (in components)
- MLflow, DagsHub for experiment tracking
- Docker for containerization
- GitHub Actions for CI/CD

##### Final recommendations
- Explicitly indicate where each component reads from and writes to (e.g., ingested raw files, transformed data folder, model.pkl in artifact store)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration & Data Sources
    subgraph "Configuration & Data Sources"
        RawData["CSV/Database/External API"]:::external
        ConfigYaml["config/config.yaml"]:::config
        ParamsYaml["params.yaml"]:::config
        SchemaYaml["schema.yaml"]:::config
    end

    %% Configuration Manager
    ConfigurationManager["Configuration Manager"]:::config

    %% ML Pipeline
    subgraph "ML Pipeline"
        Orchestrator["main.py (Pipeline Orchestrator)"]:::config
        IngestComp["1. Data Ingestion"]:::ingest
        IngestPipe["Ingestion Pipeline"]:::ingest
        ValidateComp["2. Data Validation"]:::validate
        ValidatePipe["Validation Pipeline"]:::validate
        TransformComp["3. Data Transformation"]:::transform
        TransformPipe["Transformation Pipeline"]:::transform
        TrainComp["4. Model Trainer"]:::train
        TrainPipe["Trainer Pipeline"]:::train
        EvalComp["5. Model Evaluation"]:::eval
        EvalPipe["Evaluation Pipeline"]:::eval
    end

    %% Artifact Store & Tracking
    subgraph "Artifact Store & Tracking"
        ArtifactStore["Model Artifacts"]:::external
        MLflow["MLflow Tracking Server"]:::external
        DagsHub["DagsHub"]:::external
    end

    %% Serving Layer
    subgraph "Serving Layer"
        PredictionPipeline["Prediction Pipeline"]:::predict
        WebUI["Flask Web UI"]:::web
    end

    %% Support Modules
    subgraph "Support Utilities & Constants"
        Utils["common.py"]:::config
        Constants["constants/"]:::config
        Entity["config_entity.py"]:::config
    end

    %% Connections
    RawData --> IngestComp
    ConfigYaml --> ConfigurationManager
    ParamsYaml --> ConfigurationManager
    SchemaYaml --> ConfigurationManager

    ConfigurationManager --> Orchestrator
    Orchestrator --> IngestPipe
    IngestPipe --> ValidateComp
    ValidateComp --> ValidatePipe
    ValidatePipe --> TransformComp
    TransformComp --> TransformPipe
    TransformPipe --> TrainComp
    TrainComp --> TrainPipe
    TrainPipe --> EvalComp
    EvalComp --> EvalPipe

    EvalPipe --> ArtifactStore
    TrainPipe --> ArtifactStore
    ArtifactStore --> MLflow
    ArtifactStore --> DagsHub

    ConfigurationManager --> IngestComp
    ConfigurationManager --> ValidateComp
    ConfigurationManager --> TransformComp
    ConfigurationManager --> TrainComp
    ConfigurationManager --> EvalComp
    ConfigurationManager --> PredictionPipeline
    ConfigurationManager --> WebUI

    ArtifactStore --> PredictionPipeline
    PredictionPipeline --> WebUI

    Utils --> IngestComp
    Utils --> ValidateComp
    Utils --> TransformComp
    Utils --> TrainComp
    Utils --> EvalComp
    Constants --> IngestComp
    Entity --> ConfigurationManager

    %% Click Events
    click ConfigYaml "https://github.com/ishanpandey15/datascienceproject/blob/main/config/config.yaml"
    click ParamsYaml "https://github.com/ishanpandey15/datascienceproject/blob/main/params.yaml"
    click SchemaYaml "https://github.com/ishanpandey15/datascienceproject/blob/main/schema.yaml"
    click ConfigurationManager "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/config/configuration.py"
    click Orchestrator "https://github.com/ishanpandey15/datascienceproject/blob/main/main.py"
    click IngestComp "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/components/data_ingestion.py"
    click IngestPipe "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/pipeline/data_ingestion_pipeline.py"
    click ValidateComp "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/components/data_validation.py"
    click ValidatePipe "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/pipeline/data_validation_pipeline.py"
    click TransformComp "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/components/data_transformation.py"
    click TransformPipe "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/pipeline/data_transformation_pipeline.py"
    click TrainComp "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/components/model_trainer.py"
    click TrainPipe "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/pipeline/model_trainer_pipeline.py"
    click EvalComp "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/components/model_evaluation.py"
    click EvalPipe "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/pipeline/model_evaluation_pipeline.py"
    click PredictionPipeline "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/pipeline/prediction_pipeline.py"
    click WebUI "https://github.com/ishanpandey15/datascienceproject/blob/main/app.py"
    click Utils "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/utils/common.py"
    click Constants "https://github.com/ishanpandey15/datascienceproject/tree/main/src/datascience/constants/"
    click Entity "https://github.com/ishanpandey15/datascienceproject/blob/main/src/datascience/entity/config_entity.py"
    click WebUI "https://github.com/ishanpandey15/datascienceproject/blob/main/templates/index.html"
    click WebUI "https://github.com/ishanpandey15/datascienceproject/blob/main/templates/results.html"

    %% Styles
    classDef config fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef ingest fill:#98FB98,stroke:#000,stroke-width:1px;
    classDef validate fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef transform fill:#8FBC8F,stroke:#000,stroke-width:1px;
    classDef train fill:#2E8B57,stroke:#000,stroke-width:1px;
    classDef eval fill:#006400,stroke:#fff,stroke-width:1px;
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef predict fill:#DDA0DD,stroke:#000,stroke-width:1px;
    classDef web fill:#9370DB,stroke:#000,stroke-width:1px;
```
