# aaron2rodrigues/pp architecture

> Architecture diagram of the GitHub repository aaron2rodrigues/pp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aaron2rodrigues/pp
- Repository: https://github.com/aaron2rodrigues/pp

## Overview

- Data Ingestion: Represented both as a dedicated component (data_ingestion.py) and its corresponding pipeline module (data_ingestion_pipeline.py). This component reads raw data and possibly interfaces with external data sources.
- Data Validation: Shown through the data_validation.py component and the dedicated data_validation_pipeline.py module. This part uses the schema (schema.yaml) to check incoming data quality.
- Data Transformation: Captured by data_transformation.py and its pipeline partner (data_transformation_pipeline.py) where the raw data is prepared for model training.
- Model Training: Found as model_trainer.py (with its pipeline implementation in model_trianer_pipeline.py). This module is responsible for creating a predictive model using the processed data.
- Model Evaluation: Illustrated via model_evaluation.py with its pipelining counterpart (model_evaluation_pipeline.py) for testing and evaluating model performances.
- Prediction Pipeline: There is also a prediction_pipeline.py which likely connects the trained model to a production inference service.
- Configuration Manager: This layer uses files like config.yaml in the config/ and src/Project1/config directories along with an entity definition (entity_config.py) to drive the project’s dynamic behavior.
- Common Utilities: Tools stored in the utils directory are supporting functions needed across components.
- Web Interface / API Layer: With templates (index.html and results.html) and app.py, there might be an interactive front-end or API endpoint to show results or interact with the model.

##### Component relationships and interactions
- Begin the flow by reading configuration and parameters (using config/config.yaml, params.yaml, and schema.yaml). Use the configuration manager (in src/Project1/config/configuration.py) to load and validate these settings.
- After transformation, the system splits into training and then evaluation. Note that the evaluation results could then be exposed via the prediction pipeline and ultimately through the web interface (index and results templates).
- The project also contains research notebooks that represent exploratory phases and may serve as a documentation or prototype source; however, they might not be “part” of the live architecture.

##### Architectural patterns and design principles
- Modularity: Notice how the project separates concerns clearly (components, pipelines, configuration, and utilities). This modular design allows for easy updates to individual stages.
- Config-driven behavior: Showing that the system’s behavior is driven by external YAML configuration files encourages reusability and flexibility.
- Pipeline Abstraction: The different pipelines (for ingestion, transformation, etc.) invoke their respective components in a sequential manner. This reflects a typical ETL (Extract, Transform, Load) framework commonly used in data science projects.

##### Relevant technologies and frameworks
- Python is the primary language, as seen by all the .py files.
- YAML for configuration management (config.yaml, schema.yaml, params.yaml).
- Docker for containerization (Dockerfile).
- Jupyter Notebooks for research and exploration (in the research directory).
- Basic web templating (HTML files in templates) possibly with a lightweight framework hinted at by app.py.
- GitHub Actions (under .github/workflows) indicate CI/CD integration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    DF("Dockerfile"):::docker
    DF --> CM

    subgraph "Configuration"
        CF("Config File: config.yaml"):::config
        PM("Params File: params.yaml"):::config
        SC("Schema File: schema.yaml"):::config
        EC("Entity Config"):::config
        CM("Configuration Manager"):::config
        CF --> CM
        PM --> CM
        SC --> CM
        EC --> CM
    end

    subgraph "Data Ingestion"
        DI_P("Data Ingestion Pipeline"):::ingestion
        DI_C("Data Ingestion Component"):::ingestion
        DI_P --> DI_C
    end

    subgraph "Data Validation"
        DV_P("Data Validation Pipeline"):::validation
        DV_C("Data Validation Component"):::validation
        DV_P --> DV_C
    end

    subgraph "Data Transformation"
        DT_P("Data Transformation Pipeline"):::transformation
        DT_C("Data Transformation Component"):::transformation
        DT_P --> DT_C
    end

    subgraph "Model Training"
        MT_P("Model Training Pipeline"):::training
        MT_C("Model Training Component"):::training
        MT_P --> MT_C
    end

    subgraph "Model Evaluation"
        ME_P("Model Evaluation Pipeline"):::evaluation
        ME_C("Model Evaluation Component"):::evaluation
        ME_P --> ME_C
    end

    PP("Prediction Pipeline"):::prediction

    CU("Common Utilities"):::utils

    subgraph "Web Interface/API"
        APP("App.py"):::web
        T1("Template: Index.html"):::web
        T2("Template: Results.html"):::web
        APP --- T1
        APP --- T2
    end

    %% Main Pipeline Flow
    CM --> DI_P
    DI_C --> DV_P
    DV_C --> DT_P
    DT_C --> MT_P
    MT_C --> ME_P
    ME_C --> PP
    PP --> APP

    %% Dashed support arrows from Common Utilities
    CU -.-> DI_P
    CU -.-> DV_P
    CU -.-> DT_P
    CU -.-> MT_P
    CU -.-> ME_P
    CU -.-> PP

    %% Styles
    classDef config fill:#f9e79f,stroke:#af7ac5,stroke-width:2px;
    classDef ingestion fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef validation fill:#a9dfbf,stroke:#1e8449,stroke-width:2px;
    classDef transformation fill:#f5cba7,stroke:#d35400,stroke-width:2px;
    classDef training fill:#fdebd0,stroke:#d35400,stroke-width:2px;
    classDef evaluation fill:#d5f5e3,stroke:#117864,stroke-width:2px;
    classDef prediction fill:#f9c5d1,stroke:#c0392b,stroke-width:2px;
    classDef web fill:#d6eaf8,stroke:#2874a6,stroke-width:2px;
    classDef utils fill:#d2b4de,stroke:#6c3483,stroke-width:2px;
    classDef docker fill:#f9e79f,stroke:#af7ac5,stroke-width:2px;

    %% Click Events
    click DF "https://github.com/aaron2rodrigues/pp/tree/main/Dockerfile"
    click CF "https://github.com/aaron2rodrigues/pp/blob/main/config/config.yaml"
    click PM "https://github.com/aaron2rodrigues/pp/blob/main/params.yaml"
    click SC "https://github.com/aaron2rodrigues/pp/blob/main/schema.yaml"
    click EC "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/entity/entity_config.py"
    click CM "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/config/configuration.py"
    click DI_C "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/components/data_ingestion.py"
    click DI_P "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/pipeline/data_ingestion_pipeline.py"
    click DV_C "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/components/data_validation.py"
    click DV_P "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/pipeline/data_validation_pipeline.py"
    click DT_C "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/components/data_transformation.py"
    click DT_P "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/pipeline/data_transformation_pipeline.py"
    click MT_C "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/components/model_trainer.py"
    click MT_P "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/pipeline/model_trianer_pipeline.py"
    click ME_C "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/components/model_evaluation.py"
    click ME_P "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/pipeline/model_evaluation_pipeline.py"
    click PP "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/pipeline/prediction_pipeline.py"
    click CU "https://github.com/aaron2rodrigues/pp/blob/main/src/Project1/utils/common.py"
    click APP "https://github.com/aaron2rodrigues/pp/blob/main/app.py"
    click T1 "https://github.com/aaron2rodrigues/pp/blob/main/templates/index.html"
    click T2 "https://github.com/aaron2rodrigues/pp/blob/main/templates/results.html"
```
