# aaron2rodrigues/project1 architecture

> Architecture diagram of the GitHub repository aaron2rodrigues/project1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aaron2rodrigues/project1
- Repository: https://github.com/aaron2rodrigues/project1

## Overview

##### Project Type & Purpose
- The project is a data science workflow pipeline. Its main goal is to process data from ingestion to transformation, then to training and evaluation of models.
- The README outlines the stages of the workflow, and the file tree shows that the code is modularized using components (for each pipeline step), pipelines (grouping these steps), configuration management, utility functions, and even research notebooks for experimental work.

##### Main Components
a. Data Ingestion Module:
- Components located in src/Project1/components/data_ingestion.py and the corresponding pipeline in src/Project1/pipeline/data_ingestion_pipeline.py.
- How external data sources are integrated (if applicable) and note that the research directory also contains notebooks (e.g., research/data_ingestion-01.ipynb) that might prototype aspects of ingestion.

b. Data Validation Module:
- Data validation component found in src/Project1/components/data_validation.py along with its pipeline in src/Project1/pipeline/data_validation_pipeline.py.
- How this module uses configuration files (config/config.yaml and src/Project1/config/configuration.py) and the schema definition (schema.yaml) to enforce data quality rules.

c. Data Transformation Module:
- Transformation component (src/Project1/components/data_transformation.py) and its pipeline (src/Project1/pipeline/data_transformation_pipeline.py).
- Common utilities (found in src/Project1/utils/common.py) that support the transformation logic.

d. Configuration & Constants:
- Configuration manager located in the configuration directories (both at the top level and within src/Project1/config) that coordinate parameters (params.yaml) and other essential settings.
- Constants module (src/Project1/constants) where key constant values are stored, along with the data entity definitions (src/Project1/entity/entity_config.py).

e. Main Application Entry:
- How main.py triggers the pipeline processes and integrates with the configuration and component modules.

f. Ancillary Elements:
- Supplementary components like Dockerfile (for containerization), GitHub workflows (CI/CD as per .github/workflows), and research notebooks that help in prototyping and validating ideas.

##### Relationships & Interactions
- Start from Data Ingestion → Data Validation → Data Transformation → (subsequently) Model Training & Evaluation (the later stages suggested in the README).
- Main.py to each pipeline module to show that it orchestrates the overall workflow.
- Interactions between utility functions (e.g., common functions) and the main components that help normalize data or handle common tasks.

##### Architectural Patterns
- Layered and modular design:
- Each “component” (ingestion, validation, transformation) encapsulates business logic.
- Pipelines that wrap around these components serve as orchestrators.
- Configuration management is central, ensuring that parameters and schemas are consistent across modules.
- The design hints at a separation of concerns where research notebooks assist in prototyping separate from production pipeline code.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Orchestration Layer
    mainApp["Main Orchestrator (main.py)"]:::orchestrator

    %% Data Pipeline Layer
    subgraph "Data Pipeline"
        %% Data Ingestion Stage
        subgraph "Data Ingestion Stage"
            ingestion_pipeline["Data Ingestion Pipeline"]:::pipeline
            ingestion_component["Data Inestion Module"]:::component
            ingestion_pipeline -->|"processes"| ingestion_component
            ingestion_pipeline -->|"prototyping"| ingestion_research["Data Ingestion Research Notebook"]:::research
        end

        %% Data Validation Stage
        subgraph "Data Validation Stage"
            validation_pipeline["Data Validation Pipeline"]:::pipeline
            validation_component["Data Validation Module"]:::component
            validation_pipeline -->|"processes"| validation_component
            validation_pipeline -->|"prototyping"| validation_research["Data Validation Research Notebook"]:::research
        end

        %% Data Transformation Stage
        subgraph "Data Transformation Stage"
            transformation_pipeline["Data Transformation Pipeline"]:::pipeline
            transformation_component["Data Transformation Module"]:::component
            transformation_pipeline -->|"processes"| transformation_component
            transformation_pipeline -->|"prototyping"| transformation_research["Data Transformation Research Notebook"]:::research
            transformation_component -->|"uses"| transformation_util["Utility Support: Common"]:::utility
        end

        %% Model Training & Evaluation Stage
        model_training["Model Training & Evaluation"]:::training
    end

    %% Configuration & Constants Layer
    subgraph "Configuration & Constants"
        global_config["Global Configurations"]:::config
        project_config["Project Config Module"]:::config
        constants["Constants"]:::config
        entity_config["Entity Configurations"]:::config
    end

    %% Ancillary Elements
    subgraph "Ancillary"
        docker["Docker Container"]:::ancillary
        cicd["CI/CD Workflows"]:::ancillary
        ui["UI Reporting"]:::ancillary
    end

    %% Connections from Main Orchestrator to Pipelines
    mainApp -->|"triggers"| ingestion_pipeline
    mainApp -->|"triggers"| validation_pipeline
    mainApp -->|"triggers"| transformation_pipeline
    mainApp -->|"triggers"| model_training

    %% Sequential Pipeline Flow
    ingestion_pipeline -->|"next"| validation_pipeline
    validation_pipeline -->|"next"| transformation_pipeline
    transformation_pipeline -->|"next"| model_training

    %% Configuration Connections
    global_config -->|"configures"| ingestion_component
    global_config -->|"configures"| validation_component
    global_config -->|"configures"| transformation_component

    project_config -->|"applies"| ingestion_component
    project_config -->|"applies"| validation_component
    project_config -->|"applies"| transformation_component

    constants -->|"defines"| ingestion_component
    constants -->|"defines"| validation_component
    constants -->|"defines"| transformation_component

    entity_config -->|"defines"| ingestion_component
    entity_config -->|"defines"| validation_component
    entity_config -->|"defines"| transformation_component

    %% Ancillary Connections
    mainApp -->|"integrates"| docker
    mainApp -->|"integrates"| cicd
    mainApp -->|"integrates"| ui

    %% Styling
    classDef orchestrator fill:#f9c,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#bbf,stroke:#333,stroke-width:2px;
    classDef component fill:#9f9,stroke:#333,stroke-width:2px;
    classDef config fill:#ff9,stroke:#333,stroke-width:2px;
    classDef research fill:#fc9,stroke:#333,stroke-width:2px;
    classDef utility fill:#cff,stroke:#333,stroke-width:2px;
    classDef ancillary fill:#f99,stroke:#333,stroke-width:2px;
    classDef training fill:#d9d,stroke:#333,stroke-width:2px;

    %% Click Events
    click mainApp "https://github.com/aaron2rodrigues/project1/blob/main/main.py"
    click ingestion_component "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/components/data_ingestion.py"
    click ingestion_pipeline "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/pipeline/data_ingestion_pipeline.py"
    click ingestion_research "https://github.com/aaron2rodrigues/project1/blob/main/research/data_ingestion-01.ipynb"
    click validation_component "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/components/data_validation.py"
    click validation_pipeline "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/pipeline/data_validation_pipeline.py"
    click validation_research "https://github.com/aaron2rodrigues/project1/blob/main/research/02_data_validation.ipynb"
    click transformation_component "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/components/data_transformation.py"
    click transformation_pipeline "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/pipeline/data_transformation_pipeline.py"
    click transformation_util "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/utils/common.py"
    click transformation_research "https://github.com/aaron2rodrigues/project1/blob/main/research/data_transformation.ipynb"
    click global_config "https://github.com/aaron2rodrigues/project1/blob/main/config/config.yaml"
    click project_config "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/config/configuration.py"
    click constants "https://github.com/aaron2rodrigues/project1/tree/main/src/Project1/constants"
    click entity_config "https://github.com/aaron2rodrigues/project1/blob/main/src/Project1/entity/entity_config.py"
    click docker "https://github.com/aaron2rodrigues/project1/tree/main/Dockerfile"
    click cicd "https://github.com/aaron2rodrigues/project1/tree/main/.github/workflows"
    click ui "https://github.com/aaron2rodrigues/project1/blob/main/templates/index.html"
```
