# antonio-gpmp/pro-disia architecture

> Architecture diagram of the GitHub repository antonio-gpmp/pro-disia, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antonio-gpmp/pro-disia
- Repository: https://github.com/antonio-gpmp/pro-disia

## Overview

##### Main Components
- Data Source Layer:
- Three primary sources: Harvest data (cosecha), Price data (precios), and Meteorological data (meteo). Each one contains cleaned JSON files (limpio_*.json) for multiple years (2018-2023).
- Data Transformation Modules:
- Individual transformation scripts in each directory (e.g., trans_cosecha.py and trans_cosecha2.py for harvest data; transform_precipitation.py in meteo; transform_precios.py in precios).
- A central pipeline orchestrator (pipeline.py and transform_final_json.py) that reads these preprocessed data files, merges them based on common keys (like region and year), and outputs the consolidated CSV (dataset_final_completo.csv).
- Machine Learning Components (Optional/Future Usage):
- Scripts for model training and evaluation (RandomForest.py, RegresionLineal.py, XGBoost.py).
- Pre-trained model files (modelo_random_forest.pkl, random_forest_model.pkl) that indicate the potential to use the generated dataset in training pipelines.

##### Relationships and Data Flow
- Each data source undergoes its own cleaning/transformation process via dedicated transformation scripts.
- Transformed data is passed to a central pipeline module (triggered via transform_final.py as noted in the README) which consolidates the information by aligning harvest data with prices and weather data based on time/year and region.
- Finally, the consolidated data is output as dataset_final_completo.csv, making it ready for subsequent ML training or data analysis tasks.
- Optionally, machine learning scripts are executed later to build models based on the final dataset.

##### Architectural Patterns and Design Principles
- Modularity: Each data domain (cosecha, precios, meteo) is separated into its own directory with self-contained scripts and data files.
- Pipeline/ETL Design: The system follows a step-by-step Extract-Transform-Load approach.
- Reusability and Separation of Concerns: Transformation scripts are clearly separated from model scripts and data consolidation logic.
- Extensibility: The design makes it straightforward to add new years or even additional domains following the same structure.

##### Relevant Technologies and Artifacts
- Python 3 is the core technology.
- JSON and CSV files are the primary data formats.
- Pickle files (.pkl) for storing pre-trained ML models.
- The directory structure itself suggests that the project might eventually integrate various machine learning approaches using frameworks like scikit-learn.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources Layer
    subgraph "Data Sources"
        direction TB
        %% Harvest Data (cosecha)
        subgraph "Harvest Data (cosecha)"
            direction TB
            H1["limpio_cosecha_2018.json"]:::rawData
            H2["limpio_cosecha_2019.json"]:::rawData
            H3["limpio_cosecha_2020.json"]:::rawData
            H4["limpio_cosecha_2021.json"]:::rawData
            H5["limpio_cosecha_2022.json"]:::rawData
            H6["limpio_cosecha_2023.json"]:::rawData
            H7["Transformation Script: trans_cosecha.py"]:::transformation
            H8["Transformation Script: trans_cosecha2.py"]:::transformation
        end

        %% Price Data (precios)
        subgraph "Price Data (precios)"
            direction TB
            P1["limpio_precios_2018.json"]:::rawData
            P2["limpio_precios_2019.json"]:::rawData
            P3["limpio_precios_2020.json"]:::rawData
            P4["limpio_precios_2021.json"]:::rawData
            P5["limpio_precios_2022.json"]:::rawData
            P6["limpio_precios_2023.json"]:::rawData
            P7["Transformation Script: transform_precios.py"]:::transformation
        end

        %% Meteorological Data (meteo)
        subgraph "Meteorological Data (meteo)"
            direction TB
            M1["limpio_precipitaciones_2018.json"]:::rawData
            M2["limpio_precipitaciones_2019.json"]:::rawData
            M3["limpio_precipitaciones_2020.json"]:::rawData
            M4["limpio_precipitaciones_2021.json"]:::rawData
            M5["limpio_precipitaciones_2022.json"]:::rawData
            M6["limpio_precipitaciones_2023.json"]:::rawData
            M7["Transformation Script: transform_precipitation.py"]:::transformation
        end
    end

    %% Data Consolidation Pipeline
    subgraph "Data Consolidation Pipeline"
        direction TB
        CP1["Consolidation Script: pipeline.py"]:::pipeline
        CP2["Final Transformation Script: transform_final_json.py"]:::pipeline
        FD1["Consolidated CSV: dataset_final_completo.csv"]:::output
        FD2["Consolidated JSON: dataset_final_completo.json"]:::output
    end

    %% Machine Learning Components
    subgraph "Machine Learning Components"
        direction TB
        ML1["ML Script: RandomForest.py"]:::ml
        ML2["ML Script: RegresionLineal.py"]:::ml
        ML3["ML Script: XGBoost.py"]:::ml
        MLM1["Pre-trained Model: modelo_random_forest.pkl"]:::ml
        MLM2["Pre-trained Model: random_forest_model.pkl"]:::ml
    end

    %% Connections from Raw Data to Transformation Scripts
    %% Harvest Data connections
    H1 -->|"ingest"| H7
    H2 -->|"ingest"| H7
    H3 -->|"ingest"| H7
    H4 -->|"ingest"| H7
    H5 -->|"ingest"| H7
    H6 -->|"ingest"| H7
    H1 -->|"optional"| H8
    H2 -->|"optional"| H8
    H3 -->|"optional"| H8
    H4 -->|"optional"| H8
    H5 -->|"optional"| H8
    H6 -->|"optional"| H8

    %% Price Data connections
    P1 -->|"ingest"| P7
    P2 -->|"ingest"| P7
    P3 -->|"ingest"| P7
    P4 -->|"ingest"| P7
    P5 -->|"ingest"| P7
    P6 -->|"ingest"| P7

    %% Meteorological Data connections
    M1 -->|"ingest"| M7
    M2 -->|"ingest"| M7
    M3 -->|"ingest"| M7
    M4 -->|"ingest"| M7
    M5 -->|"ingest"| M7
    M6 -->|"ingest"| M7

    %% Connections from Transformation Scripts to Consolidation Pipeline
    H7 --> CP1
    H8 --> CP1
    P7 --> CP1
    M7 --> CP1

    CP1 --> CP2
    CP2 --> FD1
    CP2 --> FD2

    %% Connections to Machine Learning Components
    FD1 --> ML1
    FD1 --> ML2
    FD1 --> ML3

    %% Click Events for Harvest Data Files and Scripts
    click H1 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/limpio_cosecha_2018.json"
    click H2 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/limpio_cosecha_2019.json"
    click H3 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/limpio_cosecha_2020.json"
    click H4 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/limpio_cosecha_2021.json"
    click H5 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/limpio_cosecha_2022.json"
    click H6 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/limpio_cosecha_2023.json"
    click H7 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/trans_cosecha.py"
    click H8 "https://github.com/antonio-gpmp/pro-disia/blob/main/cosecha/trans_cosecha2.py"

    %% Click Events for Price Data Files and Script
    click P1 "https://github.com/antonio-gpmp/pro-disia/blob/main/precios/limpio_precios_2018.json"
    click P2 "https://github.com/antonio-gpmp/pro-disia/blob/main/precios/limpio_precios_2019.json"
    click P3 "https://github.com/antonio-gpmp/pro-disia/blob/main/precios/limpio_precios_2020.json"
    click P4 "https://github.com/antonio-gpmp/pro-disia/blob/main/precios/limpio_precios_2021.json"
    click P5 "https://github.com/antonio-gpmp/pro-disia/blob/main/precios/limpio_precios_2022.json"
    click P6 "https://github.com/antonio-gpmp/pro-disia/blob/main/precios/limpio_precios_2023.json"
    click P7 "https://github.com/antonio-gpmp/pro-disia/blob/main/precios/transform_precios.py"

    %% Click Events for Meteorological Data Files and Script
    click M1 "https://github.com/antonio-gpmp/pro-disia/blob/main/meteo/limpio_precipitaciones_2018.json"
    click M2 "https://github.com/antonio-gpmp/pro-disia/blob/main/meteo/limpio_precipitaciones_2019.json"
    click M3 "https://github.com/antonio-gpmp/pro-disia/blob/main/meteo/limpio_precipitaciones_2020.json"
    click M4 "https://github.com/antonio-gpmp/pro-disia/blob/main/meteo/limpio_precipitaciones_2021.json"
    click M5 "https://github.com/antonio-gpmp/pro-disia/blob/main/meteo/limpio_precipitaciones_2022.json"
    click M6 "https://github.com/antonio-gpmp/pro-disia/blob/main/meteo/limpio_precipitaciones_2023.json"
    click M7 "https://github.com/antonio-gpmp/pro-disia/blob/main/meteo/transform_precipitation.py"

    %% Click Events for Data Consolidation Pipeline
    click CP1 "https://github.com/antonio-gpmp/pro-disia/blob/main/pipeline.py"
    click CP2 "https://github.com/antonio-gpmp/pro-disia/blob/main/transform_final_json.py"
    click FD1 "https://github.com/antonio-gpmp/pro-disia/blob/main/dataset_final_completo.csv"
    click FD2 "https://github.com/antonio-gpmp/pro-disia/blob/main/dataset_final_completo.json"

    %% Click Events for Machine Learning Components
    click ML1 "https://github.com/antonio-gpmp/pro-disia/blob/main/RandomForest.py"
    click ML2 "https://github.com/antonio-gpmp/pro-disia/blob/main/RegresionLineal.py"
    click ML3 "https://github.com/antonio-gpmp/pro-disia/blob/main/XGBoost.py"
    click MLM1 "https://github.com/antonio-gpmp/pro-disia/blob/main/modelo_random_forest.pkl"
    click MLM2 "https://github.com/antonio-gpmp/pro-disia/blob/main/random_forest_model.pkl"

    %% Styles
    classDef rawData fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef transformation fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef pipeline fill:#FFD700,stroke:#000,stroke-width:1px;
    classDef output fill:#FFFACD,stroke:#000,stroke-width:1px;
    classDef ml fill:#FFA07A,stroke:#000,stroke-width:1px;
```
