How antonio-gpmp/pro-disia works
Main Components
• Data Source Layer:
• Three primary sources: Harvest data (cosecha), Price data (precios), and Meteorological data (meteo). Each one contains cleaned JSON files (limpio_*.json) for multiple years (2018-2023).
• Data Transformation Modules:
• Individual transformation scripts in each directory (e.g., trans_cosecha.py and trans_cosecha2.py for harvest data; transform_precipitation.py in meteo; transform_precios.py in precios).
• A central pipeline orchestrator (pipeline.py and transform_final_json.py) that reads these preprocessed data files, merges them based on common keys (like region and year), and outputs the consolidated CSV (dataset_final_completo.csv).
• Machine Learning Components (Optional/Future Usage):
• Scripts for model training and evaluation (RandomForest.py, RegresionLineal.py, XGBoost.py).
• Pre-trained model files (modelo_random_forest.pkl, random_forest_model.pkl) that indicate the potential to use the generated dataset in training pipelines.
Relationships and Data Flow
• Each data source undergoes its own cleaning/transformation process via dedicated transformation scripts.
• Transformed data is passed to a central pipeline module (triggered via transform_final.py as noted in the README) which consolidates the information by aligning harvest data with prices and weather data based on time/year and region.
• Finally, the consolidated data is output as dataset_final_completo.csv, making it ready for subsequent ML training or data analysis tasks.
• Optionally, machine learning scripts are executed later to build models based on the final dataset.
Architectural Patterns and Design Principles
• Modularity: Each data domain (cosecha, precios, meteo) is separated into its own directory with self-contained scripts and data files.
• Pipeline/ETL Design: The system follows a step-by-step Extract-Transform-Load approach.
• Reusability and Separation of Concerns: Transformation scripts are clearly separated from model scripts and data consolidation logic.
• Extensibility: The design makes it straightforward to add new years or even additional domains following the same structure.
Relevant Technologies and Artifacts
• Python 3 is the core technology.
• JSON and CSV files are the primary data formats.
• Pickle files (.pkl) for storing pre-trained ML models.
• The directory structure itself suggests that the project might eventually integrate various machine learning approaches using frameworks like scikit-learn.