# ashupa2el/crop-yield-project architecture

> Architecture diagram of the GitHub repository ashupa2el/crop-yield-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ashupa2el/crop-yield-project
- Repository: https://github.com/ashupa2el/crop-yield-project

## Overview

1. **Repository purpose**
   - This is a Python-based MLOps project for **crop yield prediction**.
   - The architecture is organized as a **linear ML pipeline**: ingest data, preprocess, train multiple regressors, evaluate them, choose the best model, and publish artifacts.

2. **Main entrypoints**
   - `main.py` is the **primary orchestrator** for the end-to-end pipeline.
   - `demo.py` is a **consumer/demo entrypoint** that loads the persisted model and scaler to make sample predictions.
   - `App.py` exists as an additional top-level script, but the README does not describe it; it is not part of the core documented pipeline.

3. **Pipeline stage boundaries**
   - The core logic is split across `SRC/01_load_data.py` through `SRC/05_select_best_model.py`.
   - Each numbered script represents a **single pipeline stage** with a clear boundary:
     - load/validate raw data
     - preprocess and split
     - train models
     - evaluate models
     - select the best model
   - This makes the project easy to diagram as a **sequential DAG with five stages**.

4. **Data flow**
   - Input starts with `Data/crop_yield_processed.csv`.
   - The loading stage produces a normalized/raw intermediate dataset.
   - Preprocessing creates train/test artifacts under `Data/Train/` and `Data/Test/`:
     - `X_train.csv`, `y_train.csv`
     - `X_test.csv`, `y_test.csv`
   - Those datasets are then consumed by training and evaluation stages.

5. **Modeling subsystem**
   - The training stage fits **four supervised regression models**:
     - Linear Regression
     - Ridge
     - Random Forest
     - Gradient Boosting
   - A scaler is saved alongside the models, indicating the feature pipeline is part of the model contract.
   - The repository treats model training as an **offline batch job**, not an online inference service.

6. **Model selection and persistence**
   - Evaluation computes metrics for each model, then selection chooses the **best model by R²**.
   - Outputs are serialized as `.pkl` artifacts, with:
     - `best_model.pkl`
     - `scaler.pkl`
     - one file per trained model
   - This boundary is important for architecture diagrams: **training-time artifacts** are separated from **runtime inference artifacts**.

7. **Reporting and observability**
   - `Reports/` contains the product of evaluation and selection:
     - `model_metrics.json`
     - comparison tables
     - plots
     - model card documentation
   - Logs are also first-class outputs in the design, with per-stage logs and a pipeline log referenced in the README.
   - This gives the pipeline both **machine-readable metrics** and **human-readable diagnostics**.

8. **Configuration and reproducibility**
   - `params.yaml` centralizes tunable settings such as split ratio, random seed, scaling method, and model hyperparameters.
   - This makes the pipeline **config-driven** rather than hardcoded.
   - The repo is structured to support reproducible runs through the same config and data versions.

9. **DVC integration**
   - `dvc.yaml` indicates the pipeline is intended to be managed by **DVC**.
   - DVC acts as the orchestration/versioning layer for data, pipeline stages, and metrics comparison.
   - In an architecture diagram, DVC should be shown as the **reproducibility/control plane** around the Python stage scripts and tracked data artifacts.

10. **Repository organization**
   - `Data/` holds dataset inputs and intermediate split files.
   - `SRC/` holds the stage implementations.
   - `Reports/` holds outputs for analysis and governance.
   - Top-level scripts (`main.py`, `demo.py`) provide operational entrypoints.
   - `requirements.txt`, `Makefile`, and `LICENSE` support environment setup and project operations.

11. **Operational model**
   - The pipeline is designed as a **batch ML workflow** that can be run locally in a few commands.
   - There is no evidence of a serving layer, API backend, or streaming component in the documented architecture.
   - The runtime boundary is therefore: **offline training pipeline -> serialized artifacts -> local demo/inference script**.

12. **What to emphasize in a diagram**
   - Show the flow from **CSV data → preprocessing/split → four model trainers → evaluation/metrics → best-model selection → persisted model/scaler + reports**.
   - Show `params.yaml` and `dvc.yaml` as **cross-cutting controls** that influence all stages.
   - Show `demo.py` as a separate **inference consumer** of the final artifacts rather than part of training.

Main entry points: [`SRC/04_evaluate.py`](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/04_evaluate.py) (evaluate), [`SRC/03_train.py`](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/03_train.py) (train models), [`dvc.yaml`](https://github.com/ashupa2el/crop-yield-project/blob/main/dvc.yaml) (dvc.yaml), [`SRC/05_select_best_model.py`](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/05_select_best_model.py) (select best)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entrypoints"]
  node_main_py["main.py<br/>orchestrator<br/>[main.py]"]
  node_demo_py["demo.py<br/>inference demo<br/>[demo.py]"]
  node_app_py["App.py<br/>script<br/>[App.py]"]
end

subgraph group_pipeline["ML Pipeline"]
  node_load_stage["load data<br/>stage<br/>[01_load_data.py]"]
  node_preprocess_stage["preprocess<br/>stage<br/>[02_preprocess.py]"]
  node_train_stage["train models<br/>stage<br/>[03_train.py]"]
  node_evaluate_stage["evaluate<br/>stage<br/>[04_evaluate.py]"]
  node_select_stage["select best<br/>stage"]
end

subgraph group_artifacts["Artifacts"]
  node_raw_csv[("raw data<br/>dataset")]
  node_split_data[("train/test data<br/>intermediate data")]
  node_trained_models[("model pkls<br/>artifact set")]
  node_model_metrics["metrics<br/>report data<br/>[model_metrics.json]"]
  node_best_artifacts[("best artifacts<br/>release artifact")]
  node_reports["Reports<br/>governance output"]
end

subgraph group_control["Control Plane"]
  node_dvc_yaml["dvc.yaml<br/>pipeline control<br/>[dvc.yaml]"]
  node_params_yaml["params.yaml<br/>config<br/>[params.yaml]"]
end

node_dvc_yaml -->|"orchestrates"| node_load_stage
node_dvc_yaml -->|"orchestrates"| node_preprocess_stage
node_dvc_yaml -->|"orchestrates"| node_train_stage
node_dvc_yaml -->|"orchestrates"| node_evaluate_stage
node_dvc_yaml -->|"orchestrates"| node_select_stage
node_params_yaml -->|"configures"| node_load_stage
node_params_yaml -->|"configures"| node_preprocess_stage
node_params_yaml -->|"configures"| node_train_stage
node_params_yaml -->|"configures"| node_evaluate_stage
node_raw_csv -->|"input"| node_load_stage
node_main_py -->|"runs"| node_load_stage
node_load_stage -->|"passes data"| node_preprocess_stage
node_preprocess_stage -->|"writes"| node_split_data
node_split_data -->|"consumes"| node_train_stage
node_split_data -->|"consumes"| node_evaluate_stage
node_train_stage -->|"serializes"| node_trained_models
node_train_stage -->|"produces models"| node_evaluate_stage
node_evaluate_stage -->|"writes"| node_model_metrics
node_evaluate_stage -->|"feeds metrics"| node_select_stage
node_select_stage -->|"publishes"| node_best_artifacts
node_select_stage -->|"documents"| node_reports
node_best_artifacts -->|"loaded by"| node_demo_py
node_main_py -->|"runs"| node_train_stage
node_main_py -->|"runs"| node_evaluate_stage
node_main_py -->|"runs"| node_select_stage

click node_main_py "https://github.com/ashupa2el/crop-yield-project/blob/main/main.py"
click node_demo_py "https://github.com/ashupa2el/crop-yield-project/blob/main/demo.py"
click node_app_py "https://github.com/ashupa2el/crop-yield-project/blob/main/App.py"
click node_dvc_yaml "https://github.com/ashupa2el/crop-yield-project/blob/main/dvc.yaml"
click node_params_yaml "https://github.com/ashupa2el/crop-yield-project/blob/main/params.yaml"
click node_raw_csv "https://github.com/ashupa2el/crop-yield-project/blob/main/Data/crop_yield_processed.csv"
click node_load_stage "https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/01_load_data.py"
click node_preprocess_stage "https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/02_preprocess.py"
click node_train_stage "https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/03_train.py"
click node_evaluate_stage "https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/04_evaluate.py"
click node_model_metrics "https://github.com/ashupa2el/crop-yield-project/blob/main/Reports/model_metrics.json"
click node_select_stage "https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/05_select_best_model.py"
click node_reports "https://github.com/ashupa2el/crop-yield-project/tree/main/Reports"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_demo_py,node_app_py toneBlue
class node_load_stage,node_preprocess_stage,node_train_stage,node_evaluate_stage,node_select_stage toneAmber
class node_raw_csv,node_split_data,node_trained_models,node_model_metrics,node_best_artifacts,node_reports toneMint
class node_dvc_yaml,node_params_yaml toneRose
```

## Components

### Entrypoints

Top-level scripts used to run training or inference.

- [main.py](https://github.com/ashupa2el/crop-yield-project/blob/main/main.py): orchestrator · Primary end-to-end pipeline runner. · `main.py`
- [demo.py](https://github.com/ashupa2el/crop-yield-project/blob/main/demo.py): inference demo · Loads persisted artifacts for sample predictions. · `demo.py`
- [App.py](https://github.com/ashupa2el/crop-yield-project/blob/main/App.py): script · Additional top-level script not described as part of the core pipeline. · `App.py`

### ML Pipeline

Sequential batch stages for data prep, training, evaluation, and selection.

- [load data](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/01_load_data.py): stage · Validates and normalizes the input dataset. · `SRC/01_load_data.py`
- [preprocess](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/02_preprocess.py): stage · Prepares features and creates train/test splits. · `SRC/02_preprocess.py`
- [train models](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/03_train.py): stage · Fits the regression models and saves the scaler. · `SRC/03_train.py`
- [evaluate](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/04_evaluate.py): stage · Computes metrics for each trained model on the test split. · `SRC/04_evaluate.py`
- [select best](https://github.com/ashupa2el/crop-yield-project/blob/main/SRC/05_select_best_model.py): stage · Chooses the best model by R² and publishes the winner. · `SRC/05_select_best_model.py`

### Artifacts

Persisted datasets, models, and reporting outputs.

- [raw data](https://github.com/ashupa2el/crop-yield-project/blob/main/Data/crop_yield_processed.csv): dataset · Processed crop yield source CSV. · `Data/crop_yield_processed.csv`
- **train/test data**: intermediate data · Persisted X/y train and test CSV artifacts.
- **model pkls**: artifact set · Serialized trained regressors and scaler artifacts.
- [metrics](https://github.com/ashupa2el/crop-yield-project/blob/main/Reports/model_metrics.json): report data · Machine-readable comparison results and plots. · `Reports/model_metrics.json`
- **best artifacts**: release artifact · Final best_model.pkl and scaler.pkl used for inference.
- [Reports](https://github.com/ashupa2el/crop-yield-project/tree/main/Reports): governance output · Comparison tables, plots, and model-card style documentation. · `Reports`

### Control Plane

Configuration and orchestration files that govern reproducible runs.

- [dvc.yaml](https://github.com/ashupa2el/crop-yield-project/blob/main/dvc.yaml): pipeline control · DVC pipeline definition and reproducibility layer. · `dvc.yaml`
- [params.yaml](https://github.com/ashupa2el/crop-yield-project/blob/main/params.yaml): config · Centralized split, seed, scaling, and model hyperparameters. · `params.yaml`

## Connections

- dvc.yaml → load data: orchestrates · DVC tracks the stage in the reproducible pipeline.
- dvc.yaml → preprocess: orchestrates · DVC tracks preprocessing as a pipeline step.
- dvc.yaml → train models: orchestrates · DVC tracks training as a pipeline step.
- dvc.yaml → evaluate: orchestrates · DVC tracks evaluation as a pipeline step.
- dvc.yaml → select best: orchestrates · DVC tracks model selection as a pipeline step.
- params.yaml → load data: configures · Shared config influences pipeline execution.
- params.yaml → preprocess: configures · Controls splitting and preprocessing settings.
- params.yaml → train models: configures · Controls model hyperparameters and scaling.
- params.yaml → evaluate: configures · Controls evaluation setup and comparisons.
- raw data → load data: input · Source dataset consumed by the pipeline.
- main.py → load data: runs · Main entrypoint starts the end-to-end batch workflow.
- load data → preprocess: passes data · Loaded dataset flows into preprocessing.
- preprocess → train/test data: writes · Creates train/test CSV artifacts.
- train/test data → train models: consumes · Training reads the prepared training split.
- train/test data → evaluate: consumes · Evaluation uses the held-out test split.
- train models → model pkls: serializes · Saves model files and scaler artifacts.
- train models → evaluate: produces models · Evaluation consumes trained estimators.
- evaluate → metrics: writes · Stores metrics and comparison outputs.
- evaluate → select best: feeds metrics · Selection uses model scores to choose the winner.
- select best → best artifacts: publishes · Emits the final model and scaler for inference.
- select best → Reports: documents · Outputs report artifacts for review and governance.
- best artifacts → demo.py: loaded by · Demo script uses the persisted best model and scaler.
- main.py → train models: runs · Main entrypoint advances through the pipeline stages.
- main.py → evaluate: runs · Main entrypoint advances through the pipeline stages.
- main.py → select best: runs · Main entrypoint advances through the pipeline stages.
