How ashupa2el/crop-yield-project works
1. Repository purpose
- This is a Python-based MLOps project for crop yield prediction.
- The architecture is organized as a linear ML pipeline: ingest data, preprocess, train multiple regressors, evaluate them, choose the best model, and publish artifacts.
2. Main entrypoints
- main.py is the primary orchestrator for the end-to-end pipeline.
- demo.py is a consumer/demo entrypoint that loads the persisted model and scaler to make sample predictions.
- App.py exists as an additional top-level script, but the README does not describe it; it is not part of the core documented pipeline.
3. Pipeline stage boundaries
- The core logic is split across SRC/01_load_data.py through SRC/05_select_best_model.py.
- Each numbered script represents a single pipeline stage with a clear boundary:
- load/validate raw data
- preprocess and split
- train models
- evaluate models
- select the best model
- This makes the project easy to diagram as a sequential DAG with five stages.
4. Data flow
- Input starts with Data/crop_yield_processed.csv.
- The loading stage produces a normalized/raw intermediate dataset.
- Preprocessing creates train/test artifacts under Data/Train/ and Data/Test/:
- X_train.csv, y_train.csv
- X_test.csv, y_test.csv
- Those datasets are then consumed by training and evaluation stages.
5. Modeling subsystem
- The training stage fits four supervised regression models:
- Linear Regression
- Ridge
- Random Forest
- Gradient Boosting
- A scaler is saved alongside the models, indicating the feature pipeline is part of the model contract.
- The repository treats model training as an offline batch job, not an online inference service.
6. Model selection and persistence
- Evaluation computes metrics for each model, then selection chooses the best model by R².
- Outputs are serialized as .pkl artifacts, with:
- best_model.pkl
- scaler.pkl
- one file per trained model
- This boundary is important for architecture diagrams: training-time artifacts are separated from runtime inference artifacts.
7. Reporting and observability
- Reports/ contains the product of evaluation and selection:
- model_metrics.json
- comparison tables
- plots
- model card documentation
- Logs are also first-class outputs in the design, with per-stage logs and a pipeline log referenced in the README.
- This gives the pipeline both machine-readable metrics and human-readable diagnostics.
8. Configuration and reproducibility
- params.yaml centralizes tunable settings such as split ratio, random seed, scaling method, and model hyperparameters.
- This makes the pipeline config-driven rather than hardcoded.
- The repo is structured to support reproducible runs through the same config and data versions.
9. DVC integration
- dvc.yaml indicates the pipeline is intended to be managed by DVC.
- DVC acts as the orchestration/versioning layer for data, pipeline stages, and metrics comparison.
- In an architecture diagram, DVC should be shown as the reproducibility/control plane around the Python stage scripts and tracked data artifacts.
10. Repository organization
- Data/ holds dataset inputs and intermediate split files.
- SRC/ holds the stage implementations.
- Reports/ holds outputs for analysis and governance.
- Top-level scripts (main.py, demo.py) provide operational entrypoints.
- requirements.txt, Makefile, and LICENSE support environment setup and project operations.
11. Operational model
- The pipeline is designed as a batch ML workflow that can be run locally in a few commands.
- There is no evidence of a serving layer, API backend, or streaming component in the documented architecture.
- The runtime boundary is therefore: offline training pipeline -> serialized artifacts -> local demo/inference script.
12. What to emphasize in a diagram
- Show the flow from CSV data → preprocessing/split → four model trainers → evaluation/metrics → best-model selection → persisted model/scaler + reports.
- Show params.yaml and dvc.yaml as cross-cutting controls that influence all stages.
- Show demo.py as a separate inference consumer of the final artifacts rather than part of training.
Main entry points: 04_evaluate.py evaluate · 03_train.py train models · dvc.yaml dvc.yaml · 05_select_best_model.py select best
Components 16
Entrypoints
Top-level scripts used to run training or inference.
ML Pipeline
Sequential batch stages for data prep, training, evaluation, and selection.
- load data · Validates and normalizes the input dataset.
SRC/01_load_data.py - preprocess · Prepares features and creates train/test splits.
SRC/02_preprocess.py - train models · Fits the regression models and saves the scaler.
SRC/03_train.py - evaluate · Computes metrics for each trained model on the test split.
SRC/04_evaluate.py - select best · Chooses the best model by R² and publishes the winner.
SRC/05_select_best_model.py
Artifacts
Persisted datasets, models, and reporting outputs.
- raw data · Processed crop yield source CSV.
Data/crop_yield_processed.csv - train/test data · Persisted X/y train and test CSV artifacts.
- model pkls · Serialized trained regressors and scaler artifacts.
- metrics · Machine-readable comparison results and plots.
Reports/model_metrics.json - best artifacts · Final best_model.pkl and scaler.pkl used for inference.
- Reports · Comparison tables, plots, and model-card style documentation.
Reports
Control Plane
Configuration and orchestration files that govern reproducible runs.
- dvc.yaml · DVC pipeline definition and reproducibility layer.
dvc.yaml - params.yaml · Centralized split, seed, scaling, and model hyperparameters.
params.yaml
Connections 25
- dvc.yaml → load data: orchestrates · DVC tracks the stage in the reproducible pipeline.
- dvc.yaml → preprocess: orchestrates · DVC tracks preprocessing as a pipeline step.
- dvc.yaml → train models: orchestrates · DVC tracks training as a pipeline step.
- dvc.yaml → evaluate: orchestrates · DVC tracks evaluation as a pipeline step.
- dvc.yaml → select best: orchestrates · DVC tracks model selection as a pipeline step.
- params.yaml → load data: configures · Shared config influences pipeline execution.
- params.yaml → preprocess: configures · Controls splitting and preprocessing settings.
- params.yaml → train models: configures · Controls model hyperparameters and scaling.
- params.yaml → evaluate: configures · Controls evaluation setup and comparisons.
- raw data → load data: input · Source dataset consumed by the pipeline.
- main.py → load data: runs · Main entrypoint starts the end-to-end batch workflow.
- load data → preprocess: passes data · Loaded dataset flows into preprocessing.
- preprocess → train/test data: writes · Creates train/test CSV artifacts.
- train/test data → train models: consumes · Training reads the prepared training split.
- train/test data → evaluate: consumes · Evaluation uses the held-out test split.
- train models → model pkls: serializes · Saves model files and scaler artifacts.
- train models → evaluate: produces models · Evaluation consumes trained estimators.
- evaluate → metrics: writes · Stores metrics and comparison outputs.
- evaluate → select best: feeds metrics · Selection uses model scores to choose the winner.
- select best → best artifacts: publishes · Emits the final model and scaler for inference.
- select best → Reports: documents · Outputs report artifacts for review and governance.
- best artifacts → demo.py: loaded by · Demo script uses the persisted best model and scaler.
- main.py → train models: runs · Main entrypoint advances through the pipeline stages.
- main.py → evaluate: runs · Main entrypoint advances through the pipeline stages.
- main.py → select best: runs · Main entrypoint advances through the pipeline stages.