vickyguo0907/ml_pipeline

How vickyguo0907/ml_pipeline works

This is a config-driven ML pipeline template, with Airflow orchestrating data from landing files through validation, preparation, model training, evaluation, and registry staging; FastAPI serves promoted models. The main path is shared by tabular pipelines, with finance and forecasting implementations for distinct problem domains. Profiling, unsupervised exploration, drift reports, and benchmark comparisons are additional capabilities. The README documents the product workflow; the source index gives some module relationships, but excerpts are incomplete, so unsupported internal calls are omitted.

Main entry points: dag_factory.py Airflow DAG factory · clean.py Tabular cleaning · features.py Tabular features · train.py Tabular training

Components 31

Orchestration

Airflow DAG factory src/dags/dag_factory.py

Shared ML pipeline

Ingestion src/ingest.py
Raw validation src/validate.py
Tabular cleaning src/clean.py
Tabular features src/features.py
Feature validation src/schemas/features.py
Tabular training src/train.py
Tabular evaluation src/evaluate.py
Data profiling src/profile.py
Unsupervised exploration src/explore.py
Drift reporting src/monitoring.py

Finance pipelines

Finance cleaning src/finance/clean_finance.py
Finance features src/finance/features_finance.py
Finance training src/finance/train_finance.py
Finance evaluation src/finance/evaluate_finance.py
Rank IC src/finance/rank_ic.py

Forecasting pipelines

Forecast cleaning src/forecasting/clean_forecast.py
Forecast features src/forecasting/features_forecast.py
Weather processing src/forecasting/weather.py
Forecast training src/forecasting/train_forecast.py
Rolling-origin evaluation src/forecasting/rolling_origin.py
Forecast models src/forecasting/direct_forecast.py

Serving and support

MLflow registry src/utils/model_registry.py
Prediction API src/serve.py
Configuration models src/utils/config.py
Run artifacts and manifests src/utils/io.py
Shared transforms src/utils/transforms.py
Model diagnostics src/utils/diagnostics.py

Other components

Pipeline operator
Landing data
Predictions and reports

Connections 34

Pipeline operator → Airflow DAG factory: triggers run
Landing data → Ingestion: stages files
Airflow DAG factory → Ingestion: orchestrates src/dags/dag_factory.py
Ingestion → Raw validation: validates raw
Airflow DAG factory → Tabular cleaning: orchestrates src/dags/dag_factory.py
Airflow DAG factory → Tabular features: orchestrates src/dags/dag_factory.py
Airflow DAG factory → Feature validation: validates features src/dags/dag_factory.py
Airflow DAG factory → Tabular training: orchestrates src/dags/dag_factory.py
Airflow DAG factory → Tabular evaluation: orchestrates src/dags/dag_factory.py
Tabular training → MLflow registry: logs models src/train.py
Tabular evaluation → MLflow registry: registers models src/evaluate.py
Airflow DAG factory → Data profiling: optionally runs src/dags/dag_factory.py
Airflow DAG factory → Unsupervised exploration: optionally runs src/dags/dag_factory.py
Airflow DAG factory → Drift reporting: optionally reports drift src/dags/dag_factory.py
Airflow DAG factory → Finance cleaning: dispatches finance src/dags/dag_factory.py
Airflow DAG factory → Finance features: dispatches finance src/dags/dag_factory.py
Airflow DAG factory → Finance training: dispatches finance src/dags/dag_factory.py
Airflow DAG factory → Finance evaluation: dispatches finance src/dags/dag_factory.py
Finance evaluation → Rank IC: computes rank IC src/finance/evaluate_finance.py
Airflow DAG factory → Forecast cleaning: dispatches forecast src/dags/dag_factory.py
Airflow DAG factory → Forecast features: dispatches forecast src/dags/dag_factory.py
Airflow DAG factory → Forecast training: dispatches forecast src/dags/dag_factory.py
Forecast training → Rolling-origin evaluation: evaluates origins src/forecasting/train_forecast.py
Rolling-origin evaluation → Forecast models: uses forecast logic src/forecasting/rolling_origin.py
Forecast cleaning → Weather processing: aggregates weather src/forecasting/clean_forecast.py
Forecast features → Weather processing: builds weather features src/forecasting/features_forecast.py
Pipeline operator → Prediction API: requests prediction
Prediction API → MLflow registry: loads promoted model
Prediction API → Predictions and reports: returns prediction
Tabular cleaning → Run artifacts and manifests: reads and writes src/clean.py
Tabular features → Run artifacts and manifests: reads and writes src/features.py
Tabular cleaning → Configuration models: loads settings src/clean.py
Tabular features → Shared transforms: applies transforms src/features.py
Tabular training → Model diagnostics: uses diagnostics src/train.py

Drawn by GitDiagram on November 11, 2026 from the repository's file tree, README and source files. Read as Markdown ·