How kimutai-chelanga/production-ready-mlops-on-gcp works
This is a Google Cloud MLOps reference implementation. Users deploy environment infrastructure, launch Vertex training or prediction pipelines, and receive registered models or batch predictions. The training workflow covers BigQuery preprocessing and splitting, data extraction, hyperparameter tuning, model training and evaluation, then champion-model selection and registry upload. Reusable KFP components implement these stages; a separate utility triggers compiled pipeline runs. The README also documents Terraform-managed infrastructure and prediction capabilities. Some prediction and infrastructure internals were not sampled, so their detailed links are omitted.
Main entry points: training.py Training pipeline · extract_table_to_gcs_op.py Table extraction · prediction.py Prediction pipeline · trigger_pipeline.py Run trigger
Components 19
Pipeline orchestration
- Run trigger
pipelines/src/pipelines/utils/trigger_pipeline.py - Training pipeline
pipelines/src/pipelines/training.py - Prediction pipeline
pipelines/src/pipelines/prediction.py - SQL templating
pipelines/src/pipelines/utils/query.py
Reusable components
- Table extraction
components/src/components/extract_table_to_gcs_op.py - Training arguments
components/src/components/get_training_args_dict_op.py - Worker spec
components/src/components/get_workerpool_spec_op.py - Tuning results
components/src/components/get_hyperparameter_tuning_results_op.py - Training results
components/src/components/get_custom_job_results_op.py - Champion selection
components/src/components/upload_best_model_op.py - Model lookup
components/src/components/lookup_model_op.py - Batch prediction
components/src/components/model_batch_predict_op.py
Model training
- Training entry point
model/trainer/task.py - Training logic
model/trainer/model.py
Cloud services and infrastructure
- BigQuery
- Cloud Storage
- Vertex AI
- Cloud infrastructure
terraform/environments
Other components
- Pipeline user
Connections 21
- Pipeline user → Run trigger: starts run
README.md - Run trigger → Vertex AI: submits job
pipelines/src/pipelines/utils/trigger_pipeline.py - Training pipeline → SQL templating: renders SQL
pipelines/src/pipelines/training.py - Training pipeline → BigQuery: runs queries
pipelines/src/pipelines/training.py - Training pipeline → Table extraction: extracts tables
pipelines/src/pipelines/training.py - Table extraction → BigQuery: reads table
components/src/components/extract_table_to_gcs_op.py - Table extraction → Cloud Storage: writes dataset
components/src/components/extract_table_to_gcs_op.py - Training pipeline → Training arguments: builds arguments
pipelines/src/pipelines/training.py - Training pipeline → Worker spec: builds worker specs
pipelines/src/pipelines/training.py - Training pipeline → Tuning results: gets tuning results
pipelines/src/pipelines/training.py - Training pipeline → Training results: gets job results
pipelines/src/pipelines/training.py - Training pipeline → Champion selection: uploads candidate
pipelines/src/pipelines/training.py - Training pipeline → Training entry point: runs training image
README.md - Training entry point → Training logic: invokes model code
model/trainer/task.py - Training logic → Cloud Storage: persists artifacts
model/trainer/model.py - Champion selection → Vertex AI: registers model
components/src/components/upload_best_model_op.py - Prediction pipeline → Model lookup: uses model lookup
pipelines/src/pipelines/prediction.py - Prediction pipeline → Batch prediction: runs batch prediction
pipelines/src/pipelines/prediction.py - Model lookup → Vertex AI: lists models
components/src/components/lookup_model_op.py - Batch prediction → Vertex AI: submits prediction job
components/src/components/model_batch_predict_op.py - Cloud infrastructure → Vertex AI: provisions services