# kimutai-chelanga/production-ready-mlops-on-gcp architecture

> Architecture diagram of the GitHub repository kimutai-chelanga/production-ready-mlops-on-gcp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kimutai-chelanga/production-ready-mlops-on-gcp
- Repository: https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp

## Overview

This is a Google Cloud MLOps reference implementation. Users deploy environment infrastructure, launch Vertex training or prediction pipelines, and receive registered models or batch predictions. The training workflow covers BigQuery preprocessing and splitting, data extraction, hyperparameter tuning, model training and evaluation, then champion-model selection and registry upload. Reusable KFP components implement these stages; a separate utility triggers compiled pipeline runs. The README also documents Terraform-managed infrastructure and prediction capabilities. Some prediction and infrastructure internals were not sampled, so their detailed links are omitted.

Main entry points: [`pipelines/src/pipelines/training.py`](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/training.py) (Training pipeline), [`components/src/components/extract_table_to_gcs_op.py`](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/extract_table_to_gcs_op.py) (Table extraction), [`pipelines/src/pipelines/prediction.py`](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/prediction.py) (Prediction pipeline), [`pipelines/src/pipelines/utils/trigger_pipeline.py`](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/utils/trigger_pipeline.py) (Run trigger)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_trigger["Run trigger"]
  node_training["Training pipeline<br/>[training.py]"]
  node_prediction["Prediction pipeline<br/>[prediction.py]"]
  node_query["SQL templating<br/>[query.py]"]
end

subgraph group_components["Reusable components"]
  node_extract["Table extraction"]
  node_args["Training arguments"]
  node_worker["Worker spec"]
  node_tuning_results["Tuning results"]
  node_job_results["Training results"]
  node_model_upload["Champion selection"]
  node_model_lookup["Model lookup<br/>[lookup_model_op.py]"]
  node_batch_predict["Batch prediction"]
end

subgraph group_model["Model training"]
  node_trainer_task["Training entry point<br/>[task.py]"]
  node_trainer["Training logic<br/>[model.py]"]
end

subgraph group_cloud["Cloud services and infrastructure"]
  node_bigquery[("BigQuery")]
  node_gcs[("Cloud Storage")]
  node_vertex["Vertex AI"]
  node_terraform["Cloud infrastructure"]
end

node_user(("Pipeline user"))

node_user -->|"starts run"| node_trigger
node_trigger -->|"submits job"| node_vertex
node_training -->|"renders SQL"| node_query
node_training -->|"runs queries"| node_bigquery
node_training -->|"extracts tables"| node_extract
node_extract -->|"reads table"| node_bigquery
node_extract -->|"writes dataset"| node_gcs
node_training -->|"builds arguments"| node_args
node_training -->|"builds worker specs"| node_worker
node_training -->|"gets tuning results"| node_tuning_results
node_training -->|"gets job results"| node_job_results
node_training -->|"uploads candidate"| node_model_upload
node_training -->|"runs training image"| node_trainer_task
node_trainer_task -->|"invokes model code"| node_trainer
node_trainer -->|"persists artifacts"| node_gcs
node_model_upload -->|"registers model"| node_vertex
node_prediction -.->|"uses model lookup"| node_model_lookup
node_prediction -.->|"runs batch prediction"| node_batch_predict
node_model_lookup -->|"lists models"| node_vertex
node_batch_predict -->|"submits prediction job"| node_vertex
node_terraform -.->|"provisions services"| node_vertex

click node_trigger "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/utils/trigger_pipeline.py"
click node_training "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/training.py"
click node_prediction "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/prediction.py"
click node_query "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/utils/query.py"
click node_extract "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/extract_table_to_gcs_op.py"
click node_args "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_training_args_dict_op.py"
click node_worker "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_workerpool_spec_op.py"
click node_tuning_results "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_hyperparameter_tuning_results_op.py"
click node_job_results "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_custom_job_results_op.py"
click node_model_upload "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/upload_best_model_op.py"
click node_model_lookup "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/lookup_model_op.py"
click node_batch_predict "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/model_batch_predict_op.py"
click node_trainer_task "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/model/trainer/task.py"
click node_trainer "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/model/trainer/model.py"
click node_terraform "https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/tree/main/terraform/environments"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_trigger,node_training,node_prediction,node_query,node_user toneBlue
class node_extract,node_args,node_worker,node_tuning_results,node_job_results,node_model_upload,node_model_lookup,node_batch_predict toneAmber
class node_trainer_task,node_trainer toneMint
class node_bigquery,node_gcs,node_vertex,node_terraform toneRose
```

## Components

### Pipeline orchestration

- [Run trigger](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/utils/trigger_pipeline.py): `pipelines/src/pipelines/utils/trigger_pipeline.py`
- [Training pipeline](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/training.py): `pipelines/src/pipelines/training.py`
- [Prediction pipeline](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/prediction.py): `pipelines/src/pipelines/prediction.py`
- [SQL templating](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/pipelines/src/pipelines/utils/query.py): `pipelines/src/pipelines/utils/query.py`

### Reusable components

- [Table extraction](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/extract_table_to_gcs_op.py): `components/src/components/extract_table_to_gcs_op.py`
- [Training arguments](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_training_args_dict_op.py): `components/src/components/get_training_args_dict_op.py`
- [Worker spec](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_workerpool_spec_op.py): `components/src/components/get_workerpool_spec_op.py`
- [Tuning results](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_hyperparameter_tuning_results_op.py): `components/src/components/get_hyperparameter_tuning_results_op.py`
- [Training results](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/get_custom_job_results_op.py): `components/src/components/get_custom_job_results_op.py`
- [Champion selection](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/upload_best_model_op.py): `components/src/components/upload_best_model_op.py`
- [Model lookup](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/lookup_model_op.py): `components/src/components/lookup_model_op.py`
- [Batch prediction](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/components/src/components/model_batch_predict_op.py): `components/src/components/model_batch_predict_op.py`

### Model training

- [Training entry point](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/model/trainer/task.py): `model/trainer/task.py`
- [Training logic](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/blob/main/model/trainer/model.py): `model/trainer/model.py`

### Cloud services and infrastructure

- **BigQuery**
- **Cloud Storage**
- **Vertex AI**
- [Cloud infrastructure](https://github.com/kimutai-chelanga/production-ready-mlops-on-gcp/tree/main/terraform/environments): `terraform/environments`

### Other components

- **Pipeline user**

## Connections

- Pipeline user → Run trigger: starts run (evidence: `README.md`)
- Run trigger → Vertex AI: submits job (evidence: `pipelines/src/pipelines/utils/trigger_pipeline.py`)
- Training pipeline → SQL templating: renders SQL (evidence: `pipelines/src/pipelines/training.py`)
- Training pipeline → BigQuery: runs queries (evidence: `pipelines/src/pipelines/training.py`)
- Training pipeline → Table extraction: extracts tables (evidence: `pipelines/src/pipelines/training.py`)
- Table extraction → BigQuery: reads table (evidence: `components/src/components/extract_table_to_gcs_op.py`)
- Table extraction → Cloud Storage: writes dataset (evidence: `components/src/components/extract_table_to_gcs_op.py`)
- Training pipeline → Training arguments: builds arguments (evidence: `pipelines/src/pipelines/training.py`)
- Training pipeline → Worker spec: builds worker specs (evidence: `pipelines/src/pipelines/training.py`)
- Training pipeline → Tuning results: gets tuning results (evidence: `pipelines/src/pipelines/training.py`)
- Training pipeline → Training results: gets job results (evidence: `pipelines/src/pipelines/training.py`)
- Training pipeline → Champion selection: uploads candidate (evidence: `pipelines/src/pipelines/training.py`)
- Training pipeline → Training entry point: runs training image (evidence: `README.md`)
- Training entry point → Training logic: invokes model code (evidence: `model/trainer/task.py`)
- Training logic → Cloud Storage: persists artifacts (evidence: `model/trainer/model.py`)
- Champion selection → Vertex AI: registers model (evidence: `components/src/components/upload_best_model_op.py`)
- Prediction pipeline → Model lookup: uses model lookup (evidence: `pipelines/src/pipelines/prediction.py`)
- Prediction pipeline → Batch prediction: runs batch prediction (evidence: `pipelines/src/pipelines/prediction.py`)
- Model lookup → Vertex AI: lists models (evidence: `components/src/components/lookup_model_op.py`)
- Batch prediction → Vertex AI: submits prediction job (evidence: `components/src/components/model_batch_predict_op.py`)
- Cloud infrastructure → Vertex AI: provisions services
