# kubeflow/pipelines architecture

> Architecture diagram of the GitHub repository kubeflow/pipelines, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kubeflow/pipelines
- Repository: https://github.com/kubeflow/pipelines

## Overview

##### Project Type & Purpose
- Kubeflow Pipelines is a full-stack, Kubernetes-native orchestration system for ML workflows.
- It comprises a UI (React/TypeScript), a set of backend microservices (Go and Python), Kubernetes custom controllers (CRDs), and client SDKs/CLIs.
- Goals: end-to-end orchestration of ML pipelines, metadata management, artifact tracking, and interactive visualization.

##### High-Level Architecture Components
a. Frontend Layer
– “frontend/”: React-based UI, static web assets, proxy server.
– Communicates via REST/gRPC-web to the API server.
b. API Layer
– “backend/src/apiserver”: ML-Pipeline API server (gRPC + HTTP gateways), authentication, validation, CRD conversion.
– Exposes v1beta1/v2beta1 APIs.
c. Control Plane
– “backend/src/cache”: Cache admission webhook + cache service for template caching.
– “backend/src/crd”: Kubernetes controllers for ScheduledWorkflows and Viewer CRDs.
d. Agents & Executors
– “backend/src/agent/persistence”: Persistence agent that watches Argo workflows and writes status to DB.
– “backend/src/v2/launcher‐v2”: Emissary executor and launcher for v2.
– “backend/src/v2/driver”: Driver orchestrating step execution for v2 pipelines.
– “backend/src/v2/compiler”: Translates Python DSL into Argo workflow specs.
e. Data & Storage
– Metadata Store: ML-Metadata (gRPC service, protobuf under “third_party/ml-metadata”).
– SQL Database: Postgres/MySQL via “backend/src/apiserver/storage/*.go”.
– Object Store: Minio (S3 API) for artifacts.
– Cache Service: SQL + in-memory caches.
f. Kubernetes Components
– CRDs (Pipeline, PipelineVersion, ScheduledWorkflow, Viewer).
– Argo Workflow Controller invoked to execute pipeline DAG tasks.
– Deployments/Services for API server, cache, agents, metadata, webhook.
g. Clients & SDKs
– Python SDK (“sdk/python/kfp”), Go and TS clients under “api/v2alpha1” and code-gen folders.
– CLI commands in “sdk/python/kfp/cli”.

##### Interactions & Data Flow
1. User action (UI or SDK CLI) → REST/gRPC call to API server.
2. API server → stores pipeline spec in DB + creates PipelineVersion CRD.
3. API server → triggers Argo Workflow via K8s client.
4. Argo → launches Pods (Emissary executor) for each task.
5. Persistence Agent → watches Argo pod events, writes run status to SQL DB & ML-Metadata.
6. Visualization Service → queries metadata store + object store to render metrics/graphs.
7. Cache Service → admission webhook intercepts pipeline template CRs to cache compiled templates.

##### Architectural Patterns & Principles
- Microservices: independent services for API, cache, metadata, agents, visualization.
- Kubernetes-Native Operator Pattern: use CRDs + controllers for custom resources (ScheduledWorkflow, Viewer).
- Event-Driven: agents subscribe to workflow events.
- Layered Separation: UI ←→ API ←→ control plane ←→ data stores.
- Pluggable Executors: v1 (Argo) vs v2 (Emissary) executor abstraction.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Entities
    subgraph "External Services" 
        GH["GitHub Repo"]:::external
        CR["Container Registry"]:::external
    end

    %% User/UI Layer
    subgraph "User & Clients Layer"
        direction TB
        UI["Web UI (React/TS)"]:::ui
        SDK["Python SDK & CLI"]:::ui
        CLIGoTS["Go/TS Client"]:::ui
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        APIServer["API Server (gRPC+HTTP)"]:::api
        VisService["Visualization Service"]:::api
        Webhook["Admission Webhook"]:::api
        ClientMgr["Client Manager (v1/v2)"]:::api
    end

    %% Control Plane
    subgraph "Control Plane"
        direction TB
        CacheSvc["Cache Service & Webhook"]:::control
        CRDSW["ScheduledWorkflow Controller"]:::control
        CRDViewer["Viewer Controller"]:::control
    end

    %% Execution Plane
    subgraph "Execution Plane"
        direction TB
        ArgoCtrl["Argo Workflow Controller"]:::exec
        V2Compiler["v2 Compiler"]:::exec
        V2Driver["v2 Driver"]:::exec
        V2Launcher["v2 Launcher/Emissary"]:::exec
        PersistAgent["Persistence Agent"]:::exec
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        SQLDB["SQL Database"]:::data
        MLMD["ML-Metadata Store"]:::data
        ObjStore["Artifact Store (Minio)"]:::data
    end

    %% Kubernetes Cluster Boundary
    subgraph "Kubernetes Cluster" 
        direction TB
        APIServer
        VisService
        Webhook
        CacheSvc
        CRDSW
        CRDViewer
        ArgoCtrl
        V2Compiler
        V2Driver
        V2Launcher
        PersistAgent
    end

    %% Relationships
    UI -->|"HTTP/gRPC-web"| APIServer
    SDK -->|"gRPC"| APIServer
    CLIGoTS -->|"HTTP/gRPC"| APIServer

    GH -->|CI/CD Pipelines| CR["Container Registry"]
    CR -->|Deploy Images| APIServer

    APIServer -->|"Create PipelineVersion CRD"| CRDSW
    APIServer -->|"Trigger Workflow via Kubernetes API"| ArgoCtrl
    APIServer -->|"Write Visualization Data"| VisService
    APIServer -->|"Authenticate & Validate"| Webhook

    CacheSvc -->|"Intercept CRD"| CRDSW
    CacheSvc -->|"Cache Templates"| SQLDB

    CRDSW -->|"Watch CRD"| ArgoCtrl
    CRDViewer -->|"Watch Viewer CRD"| VisService

    ArgoCtrl -->|"Launch Pods"| V2Launcher
    V2Compiler -->|"Compile DSL to Argo Spec"| ArgoCtrl
    V2Driver -->|"Orchestrate Steps"| V2Launcher

    V2Launcher -->|"Execute Tasks"| PersistAgent
    PersistAgent -->|"Write Status"| SQLDB
    PersistAgent -->|"Write Metadata"| MLMD

    VisService -->|"Query Metadata"| MLMD
    VisService -->|"Fetch Artifacts"| ObjStore

    APIServer -->|"Store Specs"| SQLDB
    APIServer -->|"Store Artifacts"| ObjStore

    %% Click Events
    click UI "https://github.com/kubeflow/pipelines/tree/master/frontend/"
    click SDK "https://github.com/kubeflow/pipelines/tree/master/sdk/python/kfp/cli/"
    click CLIGoTS "https://github.com/kubeflow/pipelines/tree/master/api/v2alpha1/"
    click APIServer "https://github.com/kubeflow/pipelines/tree/master/backend/src/apiserver/"
    click Webhook "https://github.com/kubeflow/pipelines/tree/master/backend/src/apiserver/webhook/"
    click VisService "https://github.com/kubeflow/pipelines/tree/master/backend/src/apiserver/visualization/"
    click CacheSvc "https://github.com/kubeflow/pipelines/tree/master/backend/src/cache/"
    click CRDSW "https://github.com/kubeflow/pipelines/tree/master/backend/src/crd/controller/scheduledworkflow/"
    click CRDViewer "https://github.com/kubeflow/pipelines/tree/master/backend/src/crd/controller/viewer/"
    click PersistAgent "https://github.com/kubeflow/pipelines/tree/master/backend/src/agent/persistence/"
    click V2Compiler "https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/compiler/argocompiler/"
    click V2Driver "https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/driver/"
    click V2Launcher "https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/cmd/launcher-v2/"
    click ClientMgr "https://github.com/kubeflow/pipelines/tree/master/backend/src/common/client/api_server/"
    click MLMD "https://github.com/kubeflow/pipelines/tree/master/third_party/ml-metadata/"
    click SQLDB "https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/storage/*.go"
    click ObjStore "https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/storage/minio_client.go"
    click GH "https://github.com/kubeflow/pipelines/tree/master/.github/workflows/"
    click CR "https://github.com/kubeflow/pipelines/tree/master/Container Registry"
    classDef ui fill:#D4E4FF,stroke:#0366D6,color:#0366D6;
    classDef api fill:#E6F4EA,stroke:#1B8F00,color:#1B8F00;
    classDef control fill:#FFF4E5,stroke:#D9822B,color:#D9822B;
    classDef exec fill:#FFF0F6,stroke:#D6336C,color:#D6336C;
    classDef data fill:#F3E5F5,stroke:#6A1B9A,color:#6A1B9A;
    classDef external fill:#E0E0E0,stroke:#888888,color:#333333;
```
