How kubeflow/pipelines works
Project Type & Purpose
• Kubeflow Pipelines is a full-stack, Kubernetes-native orchestration system for ML workflows.
• It comprises a UI (React/TypeScript), a set of backend microservices (Go and Python), Kubernetes custom controllers (CRDs), and client SDKs/CLIs.
• Goals: end-to-end orchestration of ML pipelines, metadata management, artifact tracking, and interactive visualization.
High-Level Architecture Components
a. Frontend Layer
– “frontend/”: React-based UI, static web assets, proxy server.
– Communicates via REST/gRPC-web to the API server.
b. API Layer
– “backend/src/apiserver”: ML-Pipeline API server (gRPC + HTTP gateways), authentication, validation, CRD conversion.
– Exposes v1beta1/v2beta1 APIs.
c. Control Plane
– “backend/src/cache”: Cache admission webhook + cache service for template caching.
– “backend/src/crd”: Kubernetes controllers for ScheduledWorkflows and Viewer CRDs.
d. Agents & Executors
– “backend/src/agent/persistence”: Persistence agent that watches Argo workflows and writes status to DB.
– “backend/src/v2/launcher‐v2”: Emissary executor and launcher for v2.
– “backend/src/v2/driver”: Driver orchestrating step execution for v2 pipelines.
– “backend/src/v2/compiler”: Translates Python DSL into Argo workflow specs.
e. Data & Storage
– Metadata Store: ML-Metadata (gRPC service, protobuf under “third_party/ml-metadata”).
– SQL Database: Postgres/MySQL via “backend/src/apiserver/storage/*.go”.
– Object Store: Minio (S3 API) for artifacts.
– Cache Service: SQL + in-memory caches.
f. Kubernetes Components
– CRDs (Pipeline, PipelineVersion, ScheduledWorkflow, Viewer).
– Argo Workflow Controller invoked to execute pipeline DAG tasks.
– Deployments/Services for API server, cache, agents, metadata, webhook.
g. Clients & SDKs
– Python SDK (“sdk/python/kfp”), Go and TS clients under “api/v2alpha1” and code-gen folders.
– CLI commands in “sdk/python/kfp/cli”.
Interactions & Data Flow
1. User action (UI or SDK CLI) → REST/gRPC call to API server.
2. API server → stores pipeline spec in DB + creates PipelineVersion CRD.
3. API server → triggers Argo Workflow via K8s client.
4. Argo → launches Pods (Emissary executor) for each task.
5. Persistence Agent → watches Argo pod events, writes run status to SQL DB & ML-Metadata.
6. Visualization Service → queries metadata store + object store to render metrics/graphs.
7. Cache Service → admission webhook intercepts pipeline template CRs to cache compiled templates.
Architectural Patterns & Principles
• Microservices: independent services for API, cache, metadata, agents, visualization.
• Kubernetes-Native Operator Pattern: use CRDs + controllers for custom resources (ScheduledWorkflow, Viewer).
• Event-Driven: agents subscribe to workflow events.
• Layered Separation: UI ←→ API ←→ control plane ←→ data stores.
• Pluggable Executors: v1 (Argo) vs v2 (Emissary) executor abstraction.