# kubeflow/kale architecture

> Architecture diagram of the GitHub repository kubeflow/kale, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kubeflow/kale
- Repository: https://github.com/kubeflow/kale

## Overview

Kale turns tagged Jupyter notebooks into Kubeflow Pipelines: users configure cell metadata in JupyterLab, compile and launch a run, then use deployment/status views. The Python runtime processes notebook cells into a dependency-aware pipeline, compiles it to KFP DSL, and interacts with Kubeflow and Kubernetes. The README documents the principal workflow and capabilities; source-index references support many internal dependencies, but unsampled files limit certainty about some runtime links.

Main entry points: [`kale/processors/nbprocessor.py`](https://github.com/kubeflow/kale/blob/main/kale/processors/nbprocessor.py) (Notebook processor), [`kale/pipeline.py`](https://github.com/kubeflow/kale/blob/main/kale/pipeline.py) (Pipeline model), [`kale/compiler.py`](https://github.com/kubeflow/kale/blob/main/kale/compiler.py) (KFP DSL compiler), [`kale/rpc/nb.py`](https://github.com/kubeflow/kale/blob/main/kale/rpc/nb.py) (Notebook RPC operations)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_notebook_ui["Notebook authoring"]
  node_cell_editor["Cell metadata editor"]
  node_cell_tags["Cell tags and metadata<br/>[TagsUtils.ts]"]
  node_notebook_loader["Notebook state loader"]
  node_commands["Notebook commands<br/>[Commands.ts]"]
  node_deployment_ui["Deploy and status UI<br/>[DeployProgress.tsx]"]
  node_rpc["Jupyter RPC handlers<br/>[handlers.py]"]
  node_nb_rpc["Notebook RPC operations<br/>[nb.py]"]
end

subgraph group_pipeline_build["Pipeline construction"]
  node_notebook_processor["Notebook processor<br/>[nbprocessor.py]"]
  node_ast_analysis["Code and dependency analysis<br/>[astutils.py]"]
  node_dependency_graph["Pipeline graph utilities<br/>[graphutils.py]"]
  node_pipeline_model["Pipeline model<br/>[pipeline.py]"]
  node_step_model["Step model<br/>[step.py]"]
  node_compiler["KFP DSL compiler<br/>[compiler.py]"]
end

subgraph group_runtime_integrations["Execution integrations"]
  node_kfp_client["KFP client factory"]
  node_kfp_utils["KFP operations<br/>[kfputils.py]"]
  node_kubernetes["Kubernetes utilities<br/>[k8sutils.py]"]
  node_pod_utils["Pod utilities<br/>[podutils.py]"]
  node_katib_rpc["Katib RPC<br/>[katib.py]"]
  node_katib_utils["Katib utilities<br/>[katibutils.py]"]
end

subgraph group_shared_services["Shared services"]
  node_validation["Configuration validation<br/>[validators.py]"]
  node_marshal["Object serialization<br/>[backend.py]"]
end

node_author(("Notebook user"))
node_kfp_service{{"Kubeflow Pipelines"}}
node_kubernetes_service{{"Kubernetes cluster"}}

node_author -->|"tags cells"| node_cell_editor
node_cell_editor -->|"updates tags"| node_cell_tags
node_notebook_loader -->|"loads notebook"| node_commands
node_commands -.->|"requests backend"| node_rpc
node_rpc -.->|"dispatches operations"| node_nb_rpc
node_author -->|"starts compile and run"| node_deployment_ui
node_deployment_ui -->|"uses commands"| node_commands
node_nb_rpc -->|"processes notebook"| node_notebook_processor
node_notebook_processor -->|"analyzes code"| node_ast_analysis
node_notebook_processor -->|"builds dependencies"| node_dependency_graph
node_notebook_processor -->|"creates pipeline"| node_pipeline_model
node_pipeline_model -->|"stores steps"| node_step_model
node_pipeline_model -->|"validates configuration"| node_validation
node_compiler -->|"compiles pipeline"| node_pipeline_model
node_compiler -->|"emits step tasks"| node_step_model
node_step_model -->|"analyzes source"| node_ast_analysis
node_nb_rpc -->|"serializes objects"| node_marshal
node_compiler -->|"uses KFP operations"| node_kfp_utils
node_kfp_utils -->|"creates client"| node_kfp_client
node_kfp_utils -.->|"calls KFP"| node_kfp_service
node_pod_utils -->|"gets API client"| node_kubernetes
node_katib_rpc -->|"uses helpers"| node_katib_utils
node_katib_rpc -->|"uses pod helpers"| node_pod_utils
node_kubernetes -.->|"accesses cluster"| node_kubernetes_service

click node_cell_editor "https://github.com/kubeflow/kale/blob/main/labextension/src/widgets/cell-metadata/CellMetadataEditor.tsx"
click node_cell_tags "https://github.com/kubeflow/kale/blob/main/labextension/src/lib/TagsUtils.ts"
click node_notebook_loader "https://github.com/kubeflow/kale/blob/main/labextension/src/widgets/hooks/useNotebookLoader.ts"
click node_commands "https://github.com/kubeflow/kale/blob/main/labextension/src/lib/Commands.ts"
click node_deployment_ui "https://github.com/kubeflow/kale/blob/main/labextension/src/widgets/deploys-progress/DeployProgress.tsx"
click node_rpc "https://github.com/kubeflow/kale/blob/main/jupyterlab_kubeflow_kale/handlers.py"
click node_nb_rpc "https://github.com/kubeflow/kale/blob/main/kale/rpc/nb.py"
click node_notebook_processor "https://github.com/kubeflow/kale/blob/main/kale/processors/nbprocessor.py"
click node_ast_analysis "https://github.com/kubeflow/kale/blob/main/kale/common/astutils.py"
click node_dependency_graph "https://github.com/kubeflow/kale/blob/main/kale/common/graphutils.py"
click node_pipeline_model "https://github.com/kubeflow/kale/blob/main/kale/pipeline.py"
click node_step_model "https://github.com/kubeflow/kale/blob/main/kale/step.py"
click node_compiler "https://github.com/kubeflow/kale/blob/main/kale/compiler.py"
click node_validation "https://github.com/kubeflow/kale/blob/main/kale/config/validators.py"
click node_marshal "https://github.com/kubeflow/kale/blob/main/kale/marshal/backend.py"
click node_kfp_client "https://github.com/kubeflow/kale/blob/main/kale/common/kfp_client_factory.py"
click node_kfp_utils "https://github.com/kubeflow/kale/blob/main/kale/common/kfputils.py"
click node_kubernetes "https://github.com/kubeflow/kale/blob/main/kale/common/k8sutils.py"
click node_pod_utils "https://github.com/kubeflow/kale/blob/main/kale/common/podutils.py"
click node_katib_rpc "https://github.com/kubeflow/kale/blob/main/kale/rpc/katib.py"
click node_katib_utils "https://github.com/kubeflow/kale/blob/main/kale/common/katibutils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cell_editor,node_cell_tags,node_notebook_loader,node_commands,node_deployment_ui,node_rpc,node_nb_rpc,node_author toneBlue
class node_notebook_processor,node_ast_analysis,node_dependency_graph,node_pipeline_model,node_step_model,node_compiler toneAmber
class node_kfp_client,node_kfp_utils,node_kubernetes,node_pod_utils,node_katib_rpc,node_katib_utils toneMint
class node_validation,node_marshal toneRose
class node_kfp_service,node_kubernetes_service toneIndigo
```

## Components

### Notebook authoring

- [Cell metadata editor](https://github.com/kubeflow/kale/blob/main/labextension/src/widgets/cell-metadata/CellMetadataEditor.tsx): `labextension/src/widgets/cell-metadata/CellMetadataEditor.tsx`
- [Cell tags and metadata](https://github.com/kubeflow/kale/blob/main/labextension/src/lib/TagsUtils.ts): `labextension/src/lib/TagsUtils.ts`
- [Notebook state loader](https://github.com/kubeflow/kale/blob/main/labextension/src/widgets/hooks/useNotebookLoader.ts): `labextension/src/widgets/hooks/useNotebookLoader.ts`
- [Notebook commands](https://github.com/kubeflow/kale/blob/main/labextension/src/lib/Commands.ts): `labextension/src/lib/Commands.ts`
- [Deploy and status UI](https://github.com/kubeflow/kale/blob/main/labextension/src/widgets/deploys-progress/DeployProgress.tsx): `labextension/src/widgets/deploys-progress/DeployProgress.tsx`
- [Jupyter RPC handlers](https://github.com/kubeflow/kale/blob/main/jupyterlab_kubeflow_kale/handlers.py): `jupyterlab_kubeflow_kale/handlers.py`
- [Notebook RPC operations](https://github.com/kubeflow/kale/blob/main/kale/rpc/nb.py): `kale/rpc/nb.py`

### Pipeline construction

- [Notebook processor](https://github.com/kubeflow/kale/blob/main/kale/processors/nbprocessor.py): `kale/processors/nbprocessor.py`
- [Code and dependency analysis](https://github.com/kubeflow/kale/blob/main/kale/common/astutils.py): `kale/common/astutils.py`
- [Pipeline graph utilities](https://github.com/kubeflow/kale/blob/main/kale/common/graphutils.py): `kale/common/graphutils.py`
- [Pipeline model](https://github.com/kubeflow/kale/blob/main/kale/pipeline.py): `kale/pipeline.py`
- [Step model](https://github.com/kubeflow/kale/blob/main/kale/step.py): `kale/step.py`
- [KFP DSL compiler](https://github.com/kubeflow/kale/blob/main/kale/compiler.py): `kale/compiler.py`

### Execution integrations

- [KFP client factory](https://github.com/kubeflow/kale/blob/main/kale/common/kfp_client_factory.py): `kale/common/kfp_client_factory.py`
- [KFP operations](https://github.com/kubeflow/kale/blob/main/kale/common/kfputils.py): `kale/common/kfputils.py`
- [Kubernetes utilities](https://github.com/kubeflow/kale/blob/main/kale/common/k8sutils.py): `kale/common/k8sutils.py`
- [Pod utilities](https://github.com/kubeflow/kale/blob/main/kale/common/podutils.py): `kale/common/podutils.py`
- [Katib RPC](https://github.com/kubeflow/kale/blob/main/kale/rpc/katib.py): `kale/rpc/katib.py`
- [Katib utilities](https://github.com/kubeflow/kale/blob/main/kale/common/katibutils.py): `kale/common/katibutils.py`

### Shared services

- [Configuration validation](https://github.com/kubeflow/kale/blob/main/kale/config/validators.py): `kale/config/validators.py`
- [Object serialization](https://github.com/kubeflow/kale/blob/main/kale/marshal/backend.py): `kale/marshal/backend.py`

### Other components

- **Notebook user**
- **Kubeflow Pipelines**
- **Kubernetes cluster**

## Connections

- Notebook user → Cell metadata editor: tags cells (evidence: `README.md`)
- Cell metadata editor → Cell tags and metadata: updates tags (evidence: `labextension/src/widgets/cell-metadata/CellMetadataEditor.tsx`)
- Notebook state loader → Notebook commands: loads notebook (evidence: `labextension/src/widgets/hooks/useNotebookLoader.ts`)
- Notebook commands → Jupyter RPC handlers: requests backend
- Jupyter RPC handlers → Notebook RPC operations: dispatches operations
- Notebook user → Deploy and status UI: starts compile and run (evidence: `README.md`)
- Deploy and status UI → Notebook commands: uses commands (evidence: `labextension/src/lib/Commands.ts`)
- Notebook RPC operations → Notebook processor: processes notebook (evidence: `kale/rpc/nb.py`)
- Notebook processor → Code and dependency analysis: analyzes code (evidence: `kale/processors/nbprocessor.py`)
- Notebook processor → Pipeline graph utilities: builds dependencies (evidence: `kale/processors/nbprocessor.py`)
- Notebook processor → Pipeline model: creates pipeline (evidence: `kale/processors/nbprocessor.py`)
- Pipeline model → Step model: stores steps (evidence: `kale/pipeline.py`)
- Pipeline model → Configuration validation: validates configuration (evidence: `kale/pipeline.py`)
- KFP DSL compiler → Pipeline model: compiles pipeline (evidence: `kale/compiler.py`)
- KFP DSL compiler → Step model: emits step tasks (evidence: `kale/compiler.py`)
- Step model → Code and dependency analysis: analyzes source (evidence: `kale/step.py`)
- Notebook RPC operations → Object serialization: serializes objects (evidence: `kale/rpc/nb.py`)
- KFP DSL compiler → KFP operations: uses KFP operations (evidence: `kale/compiler.py`)
- KFP operations → KFP client factory: creates client (evidence: `kale/common/kfputils.py`)
- KFP operations → Kubeflow Pipelines: calls KFP
- Pod utilities → Kubernetes utilities: gets API client (evidence: `kale/common/podutils.py`)
- Katib RPC → Katib utilities: uses helpers (evidence: `kale/rpc/katib.py`)
- Katib RPC → Pod utilities: uses pod helpers (evidence: `kale/rpc/katib.py`)
- Kubernetes utilities → Kubernetes cluster: accesses cluster
