# kubeflow/pipelines-components architecture

> Architecture diagram of the GitHub repository kubeflow/pipelines-components, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kubeflow/pipelines-components
- Repository: https://github.com/kubeflow/pipelines-components

## Overview

This repository is a Kubeflow Pipelines asset library: reusable components and complete pipeline definitions are the assets users compose into AI workflows. The sampled runtime shows SDG loading data and a flow, validating and generating synthetic data, then writing dataset output and metrics. Separate repository tooling supports asset discovery, documentation generation, validation, and maintenance; these are meaningful repository capabilities rather than runtime stages. Several such modules are indexed but unsampled, so their internal behavior and unsupported connections are not expanded here.

Main entry points: [`scripts/lib/discovery.py`](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/discovery.py) (Asset Discovery), [`scripts/lib/base_image.py`](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/base_image.py) (Image Utilities), [`scripts/lib/kfp_compilation.py`](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/kfp_compilation.py) (KFP Compilation), [`scripts/generate_readme/writer.py`](https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/writer.py) (README Generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_assets["Workflow Assets"]
  node_sdg["Synthetic Data<br/>[component.py]"]
  node_yoda["Yoda Processor<br/>[component.py]"]
  node_sdgpipe["SDG Pipeline<br/>[pipeline.py]"]
  node_assetlib["Reusable Components"]
  node_pipelib["Pipeline Definitions"]
end

subgraph group_catalog["Asset Catalog"]
  node_readmegen["README Generation<br/>[writer.py]"]
  node_category["Category Index"]
  node_content["Content Generator"]
  node_metadata["Metadata Parser<br/>[metadata_parser.py]"]
end

subgraph group_quality["Quality Checks"]
  node_compilecheck["Compile Check<br/>[compile_check.py]"]
  node_componentcheck["Component Validation"]
  node_examplecheck["Example Validation"]
  node_metadatacheck["Metadata Validation"]
  node_packagecheck["Package Validation"]
  node_freshness["Freshness Check"]
end

subgraph group_images["Image Maintenance"]
  node_baseimages["Base Image Validation"]
  node_tagcheck["Image Tag Check"]
  node_imageutils["Image Utilities<br/>[base_image.py]"]
end

subgraph group_shared["Shared Tooling"]
  node_discovery["Asset Discovery<br/>[discovery.py]"]
  node_kfpcompile["KFP Compilation<br/>[kfp_compilation.py]"]
  node_metadatautils["Metadata Utilities<br/>[metadata_utils.py]"]
  node_oci["OCI Tag Validation<br/>[oci.py]"]
  node_parsing["Source Parsing<br/>[parsing.py]"]
end

node_user(("Pipeline Author"))

node_user -->|"uses components"| node_assetlib
node_user -->|"composes pipelines"| node_pipelib
node_readmegen -->|"generates index"| node_category
node_readmegen -->|"generates content"| node_content
node_readmegen -->|"parses metadata"| node_metadata
node_compilecheck -->|"discovers assets"| node_discovery
node_compilecheck -->|"compiles assets"| node_kfpcompile
node_compilecheck -->|"uses metadata"| node_metadatautils
node_componentcheck -->|"discovers assets"| node_discovery
node_componentcheck -->|"compiles assets"| node_kfpcompile
node_examplecheck -->|"discovers assets"| node_discovery
node_examplecheck -->|"compiles examples"| node_kfpcompile
node_examplecheck -->|"parses source"| node_parsing
node_packagecheck -->|"discovers assets"| node_discovery
node_freshness -->|"discovers assets"| node_discovery
node_baseimages -->|"discovers assets"| node_discovery
node_baseimages -->|"checks images"| node_imageutils
node_baseimages -->|"compiles assets"| node_kfpcompile
node_tagcheck -->|"checks tags"| node_imageutils
node_imageutils -->|"compiles assets"| node_kfpcompile
node_imageutils -->|"validates tags"| node_oci
node_imageutils -->|"parses image refs"| node_parsing

click node_sdg "https://github.com/kubeflow/pipelines-components/blob/main/components/data_processing/sdg/component.py"
click node_yoda "https://github.com/kubeflow/pipelines-components/blob/main/components/data_processing/yoda_data_processor/component.py"
click node_sdgpipe "https://github.com/kubeflow/pipelines-components/blob/main/pipelines/data_processing/sdg/pipeline.py"
click node_assetlib "https://github.com/kubeflow/pipelines-components/tree/main/components"
click node_pipelib "https://github.com/kubeflow/pipelines-components/tree/main/pipelines"
click node_readmegen "https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/writer.py"
click node_category "https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/category_index_generator.py"
click node_content "https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/content_generator.py"
click node_metadata "https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/metadata_parser.py"
click node_discovery "https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/discovery.py"
click node_compilecheck "https://github.com/kubeflow/pipelines-components/blob/main/scripts/compile_check/compile_check.py"
click node_componentcheck "https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_components/validate_components.py"
click node_examplecheck "https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_examples/validate_examples.py"
click node_metadatacheck "https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_metadata/validate_metadata.py"
click node_packagecheck "https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_package_entries/validate_package_entries.py"
click node_freshness "https://github.com/kubeflow/pipelines-components/blob/main/scripts/check_component_freshness/check_component_freshness.py"
click node_baseimages "https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_base_images/validate_base_images.py"
click node_tagcheck "https://github.com/kubeflow/pipelines-components/blob/main/scripts/check_base_image_tags/check_base_image_tags.py"
click node_imageutils "https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/base_image.py"
click node_kfpcompile "https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/kfp_compilation.py"
click node_metadatautils "https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/metadata_utils.py"
click node_oci "https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/oci.py"
click node_parsing "https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/parsing.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sdg,node_yoda,node_sdgpipe,node_assetlib,node_pipelib toneBlue
class node_readmegen,node_category,node_content,node_metadata toneAmber
class node_compilecheck,node_componentcheck,node_examplecheck,node_metadatacheck,node_packagecheck,node_freshness toneMint
class node_baseimages,node_tagcheck,node_imageutils toneRose
class node_discovery,node_kfpcompile,node_metadatautils,node_oci,node_parsing,node_user toneIndigo
```

## Components

### Workflow Assets

- [Synthetic Data](https://github.com/kubeflow/pipelines-components/blob/main/components/data_processing/sdg/component.py): `components/data_processing/sdg/component.py`
- [Yoda Processor](https://github.com/kubeflow/pipelines-components/blob/main/components/data_processing/yoda_data_processor/component.py): `components/data_processing/yoda_data_processor/component.py`
- [SDG Pipeline](https://github.com/kubeflow/pipelines-components/blob/main/pipelines/data_processing/sdg/pipeline.py): `pipelines/data_processing/sdg/pipeline.py`
- [Reusable Components](https://github.com/kubeflow/pipelines-components/tree/main/components): `components`
- [Pipeline Definitions](https://github.com/kubeflow/pipelines-components/tree/main/pipelines): `pipelines`

### Asset Catalog

- [README Generation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/writer.py): `scripts/generate_readme/writer.py`
- [Category Index](https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/category_index_generator.py): `scripts/generate_readme/category_index_generator.py`
- [Content Generator](https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/content_generator.py): `scripts/generate_readme/content_generator.py`
- [Metadata Parser](https://github.com/kubeflow/pipelines-components/blob/main/scripts/generate_readme/metadata_parser.py): `scripts/generate_readme/metadata_parser.py`

### Quality Checks

- [Compile Check](https://github.com/kubeflow/pipelines-components/blob/main/scripts/compile_check/compile_check.py): `scripts/compile_check/compile_check.py`
- [Component Validation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_components/validate_components.py): `scripts/validate_components/validate_components.py`
- [Example Validation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_examples/validate_examples.py): `scripts/validate_examples/validate_examples.py`
- [Metadata Validation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_metadata/validate_metadata.py): `scripts/validate_metadata/validate_metadata.py`
- [Package Validation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_package_entries/validate_package_entries.py): `scripts/validate_package_entries/validate_package_entries.py`
- [Freshness Check](https://github.com/kubeflow/pipelines-components/blob/main/scripts/check_component_freshness/check_component_freshness.py): `scripts/check_component_freshness/check_component_freshness.py`

### Image Maintenance

- [Base Image Validation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/validate_base_images/validate_base_images.py): `scripts/validate_base_images/validate_base_images.py`
- [Image Tag Check](https://github.com/kubeflow/pipelines-components/blob/main/scripts/check_base_image_tags/check_base_image_tags.py): `scripts/check_base_image_tags/check_base_image_tags.py`
- [Image Utilities](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/base_image.py): `scripts/lib/base_image.py`

### Shared Tooling

- [Asset Discovery](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/discovery.py): `scripts/lib/discovery.py`
- [KFP Compilation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/kfp_compilation.py): `scripts/lib/kfp_compilation.py`
- [Metadata Utilities](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/metadata_utils.py): `scripts/lib/metadata_utils.py`
- [OCI Tag Validation](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/oci.py): `scripts/lib/oci.py`
- [Source Parsing](https://github.com/kubeflow/pipelines-components/blob/main/scripts/lib/parsing.py): `scripts/lib/parsing.py`

### Other components

- **Pipeline Author**

## Connections

- Pipeline Author → Reusable Components: uses components (evidence: `README.md`)
- Pipeline Author → Pipeline Definitions: composes pipelines (evidence: `README.md`)
- README Generation → Category Index: generates index (evidence: `scripts/generate_readme/writer.py`)
- README Generation → Content Generator: generates content (evidence: `scripts/generate_readme/writer.py`)
- README Generation → Metadata Parser: parses metadata (evidence: `scripts/generate_readme/writer.py`)
- Compile Check → Asset Discovery: discovers assets (evidence: `scripts/compile_check/compile_check.py`)
- Compile Check → KFP Compilation: compiles assets (evidence: `scripts/compile_check/compile_check.py`)
- Compile Check → Metadata Utilities: uses metadata (evidence: `scripts/compile_check/compile_check.py`)
- Component Validation → Asset Discovery: discovers assets (evidence: `scripts/validate_components/validate_components.py`)
- Component Validation → KFP Compilation: compiles assets (evidence: `scripts/validate_components/validate_components.py`)
- Example Validation → Asset Discovery: discovers assets (evidence: `scripts/validate_examples/validate_examples.py`)
- Example Validation → KFP Compilation: compiles examples (evidence: `scripts/validate_examples/validate_examples.py`)
- Example Validation → Source Parsing: parses source (evidence: `scripts/validate_examples/validate_examples.py`)
- Package Validation → Asset Discovery: discovers assets (evidence: `scripts/validate_package_entries/validate_package_entries.py`)
- Freshness Check → Asset Discovery: discovers assets (evidence: `scripts/check_component_freshness/check_component_freshness.py`)
- Base Image Validation → Asset Discovery: discovers assets (evidence: `scripts/validate_base_images/validate_base_images.py`)
- Base Image Validation → Image Utilities: checks images (evidence: `scripts/validate_base_images/validate_base_images.py`)
- Base Image Validation → KFP Compilation: compiles assets (evidence: `scripts/validate_base_images/validate_base_images.py`)
- Image Tag Check → Image Utilities: checks tags (evidence: `scripts/check_base_image_tags/check_base_image_tags.py`)
- Image Utilities → KFP Compilation: compiles assets (evidence: `scripts/lib/base_image.py`)
- Image Utilities → OCI Tag Validation: validates tags (evidence: `scripts/lib/base_image.py`)
- Image Utilities → Source Parsing: parses image refs (evidence: `scripts/lib/base_image.py`)
