How kubeflow/pipelines-components works
This repository is a Kubeflow Pipelines asset library: reusable components and complete pipeline definitions are the assets users compose into AI workflows. The sampled runtime shows SDG loading data and a flow, validating and generating synthetic data, then writing dataset output and metrics. Separate repository tooling supports asset discovery, documentation generation, validation, and maintenance; these are meaningful repository capabilities rather than runtime stages. Several such modules are indexed but unsampled, so their internal behavior and unsupported connections are not expanded here.
Main entry points: discovery.py Asset Discovery · base_image.py Image Utilities · kfp_compilation.py KFP Compilation · writer.py README Generation
Components 24
Workflow Assets
- Synthetic Data
components/data_processing/sdg/component.py - Yoda Processor
components/data_processing/yoda_data_processor/component.py - SDG Pipeline
pipelines/data_processing/sdg/pipeline.py - Reusable Components
components - Pipeline Definitions
pipelines
Asset Catalog
- README Generation
scripts/generate_readme/writer.py - Category Index
scripts/generate_readme/category_index_generator.py - Content Generator
scripts/generate_readme/content_generator.py - Metadata Parser
scripts/generate_readme/metadata_parser.py
Quality Checks
- Compile Check
scripts/compile_check/compile_check.py - Component Validation
scripts/validate_components/validate_components.py - Example Validation
scripts/validate_examples/validate_examples.py - Metadata Validation
scripts/validate_metadata/validate_metadata.py - Package Validation
scripts/validate_package_entries/validate_package_entries.py - Freshness Check
scripts/check_component_freshness/check_component_freshness.py
Image Maintenance
- Base Image Validation
scripts/validate_base_images/validate_base_images.py - Image Tag Check
scripts/check_base_image_tags/check_base_image_tags.py - Image Utilities
scripts/lib/base_image.py
Shared Tooling
- Asset Discovery
scripts/lib/discovery.py - KFP Compilation
scripts/lib/kfp_compilation.py - Metadata Utilities
scripts/lib/metadata_utils.py - OCI Tag Validation
scripts/lib/oci.py - Source Parsing
scripts/lib/parsing.py
Other components
- Pipeline Author
Connections 22
- Pipeline Author → Reusable Components: uses components
README.md - Pipeline Author → Pipeline Definitions: composes pipelines
README.md - README Generation → Category Index: generates index
scripts/generate_readme/writer.py - README Generation → Content Generator: generates content
scripts/generate_readme/writer.py - README Generation → Metadata Parser: parses metadata
scripts/generate_readme/writer.py - Compile Check → Asset Discovery: discovers assets
scripts/compile_check/compile_check.py - Compile Check → KFP Compilation: compiles assets
scripts/compile_check/compile_check.py - Compile Check → Metadata Utilities: uses metadata
scripts/compile_check/compile_check.py - Component Validation → Asset Discovery: discovers assets
scripts/validate_components/validate_components.py - Component Validation → KFP Compilation: compiles assets
scripts/validate_components/validate_components.py - Example Validation → Asset Discovery: discovers assets
scripts/validate_examples/validate_examples.py - Example Validation → KFP Compilation: compiles examples
scripts/validate_examples/validate_examples.py - Example Validation → Source Parsing: parses source
scripts/validate_examples/validate_examples.py - Package Validation → Asset Discovery: discovers assets
scripts/validate_package_entries/validate_package_entries.py - Freshness Check → Asset Discovery: discovers assets
scripts/check_component_freshness/check_component_freshness.py - Base Image Validation → Asset Discovery: discovers assets
scripts/validate_base_images/validate_base_images.py - Base Image Validation → Image Utilities: checks images
scripts/validate_base_images/validate_base_images.py - Base Image Validation → KFP Compilation: compiles assets
scripts/validate_base_images/validate_base_images.py - Image Tag Check → Image Utilities: checks tags
scripts/check_base_image_tags/check_base_image_tags.py - Image Utilities → KFP Compilation: compiles assets
scripts/lib/base_image.py - Image Utilities → OCI Tag Validation: validates tags
scripts/lib/base_image.py - Image Utilities → Source Parsing: parses image refs
scripts/lib/base_image.py