# kedro-org/kedro architecture

> Architecture diagram of the GitHub repository kedro-org/kedro, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kedro-org/kedro
- Repository: https://github.com/kedro-org/kedro

## Overview

##### Main Components

a. Core Framework Components:
- Config Management System (kedro/config/)
- Pipeline Engine (kedro/pipeline/)
- I/O Layer (kedro/io/)
- Runner System (kedro/runner/)
- CLI Interface (kedro/framework/cli/)
- Session Management (kedro/framework/session/)
- Context Management (kedro/framework/context/)
- Hooks System (kedro/framework/hooks/)

b. Project Structure Components:
- Project Template System
- Data Catalog
- Pipeline Abstraction Layer
- Configuration Layer
- Logging System

##### Relationships and Interactions

- How the CLI interface interacts with the project context
- Data flow from Data Catalog through nodes and pipelines
- Demonstrate how runners execute pipelines
- Hook integration points throughout the system
- Configuration management flow

##### Architecture Pattern

Use these visual elements:
**┌─────────────────────────────────────────────────────────┐**
- Kedro Project                       │
- ┬─────────────────────────┬────────────────┤
- Project    │                         │    Pipeline    │
- Template   │     Data Catalog        │    Registry    │
- ┤                         ├────────────────┤

- Config     │      Pipeline           │     Nodes      │
- Management │      Engine             │                │
- ┤                         ├────────────────┤

- Runners    │                         │     Hooks      │

- ┴─────────────────────────┴────────────────┘

##### Key Flows to Highlight

a. Data Flow:
**Raw Data → Data Catalog → Nodes → Pipeline → Output Data**

b. Execution Flow:
**CLI → Context → Pipeline Registry → Runner → Nodes**

c. Configuration Flow:
**Config Files → Config Loader → Context → Components**

##### Integration Points

Integration with:
- External Tools (Jupyter, MLflow)
- Deployment Platforms (Airflow, Kubeflow)
- Storage Systems (Various dataset types)
- Visualization Tools (Kedro-Viz)

##### Additional Elements

Include:
- Version information
- Key dependencies
- Extension points for plugins
- Hook registration points

##### Layered Architecture

**┌─────────────────────────────────────────────┐**
- User Interface Layer           │
- ┤
- Business Logic Layer           │
- ┤
- Data Access Layer              │
- ┤
- Storage Layer                  │
- ┘

##### Important Notes

- Version compatibility information
- Extensibility points clearly
- Async vs sync operations
- Security boundaries
- Error handling flow

## Diagram (Mermaid)

```mermaid
flowchart LR
    subgraph "User Interface Layer"
        CLI["CLI Interface"]:::cli
        Jupyter["Jupyter Integration"]:::ui
    end

    subgraph "Business Logic Layer"
        Context["Context Management"]:::core
        Session["Session Management"]:::core
        PipelineEngine["Pipeline Engine"]:::core
        Hooks["Hooks System"]:::core
        Runner["Runner System"]:::core
        Registry["Pipeline Registry"]:::core
        Node["Node System"]:::core
        PluginSystem["Plugin System"]:::core
    end

    subgraph "Data Management Layer"
        Config["Config Management"]:::data
        DataCatalog["Data Catalog"]:::data
        IOLayer["I/O Layer"]:::data
        Settings["settings.py"]:::data
        OmegaConf["OmegaConf"]:::data
    end

    subgraph "Storage Layer"
        ProjectTemplate["Project Template"]:::storage
        ProjectStructure["Project Structure"]:::storage
        Logging["Logging System"]:::storage
    end

    %% Relationships
    CLI --> Context
    CLI --> Session
    Context --> Config
    Context --> DataCatalog
    Context --> PipelineEngine
    PipelineEngine --> Runner
    Runner --> Node
    Node --> IOLayer
    IOLayer --> DataCatalog
    Hooks --> Context
    Hooks --> PipelineEngine
    Registry --> PipelineEngine
    Session --> Context
    ProjectTemplate --> ProjectStructure
    Logging --> Context
    Settings --> Config
    OmegaConf --> Config
    Config --> Settings

    %% Plugin System Connections
    PluginSystem --> Context
    KedroViz["Kedro-Viz"]:::plugin --> PluginSystem
    OtherPlugins["Other Plugins"]:::plugin --> PluginSystem

    %% External Tool Connections
    Jupyter --> Context

    %% Click Events
    click CLI "https://github.com/kedro-org/kedro/tree/main/kedro/framework/cli/"
    click Context "https://github.com/kedro-org/kedro/tree/main/kedro/framework/context/"
    click Session "https://github.com/kedro-org/kedro/tree/main/kedro/framework/session/"
    click PipelineEngine "https://github.com/kedro-org/kedro/tree/main/kedro/pipeline/"
    click Hooks "https://github.com/kedro-org/kedro/tree/main/kedro/framework/hooks/"
    click Runner "https://github.com/kedro-org/kedro/tree/main/kedro/runner/"
    click Registry "https://github.com/kedro-org/kedro/blob/main/kedro/framework/cli/registry.py"
    click Node "https://github.com/kedro-org/kedro/blob/main/kedro/pipeline/node.py"
    click Config "https://github.com/kedro-org/kedro/tree/main/kedro/config/"
    click DataCatalog "https://github.com/kedro-org/kedro/blob/main/kedro/io/data_catalog.py"
    click IOLayer "https://github.com/kedro-org/kedro/tree/main/kedro/io/"
    click ProjectTemplate "https://github.com/kedro-org/kedro/tree/main/kedro/templates/project/"
    click ProjectStructure "https://github.com/kedro-org/kedro/tree/main/kedro/templates/project/{{ cookiecutter.repo_name }}/"
    click Logging "https://github.com/kedro-org/kedro/blob/main/kedro/logging.py"

    %% Styles
    classDef core fill:#2374ab,stroke:#2374ab,color:#fff
    classDef data fill:#57a773,stroke:#57a773,color:#fff
    classDef storage fill:#484d6d,stroke:#484d6d,color:#fff
    classDef ui fill:#c49991,stroke:#c49991,color:#fff
    classDef cli fill:#f7d794,stroke:#f7d794,color:#000
    classDef plugin fill:#8B008B,stroke:#8B008B,color:#fff

    %% Legend
    subgraph Legend
        CoreComponent["Core Component"]:::core
        DataComponent["Data Component"]:::data
        StorageComponent["Storage Component"]:::storage
        UIComponent["UI Component"]:::ui
        CLIComponent["CLI Component"]:::cli
        PluginComponent["Plugin Component"]:::plugin
    end
```
