# zenml-io/zenml architecture

> Architecture diagram of the GitHub repository zenml-io/zenml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zenml-io/zenml
- Repository: https://github.com/zenml-io/zenml

## Overview

##### Primary components
- Core CLI/SDK: The entrypoints for users, including the “zenml” command, configuration, login, and stack management (see src/zenml/cli, src/zenml/login).
- ZenML Core Library: The heart of the framework that handles pipeline construction, configuration, and execution. This lives in src/zenml and includes pipeline definitions, step implementations, orchestrators, and materializers.
- Orchestrators & Stack Components: Components for running pipelines (found under src/zenml/orchestrators, src/zenml/stack, and src/zenml/step_operators). These abstract the execution environments (local, docker, kubernetes, etc.) and handle job scheduling, resource allocation, and dependency resolution.
- Integrations and Adapters: A large set of integrations (AWS, GCP, Azure, Huggingface, great_expectations, LangChain, etc.) that allow the platform to plug into different tools and cloud providers; these live under src/zenml/integrations.
- Server & Metadata Store: Components for deployment, logging, artifact tracking, and model registries. The zen_server folder (src/zenml/zen_server) is key for serving the central orchestration, and the zen_stores subsystems manage metadata storage and migration.
- Supporting Infrastructure: Dockerfiles, Helm charts, infrastructure templates (under docker, helm, and infra directories) that support deployment of the server and other components in production environments.
- Tests and Examples: While not directly part of the runtime, the extensive tests (unit, integration, functional) and examples provide insight into expected interactions and pipeline flows.

##### Relationships and data flows
- User Interface → CLI & VS Code Extension: Users interact with ZenML via CLI commands or the VS Code extension, which invoke commands in the SDK.
- Pipeline Definition and Registration: The user-defined pipelines (using decorators such as @pipeline and @step) are processed by the core library. The CLI dispatches these to the orchestrators.
- Orchestration and Execution: The orchestrators (local, docker, kubernetes, etc.) receive the pipeline run requests, schedule the execution of steps, and inject resources as specified via ResourceSettings and DockerSettings.
- Artifact and Metadata Flow: As pipelines run, artifacts are stored (managed by artifact stores) and metadata is recorded. The zen_stores and materializers ensure versioning, reproducibility, and logging (e.g., tracking parameters, models, deployment configurations).
- Integration Layer: The integrations library connects the pipeline execution with external services (cloud platforms, experiment trackers, model registries, etc.), serving as adapters that translate between ZenML’s core abstractions and the APIs of external systems.
- Server and Dashboard: The zen_server acts as a central control plane. It communicates with orchestrators and serves endpoints (APIs) for monitoring, logging, and dashboard displays. This component also manages authentication, RBAC, and external notifications.

##### Architectural design principles
- Modular and Layered: Emphasize the separation between the user interface (CLI), core orchestration engine, integrations, and deployment infrastructure. Each module is self-contained and communicates through well-defined APIs/interfaces.
- Data Lineage and Reproducibility: Incorporate components for tracking pipeline execution history, artifact versions, and server endpoints that store lineage information.
- Infrastructure Agnostic Execution: Highlight that runners (orchestrators) can execute on a variety of platforms (local, cloud, Kubernetes, etc.) without altering pipeline code.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User-Facing Layer
    subgraph "User-Facing Layer"
        UI["User Interface (CLI & VS Code Extension)"]:::frontend
        CLI["Core CLI/SDK"]:::core
        LOGIN["Login"]:::core
    end

    %% Core Pipeline Engine
    subgraph "Core Pipeline Engine"
        CORE["ZenML Core Library"]:::core
    end

    %% Execution Orchestration
    subgraph "Execution Orchestration"
        ORCH["Orchestrators & Stack Components"]:::orchestration
    end

    %% Data & Metadata Management
    subgraph "Data & Metadata Management"
        SERVER["Zen Server & Metadata Store"]:::backend
    end

    %% External Integrations
    subgraph "External Integrations"
        INTEGRATIONS["Integrations & Adapters"]:::integration
    end

    %% Deployment Infrastructure
    subgraph "Deployment Infrastructure"
        INFRA["Supporting Infrastructure"]:::infrastructure
    end

    %% Testing & Examples
    subgraph "Testing & Examples"
        TESTS["Tests"]:::testing
        EXAMPLES["Examples"]:::testing
    end

    %% Connections
    UI -->|"triggers"| CLI
    UI -->|"triggers"| LOGIN
    CLI -->|"executes"| CORE
    LOGIN -->|"authenticates"| CORE
    CORE -->|"dispatchesPipeline"| ORCH
    ORCH -->|"storesArtifacts"| SERVER
    ORCH -->|"callsAdapters"| INTEGRATIONS
    SERVER -->|"sendsUpdates"| INTEGRATIONS
    UI -->|"viewsDashboard"| SERVER
    INFRA -->|"supportsDeployment"| SERVER
    TESTS -->|"validates"| CORE
    EXAMPLES -->|"demonstratesUsage"| CORE

    %% Click Events
    click CLI "https://github.com/zenml-io/zenml/tree/main/src/zenml/cli"
    click LOGIN "https://github.com/zenml-io/zenml/tree/main/src/zenml/login"
    click CORE "https://github.com/zenml-io/zenml/tree/main/src/zenml"
    click ORCH "https://github.com/zenml-io/zenml/tree/main/src/zenml/orchestrators"
    click ORCH "https://github.com/zenml-io/zenml/tree/main/src/zenml/stack"
    click ORCH "https://github.com/zenml-io/zenml/tree/main/src/zenml/step_operators"
    click INTEGRATIONS "https://github.com/zenml-io/zenml/tree/main/src/zenml/integrations"
    click SERVER "https://github.com/zenml-io/zenml/tree/main/src/zenml/zen_server"
    click SERVER "https://github.com/zenml-io/zenml/tree/main/src/zenml/zen_stores"
    click INFRA "https://github.com/zenml-io/zenml/tree/main/docker"
    click INFRA "https://github.com/zenml-io/zenml/tree/main/helm"
    click INFRA "https://github.com/zenml-io/zenml/tree/main/infra"
    click TESTS "https://github.com/zenml-io/zenml/tree/main/tests"
    click EXAMPLES "https://github.com/zenml-io/zenml/tree/main/examples"

    %% Styles
    classDef frontend fill:#f9e79f,stroke:#b9770e,stroke-width:2px;
    classDef core fill:#aed6f1,stroke:#1f618d,stroke-width:2px;
    classDef orchestration fill:#abebc6,stroke:#239b56,stroke-width:2px;
    classDef integration fill:#f9c9f9,stroke:#af7ac5,stroke-width:2px;
    classDef backend fill:#f5b7b1,stroke:#c0392b,stroke-width:2px;
    classDef infrastructure fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef testing fill:#fadbd8,stroke:#cb4335,stroke-width:2px;
```
