How zenml-io/zenml works
Primary components
• Core CLI/SDK: The entrypoints for users, including the “zenml” command, configuration, login, and stack management (see src/zenml/cli, src/zenml/login).
• ZenML Core Library: The heart of the framework that handles pipeline construction, configuration, and execution. This lives in src/zenml and includes pipeline definitions, step implementations, orchestrators, and materializers.
• Orchestrators & Stack Components: Components for running pipelines (found under src/zenml/orchestrators, src/zenml/stack, and src/zenml/step_operators). These abstract the execution environments (local, docker, kubernetes, etc.) and handle job scheduling, resource allocation, and dependency resolution.
• Integrations and Adapters: A large set of integrations (AWS, GCP, Azure, Huggingface, great_expectations, LangChain, etc.) that allow the platform to plug into different tools and cloud providers; these live under src/zenml/integrations.
• Server & Metadata Store: Components for deployment, logging, artifact tracking, and model registries. The zen_server folder (src/zenml/zen_server) is key for serving the central orchestration, and the zen_stores subsystems manage metadata storage and migration.
• Supporting Infrastructure: Dockerfiles, Helm charts, infrastructure templates (under docker, helm, and infra directories) that support deployment of the server and other components in production environments.
• Tests and Examples: While not directly part of the runtime, the extensive tests (unit, integration, functional) and examples provide insight into expected interactions and pipeline flows.
Relationships and data flows
• User Interface → CLI & VS Code Extension: Users interact with ZenML via CLI commands or the VS Code extension, which invoke commands in the SDK.
• Pipeline Definition and Registration: The user-defined pipelines (using decorators such as @pipeline and @step) are processed by the core library. The CLI dispatches these to the orchestrators.
• Orchestration and Execution: The orchestrators (local, docker, kubernetes, etc.) receive the pipeline run requests, schedule the execution of steps, and inject resources as specified via ResourceSettings and DockerSettings.
• Artifact and Metadata Flow: As pipelines run, artifacts are stored (managed by artifact stores) and metadata is recorded. The zen_stores and materializers ensure versioning, reproducibility, and logging (e.g., tracking parameters, models, deployment configurations).
• Integration Layer: The integrations library connects the pipeline execution with external services (cloud platforms, experiment trackers, model registries, etc.), serving as adapters that translate between ZenML’s core abstractions and the APIs of external systems.
• Server and Dashboard: The zen_server acts as a central control plane. It communicates with orchestrators and serves endpoints (APIs) for monitoring, logging, and dashboard displays. This component also manages authentication, RBAC, and external notifications.
Architectural design principles
• Modular and Layered: Emphasize the separation between the user interface (CLI), core orchestration engine, integrations, and deployment infrastructure. Each module is self-contained and communicates through well-defined APIs/interfaces.
• Data Lineage and Reproducibility: Incorporate components for tracking pipeline execution history, artifact versions, and server endpoints that store lineage information.
• Infrastructure Agnostic Execution: Highlight that runners (orchestrators) can execute on a variety of platforms (local, cloud, Kubernetes, etc.) without altering pipeline code.