# netflix/maestro architecture

> Architecture diagram of the GitHub repository netflix/maestro, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netflix/maestro
- Repository: https://github.com/netflix/maestro

## Overview

##### High-Level Architecture Components

a) Core Services:
- Maestro Server (API Layer)
- Maestro Engine (Workflow Execution)
- CockroachDB Persistence Layer
- Netflix SEL (Expression Language Engine)

- Blue for API/Server components
- Green for processing/engine components
- Yellow for persistence layer
- Purple for supporting services

##### Component Relationships

Following interactions:

**[Client Applications] → [Maestro Server (REST API)]**
- ↓
**[Maestro Engine] ←→ [Netflix SEL]**
- ↓
**[CockroachDB Persistence]**

##### Detailed Component Breakdown

For each major component, include subcomponents:

a) Maestro Server:
- Controllers (WorkflowController, WorkflowActionController, WorkflowInstanceController)
- Configuration Management
- Request Interceptors
- Exception Handlers

b) Maestro Engine:
- Workflow Executor
- Task Runner
- Step Runtime Manager
- Concurrency Handler
- Event Processors
- Job Handlers

c) Persistence Layer:
- DAO Layer
- Migration Management
- Transaction Handling

##### Cross-Cutting Concerns

- Security/Authentication
- Metrics/Monitoring
- Event Publishing
- Tracing

##### Key Integration Points

Highlight:
- External System Integration
- Event Publishers/Subscribers
- Database Connections
- Expression Language Integration

##### Deployment Context

Show:
- Service Boundaries
- Network Zones
- Load Balancers (if applicable)
- Scaling Points

##### Technical Details

Annotations for:
- Java 21 Runtime
- Spring Boot Framework
- CockroachDB
- Docker Container Boundaries

##### Scale Indicators

Note:
- Handles hundreds of thousands of workflows
- Millions of jobs daily
- Multiple user types (data scientists, engineers, analysts)

##### Integration boundaries

Remember to include version information and any significant configuration dependencies, as this is a production-grade system requiring careful attention to deployment details.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Style definitions
    classDef api fill:#2196F3,stroke:#1565C0,color:white
    classDef engine fill:#4CAF50,stroke:#388E3C,color:white
    classDef persistence fill:#FFC107,stroke:#FFA000,color:black
    classDef support fill:#9C27B0,stroke:#7B1FA2,color:white
    classDef crosscut fill:#FF5722,stroke:#E64A19,color:white

    %% Client Applications
    Client["Client Applications"]

    %% Main Components
    subgraph Server["Maestro Server"]
        API["REST API Layer"]:::api
        Controllers["Controllers"]:::api
        Config["Configuration Management"]:::api
        Interceptors["Request Interceptors"]:::api
        ExHandlers["Exception Handlers"]:::api
    end
    click Server "https://github.com/Netflix/maestro/tree/main/maestro-server/src/main/java/com/netflix/maestro/server/"
    click Controllers "https://github.com/Netflix/maestro/tree/main/maestro-server/src/main/java/com/netflix/maestro/server/controllers/"

    subgraph Engine["Maestro Engine"]
        WFExecutor["Workflow Executor"]:::engine
        TaskRunner["Task Runner"]:::engine
        StepRuntime["Step Runtime Manager"]:::engine
        ConcHandler["Concurrency Handler"]:::engine
        EventProc["Event Processors"]:::engine
        JobHandler["Job Handlers"]:::engine
    end
    click Engine "https://github.com/Netflix/maestro/tree/main/maestro-engine/src/main/java/com/netflix/maestro/engine/"
    click WFExecutor "https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/conductor/core/execution/MaestroWorkflowExecutor.java"
    click StepRuntime "https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/maestro/engine/execution/StepRuntimeManager.java"
    click EventProc "https://github.com/Netflix/maestro/tree/main/maestro-engine/src/main/java/com/netflix/maestro/engine/processors/"
    click JobHandler "https://github.com/Netflix/maestro/tree/main/maestro-engine/src/main/java/com/netflix/maestro/engine/handlers/"
    click TaskRunner "https://github.com/Netflix/maestro/tree/main/maestro-engine/src/main/java/com/netflix/maestro/engine/tasks/"

    subgraph Persistence["CockroachDB Persistence"]
        DAO["DAO Layer"]:::persistence
        Migration["Migration Management"]:::persistence
        TxHandler["Transaction Handling"]:::persistence
    end
    click Persistence "https://github.com/Netflix/maestro/tree/main/cockroachdb-persistence/src/main/java/com/netflix/conductor/cockroachdb/"
    click DAO "https://github.com/Netflix/maestro/tree/main/cockroachdb-persistence/src/main/java/com/netflix/conductor/cockroachdb/dao/"

    subgraph CrossCutting["Cross-Cutting Concerns"]
        Security["Security/Authentication"]:::crosscut
        Metrics["Metrics/Monitoring"]:::crosscut
        EventPub["Event Publishing"]:::crosscut
        Tracing["Tracing"]:::crosscut
    end
    click Metrics "https://github.com/Netflix/maestro/tree/main/maestro-engine/src/main/java/com/netflix/maestro/engine/metrics/"
    click Tracing "https://github.com/Netflix/maestro/tree/main/maestro-engine/src/main/java/com/netflix/maestro/engine/tracing/"

    SEL["Netflix SEL"]:::support
    click SEL "https://github.com/Netflix/maestro/tree/main/netflix-sel/src/main/java/com/netflix/sel/"

    %% Relationships
    Client --> API
    API --> Controllers
    Controllers --> WFExecutor
    WFExecutor --> TaskRunner
    TaskRunner --> StepRuntime
    StepRuntime --> EventProc
    EventProc --> JobHandler
    WFExecutor <--> SEL
    Engine --> DAO
    DAO --> TxHandler

    %% Cross-cutting relationships
    CrossCutting -.-> Server
    CrossCutting -.-> Engine
    CrossCutting -.-> Persistence

    %% Legend
    subgraph Legend
        API_L["API Components"]:::api
        Engine_L["Engine Components"]:::engine
        DB_L["Persistence Components"]:::persistence
        Support_L["Support Services"]:::support
        CrossCut_L["Cross-Cutting Concerns"]:::crosscut
    end
```
