# netflix/conductor architecture

> Architecture diagram of the GitHub repository netflix/conductor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netflix/conductor
- Repository: https://github.com/netflix/conductor

## Overview

##### Main Components
- Core Engine: Represented by modules like “conductor-core”, “conductor-server”, and portions of “core” which implement the workflow execution, decider service, event processors, scheduling, and task mapping.
- API Layers: Separate components for REST (in “conductor-rest”) and gRPC (modules “conductor-grpc”, “conductor-grpc-server”, “conductor-grpc-client”) that expose endpoints for workflow control, task submission, and monitoring.
- Persistence: Multiple persistence modules exist:
– Redis based persistence (conductor-redis-persistence and redis-lock, redis-concurrency-limit) which handles workflow state and task queues.
– Cassandra and ES modules for alternate storage and indexing (e.g., “conductor-cassandra-persistence” and “conductor-es6-persistence”).
- External Task Providers/Workers: Modules like “conductor-http-task”, “json-jq-task”, and SDKs such as “conductor-client”, “conductor-client-spring”, “conductor-java-sdk” indicate that worker clients interact with the conductor engine to perform tasks.
- Client/UI: The “conductor-ui” module (with its Node/React components) provides a web-based dashboard. Also the “client” module and related SDKs support workflow creation and execution.
- Integrations/Extensions: Additional modules (like “awss3-storage”, “awssqs-event-queue”, “annotations”, “grpc-client”) show how the engine extends through external systems and custom annotations support.

##### Relationships & Interactions
- A Request Flow: Workflow definitions and task requests come in via REST or gRPC APIs to the server. The core engine then uses an internal decider service to “orchestrate” the workflow and direct tasks to appropriate workers.
- Persistence Operations: The engine continuously reads/writes workflow state to primary persistence (Redis) and indexes metadata in ElasticSearch. Alternative persistence modules (like Cassandra) may be used in different deployments.
- UI & Monitoring: The UI module interacts with the API layer to monitor live workflows. Additionally, logging, metric modules (inside core and client packages) feed data into external monitoring systems.

##### Architectural Patterns & Design Principles
- Modularity & Separation of Concerns: Each major feature (workflow orchestration, task execution, persistence, API exposure, UI) has its own module and can be represented separately.
- Extensibility: The support for community modules and plugins (e.g., alternative storage like AWS S3, SQS, gRPC client/server modules) should be clearly illustrated, showing hooks or extension points into the core engine.
- Use of Spring Boot: For server modules, the use of Spring Boot means that dependency injection, application configuration, and auto-configuration are key; showing a “Configuration & Bootstrapping” block can be useful.

##### Relevant Technologies
- Java (with JDK 17+), Spring Boot for the server.
- Redis, Cassandra, and Elasticsearch for persistence and indexing.
- Node.js and React (or similar) for the UI.
- gRPC & REST as the communication protocols.
- Docker for containerization (as indicated by extensive Docker files).
- Gradle as a build tool.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Client Layer
    subgraph "Client Layer"
        UIDashboard["UI Dashboard"]:::client
        subgraph "Client/SDK Ecosystem"
            JavaSDK["Java SDK"]:::client
            SpringClient["Spring Client"]:::client
            GeneralClient["General Client"]:::client
        end
    end

    %% API Layer
    subgraph "API Layer"
        RestAPI["REST API Layer"]:::api
        gRPCServer["gRPC Server"]:::api
        gRPCClient["gRPC Client"]:::api
    end

    %% Core Engine
    subgraph "Core Engine"
        CoreEngine["Conductor Core"]:::core
        ServerEngine["Conductor Server"]:::core
        subgraph "Annotation Processing"
            AnnotationsProcessor["Annotations Processor"]:::annotation
            Annotations["Annotations"]:::annotation
        end
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        RedisPersistence(("Redis Persistence")):::persistence
        CassandraPersistence(("Cassandra Persistence")):::persistence
        ES(("Elasticsearch Persistence")):::persistence
    end

    %% External Task Providers
    subgraph "External Task Providers"
        HTTPTask["HTTP Task"]:::external
        JSONTask["JSON/JQ Task"]:::external
    end

    %% Integrations/Extensions
    subgraph "Integrations/Extensions"
        AWSS3["AWS S3 Storage"]:::integration
        AWSSQS["AWS SQS Event Queue"]:::integration
    end

    %% Concurrency and Locking
    subgraph "Concurrency and Locking"
        RedisLock["Redis Lock"]:::lock
        RedisConcurrency["Redis Concurrency Limit"]:::lock
    end

    %% Connections from Client to API Layer
    UIDashboard -->|"sends requests"| RestAPI
    UIDashboard -->|"sends requests"| gRPCServer
    JavaSDK -->|"invokes APIs"| RestAPI
    JavaSDK -->|"invokes APIs"| gRPCClient
    SpringClient -->|"invokes APIs"| RestAPI
    SpringClient -->|"invokes APIs"| gRPCClient
    GeneralClient -->|"invokes APIs"| RestAPI

    %% Connections from API Layer to Core Engine
    RestAPI -->|"forwards request"| CoreEngine
    gRPCServer -->|"forwards request"| ServerEngine
    gRPCClient -->|"forwards request"| ServerEngine

    %% Core Engine to Persistence Layer
    CoreEngine -->|"updates state"| RedisPersistence
    CoreEngine -->|"updates state"| CassandraPersistence
    CoreEngine -->|"indexes metadata"| ES

    %% Core Engine to External Task Providers
    CoreEngine -->|"dispatches task"| HTTPTask
    CoreEngine -->|"dispatches task"| JSONTask

    %% Core Engine to Integrations/Extensions
    CoreEngine -->|"integrates with"| AWSS3
    CoreEngine -->|"integrates with"| AWSSQS

    %% Core Engine to Concurrency and Locking
    CoreEngine -->|"uses locking"| RedisLock
    CoreEngine -->|"uses locking"| RedisConcurrency

    %% Server Engine Annotation Processing
    ServerEngine -->|"processes annotations"| AnnotationsProcessor
    ServerEngine -->|"processes annotations"| Annotations

    %% Click Events
    click CoreEngine "https://github.com/netflix/conductor/tree/main/core/src/main/java/com/netflix/conductor"
    click ServerEngine "https://github.com/netflix/conductor/tree/main/server/src/main/java/com/netflix/conductor"
    click RestAPI "https://github.com/netflix/conductor/tree/main/rest/src/main/java/com/netflix/conductor/rest/controllers"
    click gRPCServer "https://github.com/netflix/conductor/tree/main/grpc-server/src/main/java/com/netflix/conductor/grpc/server"
    click gRPCClient "https://github.com/netflix/conductor/tree/main/grpc-client/src/main/java/com/netflix/conductor/client/grpc"
    click RedisPersistence "https://github.com/netflix/conductor/tree/main/redis-persistence/src/main/java/com/netflix/conductor/redis"
    click CassandraPersistence "https://github.com/netflix/conductor/tree/main/cassandra-persistence/src/main/java/com/netflix/conductor/cassandra"
    click ES "https://github.com/netflix/conductor/tree/main/es6-persistence/src/main/java/com/netflix/conductor/es6"
    click HTTPTask "https://github.com/netflix/conductor/tree/main/http-task/src/main/java/com/netflix/conductor/tasks/http"
    click JSONTask "https://github.com/netflix/conductor/tree/main/json-jq-task/src/main/java/com/netflix/conductor/tasks/json"
    click JavaSDK "https://github.com/netflix/conductor/tree/main/java-sdk/src/main/java/com/netflix/conductor/sdk"
    click SpringClient "https://github.com/netflix/conductor/tree/main/client-spring/src/main/java/com/netflix/conductor/client/spring"
    click GeneralClient "https://github.com/netflix/conductor/tree/main/client/src/main/java/com/netflix/conductor/client"
    click UIDashboard "https://github.com/netflix/conductor/tree/main/ui/"
    click AWSS3 "https://github.com/netflix/conductor/tree/main/awss3-storage/src/main/java/com/netflix/conductor/s3"
    click AWSSQS "https://github.com/netflix/conductor/tree/main/awssqs-event-queue/src/main/java/com/netflix/conductor/sqs"
    click AnnotationsProcessor "https://github.com/netflix/conductor/tree/main/annotations-processor/src/main/java/com/netflix/conductor/annotationsprocessor"
    click Annotations "https://github.com/netflix/conductor/tree/main/annotations/src/main/java/com/netflix/conductor/annotations"
    click RedisLock "https://github.com/netflix/conductor/tree/main/redis-lock/src/main/java/com/netflix/conductor/redislock"
    click RedisConcurrency "https://github.com/netflix/conductor/tree/main/redis-concurrency-limit/src/main/java/com/netflix/conductor/redis/limit"

    %% Styles
    classDef client fill:#a1c4fd,stroke:#000,stroke-width:2px;
    classDef api fill:#fda085,stroke:#000,stroke-width:2px;
    classDef core fill:#f6d365,stroke:#000,stroke-width:2px;
    classDef persistence fill:#c3f584,stroke:#000,stroke-width:2px;
    classDef external fill:#d3cce3,stroke:#000,stroke-width:2px;
    classDef integration fill:#b3e5fc,stroke:#000,stroke-width:2px;
    classDef lock fill:#ffe0b2,stroke:#000,stroke-width:2px;
    classDef annotation fill:#fbc7d4,stroke:#000,stroke-width:2px;
```
