# netflix/mantis architecture

> Architecture diagram of the GitHub repository netflix/mantis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netflix/mantis
- Repository: https://github.com/netflix/mantis

## Overview

##### Project Type and Purpose
- This is an open-source, multi-module, distributed stream-processing platform (Netflix Mantis). It provides:
- A control plane (master) that accepts job submissions, schedules work, tracks state, and provides APIs.
- Clients and SDKs for submitting and managing jobs.
- Worker agents and executors that run user-defined streaming jobs.
- Pluggable connectors for data ingestion (e.g., Kafka, Iceberg) and data publish.
- A runtime library implementing reactive streams and job lifecycle.

##### File-Structure Analysis
- Top-level modules (Gradle subprojects):
- mantis-control-plane-server & mantis-control-plane-core & mantis-control-plane-store-dynamodb – the master service, persistence, leader election (Akka/ZooKeeper/DynamoDB).
- mantis-control-plane-client & mantis-client – Java/CLI clients that talk to the master via HTTP/REST or SSE.
- mantis-server-agent & mantis-server-worker & mantis-runtime-executor & mantis-runtime-loader – frameworks to start worker processes, load jobs, run task executors, heartbeat/push status.
- mantis-runtime – the core library for user-defined jobs (DSL, stages, sources, sinks, reactive operators).
- mantis-connectors – pluggable modules to source/sink data (Kafka, Iceberg, job-source, publish).
- mantis-publish & mantis-publish-netty[-guice] – publish/subscribe subsystem for streaming events across jobs.
- mantis-network & mantis-remote-observable – network layers (Netty, SSE) for push/pull event delivery.
- docs, examples, testcontainers, discovery-proto, common modules (serde, common utilities).

##### README Insights
- Core/common code separated into mantis-common and mantis-common-serde.
- Control-plane core shared by server and clients.
- Reactive, actor-based scheduling and messaging.
- Plugin architecture for connectors and publish clients.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph "Client Layer"
        direction TB
        ClientSDK["mantis-client"]:::client
        ClientHTTP["mantis-control-plane-client"]:::client
    end

    %% Control Plane
    subgraph "Control Plane" 
        direction TB
        CPServer["mantis-control-plane-server"]:::control
        CPCore["mantis-control-plane-core"]:::control
        CPPersistence["mantis-control-plane-store-dynamodb"]:::control
    end

    %% Worker Layer
    subgraph "Worker Layer"
        direction TB
        AgentLauncher["mantis-server-agent"]:::worker
        WorkerClient["mantis-server-worker-client"]:::worker
        RuntimeExecutor["mantis-runtime-executor"]:::worker
        RuntimeLoader["mantis-runtime-loader"]:::worker
    end

    %% Runtime & Connectors
    subgraph "Runtime & Connectors"
        direction TB
        RuntimeLib["mantis-runtime"]:::runtime

        subgraph "Connectors"
            direction TB
            KafkaConn["mantis-connector-kafka"]:::runtime
            IcebergConn["mantis-connector-iceberg"]:::runtime
            JobSourceConn["mantis-connector-job-source"]:::runtime
            PublishSPI["mantis-connector-publish"]:::runtime
        end

        subgraph "Publish Subsystem"
            direction TB
            PublishCore["mantis-publish-core"]:::runtime
            PublishNetty["mantis-publish-netty"]:::runtime
            PublishGuice["mantis-publish-netty-guice"]:::runtime
        end

        subgraph "Network Layer"
            direction TB
            NettyLayer["mantis-network"]:::runtime
            SSELayer["mantis-remote-observable"]:::runtime
        end

        CommonUtil["mantis-common / mantis-common-serde"]:::runtime
    end

    %% External Services
    subgraph "External Services"
        direction TB
        ZooKeeper["ZooKeeper"]:::external
        DynamoDB["DynamoDB"]:::external
        Docker["Docker"]:::external
    end

    %% Flows
    ClientSDK -->|HTTP/REST| CPServer
    ClientHTTP -->|SSE/HTTP| CPServer
    CPServer -->|schedules jobs| CPCore
    CPCore -->|persists metadata| CPPersistence
    CPCore -->|assigns tasks| AgentLauncher
    AgentLauncher -->|launch executor| RuntimeExecutor
    RuntimeExecutor -->|load jobs| RuntimeLoader
    RuntimeExecutor -->|heartbeat/SSE| CPServer
    RuntimeExecutor -->|metrics/Netty| WorkerClient
    RuntimeExecutor -->|runs stages| RuntimeLib
    RuntimeLib -->|uses| KafkaConn
    RuntimeLib -->|uses| IcebergConn
    RuntimeLib -->|uses| JobSourceConn
    RuntimeLib -->|uses| PublishSPI
    PublishSPI -->|publishes events| PublishCore
    PublishCore -->|over Netty| PublishNetty
    PublishCore -->|via Guice| PublishGuice
    RuntimeExecutor -->|push/pull| NettyLayer
    RuntimeExecutor -->|subscribe/observe| SSELayer
    CPCore -->|leader election| ZooKeeper
    CPCore -->|leader election| DynamoDB
    E1(( )):::invisible -->|examples/tests| Docker

    %% Click events
    click CPServer "https://github.com/netflix/mantis/tree/master/mantis-control-plane/mantis-control-plane-server"
    click CPCore "https://github.com/netflix/mantis/tree/master/mantis-control-plane/mantis-control-plane-core"
    click CPPersistence "https://github.com/netflix/mantis/tree/master/mantis-control-plane/mantis-control-plane-store/mantis-control-plane-store-dynamodb"
    click ClientSDK "https://github.com/netflix/mantis/tree/master/mantis-client"
    click ClientHTTP "https://github.com/netflix/mantis/tree/master/mantis-control-plane/mantis-control-plane-client"
    click AgentLauncher "https://github.com/netflix/mantis/tree/master/mantis-server/mantis-server-agent"
    click WorkerClient "https://github.com/netflix/mantis/tree/master/mantis-server/mantis-server-worker-client"
    click RuntimeExecutor "https://github.com/netflix/mantis/tree/master/mantis-runtime-executor"
    click RuntimeLoader "https://github.com/netflix/mantis/tree/master/mantis-runtime-loader"
    click RuntimeLib "https://github.com/netflix/mantis/tree/master/mantis-runtime"
    click KafkaConn "https://github.com/netflix/mantis/tree/master/mantis-connectors/mantis-connector-kafka"
    click IcebergConn "https://github.com/netflix/mantis/tree/master/mantis-connectors/mantis-connector-iceberg"
    click JobSourceConn "https://github.com/netflix/mantis/tree/master/mantis-connectors/mantis-connector-job-source"
    click PublishSPI "https://github.com/netflix/mantis/tree/master/mantis-connectors/mantis-connector-publish"
    click PublishCore "https://github.com/netflix/mantis/tree/master/mantis-publish/mantis-publish-core"
    click PublishNetty "https://github.com/netflix/mantis/tree/master/mantis-publish/mantis-publish-netty"
    click PublishGuice "https://github.com/netflix/mantis/tree/master/mantis-publish/mantis-publish-netty-guice"
    click NettyLayer "https://github.com/netflix/mantis/tree/master/mantis-network"
    click SSELayer "https://github.com/netflix/mantis/tree/master/mantis-remote-observable"
    click CommonUtil "https://github.com/netflix/mantis/tree/master/mantis-common"
    %% Styles
    classDef control fill:#c6dbf7,stroke:#3182bd,color:#000
    classDef client fill:#d9f0a3,stroke:#31a354,color:#000
    classDef worker fill:#fdd0a2,stroke:#e6550d,color:#000
    classDef runtime fill:#efedf5,stroke:#756bb1,color:#000
    classDef external fill:#eeeeee,stroke:#999999,color:#000
    classDef invisible fill:#ffffff,stroke:none, color:#ffffff
```
