# apache/storm architecture

> Architecture diagram of the GitHub repository apache/storm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/storm
- Repository: https://github.com/apache/storm

## Overview

##### Project type and purpose
- Apache Storm is a distributed, real-time stream processing system.
- It provides primitives (spouts, bolts) for building continuous computation topologies.
- Key use cases: real-time ETL, continuous analytics, event processing, micro-batch streaming.

##### High-level architecture components
a. Client/API layer
- StormSubmitter / LocalCluster: submits topologies via Thrift RPC to Nimbus.
- Thrift interfaces (Java, Python, Clojure) for cross-language support.
b. Cluster manager
- Nimbus (master): receives topology definitions, schedules tasks, coordinates with Zookeeper.
- Supervisor daemons on each node: launch and monitor worker processes.
- Worker processes: run executors (spout/bolt tasks) in JVMs or external languages.
c. Coordination & Storage
- Zookeeper: leader election for Nimbus HA, stores cluster state, assignments, and credentials.
- BlobStore: distributes JARs and resources across the cluster.
d. Messaging layer
- Netty (transport): delivers tuples between workers with optional backpressure.
- Local (“JCQueue”) mode for non-networked single-node testing.
e. Topology execution
- Spout → Bolt DAG: defines streams and groupings.
- Reliable processing via anchoring and acking framework.
- Trident API for micro-batch and stateful processing.
f. External connectors
- Modules under external/: HDFS, Kafka, Redis, JDBC, JMS, metrics, SQL, etc.
- Each provides spouts/bolts/trident state implementations.
g. Management & UI
- Storm-webapp: REST API and UI for monitoring topologies, logs, metrics.
- Metrics modules (prometheus, HDRHistogram, Sigar) feed metrics to external systems.
h. Deployment artifacts
- storm-dist: binary and source distributions.
- Docker / seccomp configs under conf/, bin/ and aws-bin/.

##### Relationships & interactions
- Client submits topology to Nimbus → Nimbus writes state to ZK.
- Nimbus computes assignments → updates ZK → Supervisors watch ZK and launch workers.
- Workers connect to each other via Netty → exchange tuples.
- Spout emits tuples → Bolts process and optionally ack.
- Metrics from workers and Nimbus flow to metrics consumers or Prometheus.
- UI queries Nimbus and ZK to display cluster and topology state.

##### Architectural patterns & design principles
- Master-worker pattern (Nimbus vs. Supervisors/Workers).
- Pub-sub messaging for tuple delivery.
- DAG-based streaming for topology definition.
- Pluggable components (serialization, grouping, metrics, credentials).
- Externalized coordination using Zookeeper for HA and consistency.
- Container/resource isolation (cgroups, OCI, Docker wrappers).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client/API Layer
    subgraph "Client/API Layer"
        CS["StormSubmitter<br/>(Java Thrift)"]:::client
        LC["LocalCluster<br/>(Java)"]:::client
        IDL["Thrift IDL & Interfaces"]:::client
    end

    %% Control Plane
    subgraph "Cluster Manager"
        NIM["Nimbus<br/>(Master)"]:::control
        SUP["Supervisor Daemon"]:::control
        WORKERPROC["Worker Process Launcher"]:::control
    end

    %% Coordination & Storage
    subgraph "Coordination & Storage"
        ZK["Zookeeper"]:::storage
        BB["BlobStore"]:::storage
    end

    %% Data Plane
    subgraph "Data Plane"
        TB["TopologyBuilder & DAG"]:::data
        RB["Reliable Processing"]:::data
        TRID["Trident API"]:::data
        SP["Spout Task"]:::data
        BLT["Bolt Task"]:::data
        NETTY["Netty Transport"]:::data
        LCQ["Local JCQueue"]:::data
    end

    %% External Systems
    subgraph "External Systems"
        HDFS["HDFS"]:::external
        KFK["Kafka"]:::external
        RED["Redis"]:::external
        JDBC["JDBC"]:::external
        JMS["JMS"]:::external
        PROM["Prometheus"]:::external
        GEN["Generic Metrics"]:::external
    end

    %% UI/Monitoring
    subgraph "UI/Monitoring"
        WEB["Storm Web UI & REST"]:::ui
        DRPC["DRPC Endpoints"]:::ui
        LOGV["Logviewer"]:::ui
    end

    %% Flows
    CS -->|"Thrift RPC submit"| NIM
    LC -->|"Thrift RPC submit"| NIM
    IDL -->|"IDL support"| CS

    NIM -->|"writes state"| ZK
    NIM -->|"compute assignments"| ZK
    ZK -->|"watch & assignment"| SUP
    SUP -->|"launch"| WORKERPROC

    WORKERPROC -->|"launch executor JVMs"| SP
    WORKERPROC -->|"launch executor JVMs"| BLT

    SP -->|"emit tuples"| NETTY
    BLT -->|"process tuples"| NETTY
    NETTY -->|"deliver tuples"| SP
    NETTY -->|"deliver tuples"| BLT
    LCQ -->|"local queue mode"| SP

    TB -->|"defines DAG"| SP
    TB -->|"defines DAG"| BLT
    RB -->|"acking/anchoring"| SP
    RB -->|"acking/anchoring"| BLT
    TRID -->|"micro-batch/state"| SP
    TRID -->|"micro-batch/state"| BLT

    SP -->|"reads from"| HDFS
    SP -->|"reads from"| KFK
    BLT -->|"writes to"| RED
    BLT -->|"writes to"| JDBC
    BLT -->|"writes to"| JMS

    SP -->|"metrics"| PROM
    BLT -->|"metrics"| PROM
    NIM -->|"metrics"| PROM

    WEB -->|"queries"| NIM
    WEB -->|"queries"| ZK
    DRPC -->|"invokes"| NIM
    LOGV -->|"fetch logs"| WORKERPROC

    %% Click Events
    click CS "https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/StormSubmitter.java"
    click LC "https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/LocalCluster.java"
    click IDL "https://github.com/apache/storm/blob/master/storm-client/src/storm.thrift"
    click NIM "https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java"
    click SUP "https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Supervisor.java"
    click WORKERPROC "https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java"
    click ZK "https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apache/storm/zookeeper/"
    click BB "https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/blobstore/BlobStore.java"
    click NETTY "https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apache/storm/messaging/netty/"
    click LCQ "https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/utils/JCQueue.java"
    click SP "https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apache/storm/spout/"
    click BLT "https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apache/storm/bolt/"
    click TB "https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java"
    click RB "https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/coordination/BatchBoltExecutor.java"
    click TRID "https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apache/storm/trident/"
    click HDFS "https://github.com/apache/storm/tree/master/external/storm-hdfs/"
    click KFK "https://github.com/apache/storm/tree/master/external/storm-kafka-client/"
    click RED "https://github.com/apache/storm/tree/master/external/storm-redis/"
    click JDBC "https://github.com/apache/storm/tree/master/external/storm-jdbc/"
    click JMS "https://github.com/apache/storm/tree/master/external/storm-jms/"
    click PROM "https://github.com/apache/storm/tree/master/external/storm-metrics-prometheus/"
    click GEN "https://github.com/apache/storm/tree/master/external/storm-metrics/"
    click WEB "https://github.com/apache/storm/tree/master/storm-webapp/src/main/java/org/apache/storm/daemon/ui/"
    click DRPC "https://github.com/apache/storm/tree/master/storm-webapp/src/main/java/org/apache/storm/daemon/drpc/"
    click LOGV "https://github.com/apache/storm/tree/master/storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/"

    %% Styles
    classDef client fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef control fill:#E6FFEB,stroke:#28A745,color:#28A745;
    classDef storage fill:#F3E5FF,stroke:#6F42C1,color:#6F42C1;
    classDef data fill:#FFF4E5,stroke:#D97706,color:#D97706;
    classDef external fill:#F0F0F0,stroke:#6A737D,color:#6A737D;
    classDef ui fill:#E6FFFA,stroke:#17A2B8,color:#17A2B8;
```
