How apache/storm works
Project type and purpose
• Apache Storm is a distributed, real-time stream processing system.
• It provides primitives (spouts, bolts) for building continuous computation topologies.
• Key use cases: real-time ETL, continuous analytics, event processing, micro-batch streaming.
High-level architecture components
a. Client/API layer
• StormSubmitter / LocalCluster: submits topologies via Thrift RPC to Nimbus.
• Thrift interfaces (Java, Python, Clojure) for cross-language support.
b. Cluster manager
• Nimbus (master): receives topology definitions, schedules tasks, coordinates with Zookeeper.
• Supervisor daemons on each node: launch and monitor worker processes.
• Worker processes: run executors (spout/bolt tasks) in JVMs or external languages.
c. Coordination & Storage
• Zookeeper: leader election for Nimbus HA, stores cluster state, assignments, and credentials.
• BlobStore: distributes JARs and resources across the cluster.
d. Messaging layer
• Netty (transport): delivers tuples between workers with optional backpressure.
• Local (“JCQueue”) mode for non-networked single-node testing.
e. Topology execution
• Spout → Bolt DAG: defines streams and groupings.
• Reliable processing via anchoring and acking framework.
• Trident API for micro-batch and stateful processing.
f. External connectors
• Modules under external/: HDFS, Kafka, Redis, JDBC, JMS, metrics, SQL, etc.
• Each provides spouts/bolts/trident state implementations.
g. Management & UI
• Storm-webapp: REST API and UI for monitoring topologies, logs, metrics.
• Metrics modules (prometheus, HDRHistogram, Sigar) feed metrics to external systems.
h. Deployment artifacts
• storm-dist: binary and source distributions.
• Docker / seccomp configs under conf/, bin/ and aws-bin/.
Relationships & interactions
• Client submits topology to Nimbus → Nimbus writes state to ZK.
• Nimbus computes assignments → updates ZK → Supervisors watch ZK and launch workers.
• Workers connect to each other via Netty → exchange tuples.
• Spout emits tuples → Bolts process and optionally ack.
• Metrics from workers and Nimbus flow to metrics consumers or Prometheus.
• UI queries Nimbus and ZK to display cluster and topology state.
Architectural patterns & design principles
• Master-worker pattern (Nimbus vs. Supervisors/Workers).
• Pub-sub messaging for tuple delivery.
• DAG-based streaming for topology definition.
• Pluggable components (serialization, grouping, metrics, credentials).
• Externalized coordination using Zookeeper for HA and consistency.
• Container/resource isolation (cgroups, OCI, Docker wrappers).