How netflix/mantis works
Project Type and Purpose
• This is an open-source, multi-module, distributed stream-processing platform (Netflix Mantis). It provides:
• A control plane (master) that accepts job submissions, schedules work, tracks state, and provides APIs.
• Clients and SDKs for submitting and managing jobs.
• Worker agents and executors that run user-defined streaming jobs.
• Pluggable connectors for data ingestion (e.g., Kafka, Iceberg) and data publish.
• A runtime library implementing reactive streams and job lifecycle.
File-Structure Analysis
• Top-level modules (Gradle subprojects):
• mantis-control-plane-server & mantis-control-plane-core & mantis-control-plane-store-dynamodb – the master service, persistence, leader election (Akka/ZooKeeper/DynamoDB).
• mantis-control-plane-client & mantis-client – Java/CLI clients that talk to the master via HTTP/REST or SSE.
• mantis-server-agent & mantis-server-worker & mantis-runtime-executor & mantis-runtime-loader – frameworks to start worker processes, load jobs, run task executors, heartbeat/push status.
• mantis-runtime – the core library for user-defined jobs (DSL, stages, sources, sinks, reactive operators).
• mantis-connectors – pluggable modules to source/sink data (Kafka, Iceberg, job-source, publish).
• mantis-publish & mantis-publish-netty[-guice] – publish/subscribe subsystem for streaming events across jobs.
• mantis-network & mantis-remote-observable – network layers (Netty, SSE) for push/pull event delivery.
• docs, examples, testcontainers, discovery-proto, common modules (serde, common utilities).
README Insights
• Core/common code separated into mantis-common and mantis-common-serde.
• Control-plane core shared by server and clients.
• Reactive, actor-based scheduling and messaging.
• Plugin architecture for connectors and publish clients.