# apache/pinot architecture

> Architecture diagram of the GitHub repository apache/pinot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/pinot
- Repository: https://github.com/apache/pinot

## Overview

Apache Pinot is a distributed real-time OLAP datastore for low-latency SQL analytics over streaming and batch data. The primary workflow is a client query entering a broker, being planned and routed to server-side query execution, then returning results; Pinot also supports ingestion, indexing, upsert, administration, and workload controls. The graph combines documented product capabilities with sampled query-planning, server, quota, and workload code. Some cross-service runtime links are shown as dashed and unverified because the relevant dispatch code was not sampled.

Main entry points: [`pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java`](https://github.com/apache/pinot/blob/master/pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java) (Worker query server), [`pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java`](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java) (Query handler), [`pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java`](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java) (Broker routing), [`pinot-plugins/pinot-stream-ingestion`](https://github.com/apache/pinot/tree/master/pinot-plugins/pinot-stream-ingestion) (Batch and stream ingest)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Query clients"]
  node_client["Java client<br/>[Connection.java]"]
  node_jdbc["JDBC client"]
end

subgraph group_broker["Broker and routing"]
  node_broker_api["Broker API"]
  node_handler["Query handler"]
  node_quota["Query quotas"]
  node_routing["Broker routing"]
  node_selector["Segment selection"]
end

subgraph group_execution["Planning and execution"]
  node_query_model["Query model<br/>[PinotQuery.java]"]
  node_planner["Query planner"]
  node_routing_builder["Leaf routing query"]
  node_worker["Worker query server"]
  node_executor["Query executor"]
  node_reducer["Broker reduction"]
  node_mv_rewrite["Materialized views"]
end

subgraph group_data["Ingestion and storage"]
  node_ingest["Batch and stream ingest"]
  node_segment["Segment creation"]
  node_indexes["Pluggable indexes<br/>[IndexService.java]"]
  node_upsert["Realtime upsert"]
  node_store[("Distributed datastore")]
end

subgraph group_control["Cluster management"]
  node_controller["Cluster controller"]
  node_workloads["Workload manager"]
  node_config["Runtime configuration"]
end

node_user(("Application user"))
node_ingest_sources(("Stream and batch sources"))

node_user -.->|"submits query"| node_client
node_user -.->|"submits SQL"| node_jdbc
node_client -.->|"sends query"| node_broker_api
node_jdbc -.->|"sends query"| node_broker_api
node_broker_api -.->|"handles request"| node_handler
node_handler -.->|"checks quota"| node_quota
node_handler -.->|"routes query"| node_routing
node_routing -.->|"selects segments"| node_selector
node_planner -->|"builds routing query"| node_routing_builder
node_routing_builder -->|"builds query model"| node_query_model
node_routing -.->|"dispatches stages"| node_worker
node_worker -.->|"executes query"| node_executor
node_worker -.->|"returns results"| node_reducer
node_planner -.->|"rewrites query"| node_mv_rewrite
node_ingest_sources -->|"supplies records"| node_ingest
node_ingest -.->|"creates segments"| node_segment
node_segment -.->|"builds indexes"| node_indexes
node_ingest -.->|"updates records"| node_upsert
node_segment -.->|"stores segments"| node_store
node_controller -.->|"manages workloads"| node_workloads
node_workloads -->|"propagates workload"| node_worker
node_config -.->|"configures runtime"| node_worker

click node_client "https://github.com/apache/pinot/blob/master/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/Connection.java"
click node_jdbc "https://github.com/apache/pinot/blob/master/pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotConnection.java"
click node_broker_api "https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java"
click node_handler "https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java"
click node_quota "https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java"
click node_routing "https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java"
click node_selector "https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/DefaultSegmentSelector.java"
click node_query_model "https://github.com/apache/pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/request/PinotQuery.java"
click node_planner "https://github.com/apache/pinot/blob/master/pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java"
click node_routing_builder "https://github.com/apache/pinot/blob/master/pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java"
click node_worker "https://github.com/apache/pinot/blob/master/pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java"
click node_executor "https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/resources/QueryExecutorService.java"
click node_reducer "https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java"
click node_mv_rewrite "https://github.com/apache/pinot/blob/master/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngine.java"
click node_ingest "https://github.com/apache/pinot/tree/master/pinot-plugins/pinot-stream-ingestion"
click node_segment "https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/TransformPipeline.java"
click node_indexes "https://github.com/apache/pinot/blob/master/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/IndexService.java"
click node_upsert "https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java"
click node_store "https://github.com/apache/pinot/blob/master/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java"
click node_controller "https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java"
click node_workloads "https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java"
click node_config "https://github.com/apache/pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/env/PinotConfiguration.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client,node_jdbc,node_user toneBlue
class node_broker_api,node_handler,node_quota,node_routing,node_selector toneAmber
class node_query_model,node_planner,node_routing_builder,node_worker,node_executor,node_reducer,node_mv_rewrite toneMint
class node_ingest,node_segment,node_indexes,node_upsert,node_store toneRose
class node_controller,node_workloads,node_config,node_ingest_sources toneIndigo
```

## Components

### Query clients

- [Java client](https://github.com/apache/pinot/blob/master/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/Connection.java): `pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/Connection.java`
- [JDBC client](https://github.com/apache/pinot/blob/master/pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotConnection.java): `pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotConnection.java`

### Broker and routing

- [Broker API](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java): `pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java`
- [Query handler](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java): `pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java`
- [Query quotas](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java): `pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java`
- [Broker routing](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java): `pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java`
- [Segment selection](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/DefaultSegmentSelector.java): `pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/DefaultSegmentSelector.java`

### Planning and execution

- [Query model](https://github.com/apache/pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/request/PinotQuery.java): `pinot-common/src/main/java/org/apache/pinot/common/request/PinotQuery.java`
- [Query planner](https://github.com/apache/pinot/blob/master/pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java): `pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java`
- [Leaf routing query](https://github.com/apache/pinot/blob/master/pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java): `pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java`
- [Worker query server](https://github.com/apache/pinot/blob/master/pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java): `pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java`
- [Query executor](https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/resources/QueryExecutorService.java): `pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/resources/QueryExecutorService.java`
- [Broker reduction](https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java): `pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java`
- [Materialized views](https://github.com/apache/pinot/blob/master/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngine.java): `pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngine.java`

### Ingestion and storage

- [Batch and stream ingest](https://github.com/apache/pinot/tree/master/pinot-plugins/pinot-stream-ingestion): `pinot-plugins/pinot-stream-ingestion`
- [Segment creation](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/TransformPipeline.java): `pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/TransformPipeline.java`
- [Pluggable indexes](https://github.com/apache/pinot/blob/master/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/IndexService.java): `pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/IndexService.java`
- [Realtime upsert](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java): `pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java`
- [Distributed datastore](https://github.com/apache/pinot/blob/master/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java): `pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java`

### Cluster management

- [Cluster controller](https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java): `pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java`
- [Workload manager](https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java): `pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java`
- [Runtime configuration](https://github.com/apache/pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/env/PinotConfiguration.java): `pinot-spi/src/main/java/org/apache/pinot/spi/env/PinotConfiguration.java`

### Other components

- **Application user**
- **Stream and batch sources**

## Connections

- Application user → Java client: submits query
- Application user → JDBC client: submits SQL
- Java client → Broker API: sends query
- JDBC client → Broker API: sends query
- Broker API → Query handler: handles request
- Query handler → Query quotas: checks quota
- Query handler → Broker routing: routes query
- Broker routing → Segment selection: selects segments
- Query planner → Leaf routing query: builds routing query (evidence: `pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java`)
- Leaf routing query → Query model: builds query model (evidence: `pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java`)
- Broker routing → Worker query server: dispatches stages
- Worker query server → Query executor: executes query
- Worker query server → Broker reduction: returns results
- Query planner → Materialized views: rewrites query
- Stream and batch sources → Batch and stream ingest: supplies records (evidence: `README.md`)
- Batch and stream ingest → Segment creation: creates segments
- Segment creation → Pluggable indexes: builds indexes
- Batch and stream ingest → Realtime upsert: updates records
- Segment creation → Distributed datastore: stores segments
- Cluster controller → Workload manager: manages workloads (evidence: `pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java`)
- Workload manager → Worker query server: propagates workload (evidence: `pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java`)
- Runtime configuration → Worker query server: configures runtime (evidence: `pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java`)
