How apache/pinot works
Apache Pinot is a distributed real-time OLAP datastore for low-latency SQL analytics over streaming and batch data. The primary workflow is a client query entering a broker, being planned and routed to server-side query execution, then returning results; Pinot also supports ingestion, indexing, upsert, administration, and workload controls. The graph combines documented product capabilities with sampled query-planning, server, quota, and workload code. Some cross-service runtime links are shown as dashed and unverified because the relevant dispatch code was not sampled.
Main entry points: WorkerQueryServer.java Worker query server · BaseBrokerRequestHandler.java Query handler · BaseBrokerRoutingManager.java Broker routing · pinot-stream-ingestion Batch and stream ingest
Components 24
Query clients
- Java client
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/Connection.java - JDBC client
pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotConnection.java
Broker and routing
- Broker API
pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java - Query handler
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java - Query quotas
pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java - Broker routing
pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java - Segment selection
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/DefaultSegmentSelector.java
Planning and execution
- Query model
pinot-common/src/main/java/org/apache/pinot/common/request/PinotQuery.java - Query planner
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java - Leaf routing query
pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java - Worker query server
pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java - Query executor
pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/resources/QueryExecutorService.java - Broker reduction
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java - Materialized views
pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngine.java
Ingestion and storage
- Batch and stream ingest
pinot-plugins/pinot-stream-ingestion - Segment creation
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/TransformPipeline.java - Pluggable indexes
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/IndexService.java - Realtime upsert
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java - Distributed datastore
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
Cluster management
- Cluster controller
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java - Workload manager
pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java - Runtime configuration
pinot-spi/src/main/java/org/apache/pinot/spi/env/PinotConfiguration.java
Other components
- Application user
- Stream and batch sources
Connections 22
- Application user → Java client: submits query
- Application user → JDBC client: submits SQL
- Java client → Broker API: sends query
- JDBC client → Broker API: sends query
- Broker API → Query handler: handles request
- Query handler → Query quotas: checks quota
- Query handler → Broker routing: routes query
- Broker routing → Segment selection: selects segments
- Query planner → Leaf routing query: builds routing query
pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java - Leaf routing query → Query model: builds query model
pinot-query-planner/src/main/java/org/apache/pinot/query/routing/PlanNodeRoutingQueryBuilder.java - Broker routing → Worker query server: dispatches stages
- Worker query server → Query executor: executes query
- Worker query server → Broker reduction: returns results
- Query planner → Materialized views: rewrites query
- Stream and batch sources → Batch and stream ingest: supplies records
README.md - Batch and stream ingest → Segment creation: creates segments
- Segment creation → Pluggable indexes: builds indexes
- Batch and stream ingest → Realtime upsert: updates records
- Segment creation → Distributed datastore: stores segments
- Cluster controller → Workload manager: manages workloads
pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java - Workload manager → Worker query server: propagates workload
pinot-controller/src/main/java/org/apache/pinot/controller/workload/QueryWorkloadManager.java - Runtime configuration → Worker query server: configures runtime
pinot-server/src/main/java/org/apache/pinot/server/worker/WorkerQueryServer.java