How apache/impala works
Impala is a distributed SQL engine for querying, transforming, and combining data across open storage systems. The principal workflow runs from a SQL client through frontend parsing and planning, admission control, and distributed backend execution, then reads or writes supported storage formats. Catalog metadata and metastore events support that workflow; query hooks and tracing provide separate observability extensions. The README establishes the product capabilities, but the sampled code is incomplete, so several workflow links are documented at subsystem level rather than verified as direct calls.
Main entry points: service Impala service · planner Query planning · exec-env.h Backend environment · exec Query operators
Components 20
SQL access
- Impala shell
shell/impala_shell/impala_client.py - Impala service
be/src/service
Frontend and catalog
- SQL parser
fe/src/main/java/org/apache/impala/analysis - Query planning
fe/src/main/java/org/apache/impala/planner - Catalog service
be/src/catalog - Metastore events
fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
Query control
- Admission control
be/src/scheduling/admission-controller.cc - Request pools
fe/src/main/java/org/apache/impala/util/RequestPoolService.java - Pool configuration
java/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
Backend execution
- Backend environment
be/src/runtime/exec-env.h - Query operators
be/src/exec - SQL expressions
be/src/exprs - LLVM code generation
be/src/codegen
Storage and observability
- File format readers
be/src/exec - Data sinks
be/src/exec - External storage
- Query event hooks
fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java - Hook API
java/query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java - OpenTelemetry tracing
be/src/observe/otel-trace-manager.cc
Other components
- SQL user
Connections 22
- SQL user → Impala shell: submits SQL
- Impala shell → Impala service: sends HS2 requests
- Impala service → SQL parser: routes queries
- SQL parser → Query planning: parses for planning
- Query planning → Catalog service: uses metadata
- Query planning → Admission control: submits plan
- Admission control → Request pools: checks pool policy
be/src/scheduling/admission-controller.cc - Request pools → Pool configuration: loads allocations
fe/src/main/java/org/apache/impala/util/RequestPoolService.java - Admission control → Backend environment: uses backend environment
be/src/scheduling/admission-controller.cc - Backend environment → Impala service: exposes server
be/src/runtime/exec-env.h - Backend environment → Request pools: exposes pool service
be/src/runtime/exec-env.h - Backend environment → Query operators: provides query execution
- Query operators → SQL expressions: evaluates expressions
- Query operators → File format readers: scans data
- Query operators → Data sinks: writes results
- File format readers → External storage: reads supported formats
- Data sinks → External storage: writes data
- Query planning → LLVM code generation: generates query code
- Metastore events → Catalog service: dispatches metadata events
fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java - Query event hooks → Hook API: invokes hooks
fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java - Impala service → Query event hooks: runs query hooks
- Impala service → OpenTelemetry tracing: traces queries
be/src/observe/otel-trace-manager.cc