# apache/impala architecture

> Architecture diagram of the GitHub repository apache/impala, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/impala
- Repository: https://github.com/apache/impala

## Overview

Impala is a distributed SQL engine for querying, transforming, and combining data across open storage systems. The principal workflow runs from a SQL client through frontend parsing and planning, admission control, and distributed backend execution, then reads or writes supported storage formats. Catalog metadata and metastore events support that workflow; query hooks and tracing provide separate observability extensions. The README establishes the product capabilities, but the sampled code is incomplete, so several workflow links are documented at subsystem level rather than verified as direct calls.

Main entry points: [`be/src/service`](https://github.com/apache/impala/tree/master/be/src/service) (Impala service), [`fe/src/main/java/org/apache/impala/planner`](https://github.com/apache/impala/tree/master/fe/src/main/java/org/apache/impala/planner) (Query planning), [`be/src/runtime/exec-env.h`](https://github.com/apache/impala/blob/master/be/src/runtime/exec-env.h) (Backend environment), [`be/src/exec`](https://github.com/apache/impala/tree/master/be/src/exec) (Query operators)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["SQL access"]
  node_shell["Impala shell<br/>[impala_client.py]"]
  node_service["Impala service"]
end

subgraph group_frontend["Frontend and catalog"]
  node_parser["SQL parser"]
  node_planner["Query planning"]
  node_catalog["Catalog service"]
  node_metastore["Metastore events"]
end

subgraph group_control["Query control"]
  node_admission["Admission control"]
  node_pool["Request pools"]
  node_allocation["Pool configuration"]
end

subgraph group_execution["Backend execution"]
  node_environment["Backend environment<br/>[exec-env.h]"]
  node_operators["Query operators"]
  node_expressions["SQL expressions"]
  node_codegen["LLVM code generation"]
end

subgraph group_data["Storage and observability"]
  node_formats["File format readers"]
  node_sinks["Data sinks"]
  node_storage[("External storage")]
  node_hooks["Query event hooks"]
  node_hookapi["Hook API"]
  node_tracing["OpenTelemetry tracing"]
end

node_user(("SQL user"))

node_user -->|"submits SQL"| node_shell
node_shell -.->|"sends HS2 requests"| node_service
node_service -.->|"routes queries"| node_parser
node_parser -.->|"parses for planning"| node_planner
node_planner -.->|"uses metadata"| node_catalog
node_planner -.->|"submits plan"| node_admission
node_admission -->|"checks pool policy"| node_pool
node_pool -->|"loads allocations"| node_allocation
node_admission -->|"uses backend environment"| node_environment
node_environment -->|"exposes server"| node_service
node_environment -->|"exposes pool service"| node_pool
node_environment -.->|"provides query execution"| node_operators
node_operators -.->|"evaluates expressions"| node_expressions
node_operators -.->|"scans data"| node_formats
node_operators -.->|"writes results"| node_sinks
node_formats -->|"reads supported formats"| node_storage
node_sinks -.->|"writes data"| node_storage
node_planner -->|"generates query code"| node_codegen
node_metastore -->|"dispatches metadata events"| node_catalog
node_hooks -->|"invokes hooks"| node_hookapi
node_service -.->|"runs query hooks"| node_hooks
node_service -->|"traces queries"| node_tracing

click node_shell "https://github.com/apache/impala/blob/master/shell/impala_shell/impala_client.py"
click node_service "https://github.com/apache/impala/tree/master/be/src/service"
click node_parser "https://github.com/apache/impala/tree/master/fe/src/main/java/org/apache/impala/analysis"
click node_planner "https://github.com/apache/impala/tree/master/fe/src/main/java/org/apache/impala/planner"
click node_catalog "https://github.com/apache/impala/tree/master/be/src/catalog"
click node_metastore "https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java"
click node_admission "https://github.com/apache/impala/blob/master/be/src/scheduling/admission-controller.cc"
click node_pool "https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/util/RequestPoolService.java"
click node_allocation "https://github.com/apache/impala/blob/master/java/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java"
click node_environment "https://github.com/apache/impala/blob/master/be/src/runtime/exec-env.h"
click node_operators "https://github.com/apache/impala/tree/master/be/src/exec"
click node_expressions "https://github.com/apache/impala/tree/master/be/src/exprs"
click node_codegen "https://github.com/apache/impala/tree/master/be/src/codegen"
click node_formats "https://github.com/apache/impala/tree/master/be/src/exec"
click node_sinks "https://github.com/apache/impala/tree/master/be/src/exec"
click node_hooks "https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java"
click node_hookapi "https://github.com/apache/impala/blob/master/java/query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java"
click node_tracing "https://github.com/apache/impala/blob/master/be/src/observe/otel-trace-manager.cc"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_shell,node_service,node_user toneBlue
class node_parser,node_planner,node_catalog,node_metastore toneAmber
class node_admission,node_pool,node_allocation toneMint
class node_environment,node_operators,node_expressions,node_codegen toneRose
class node_formats,node_sinks,node_storage,node_hooks,node_hookapi,node_tracing toneIndigo
```

## Components

### SQL access

- [Impala shell](https://github.com/apache/impala/blob/master/shell/impala_shell/impala_client.py): `shell/impala_shell/impala_client.py`
- [Impala service](https://github.com/apache/impala/tree/master/be/src/service): `be/src/service`

### Frontend and catalog

- [SQL parser](https://github.com/apache/impala/tree/master/fe/src/main/java/org/apache/impala/analysis): `fe/src/main/java/org/apache/impala/analysis`
- [Query planning](https://github.com/apache/impala/tree/master/fe/src/main/java/org/apache/impala/planner): `fe/src/main/java/org/apache/impala/planner`
- [Catalog service](https://github.com/apache/impala/tree/master/be/src/catalog): `be/src/catalog`
- [Metastore events](https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java): `fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java`

### Query control

- [Admission control](https://github.com/apache/impala/blob/master/be/src/scheduling/admission-controller.cc): `be/src/scheduling/admission-controller.cc`
- [Request pools](https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/util/RequestPoolService.java): `fe/src/main/java/org/apache/impala/util/RequestPoolService.java`
- [Pool configuration](https://github.com/apache/impala/blob/master/java/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java): `java/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java`

### Backend execution

- [Backend environment](https://github.com/apache/impala/blob/master/be/src/runtime/exec-env.h): `be/src/runtime/exec-env.h`
- [Query operators](https://github.com/apache/impala/tree/master/be/src/exec): `be/src/exec`
- [SQL expressions](https://github.com/apache/impala/tree/master/be/src/exprs): `be/src/exprs`
- [LLVM code generation](https://github.com/apache/impala/tree/master/be/src/codegen): `be/src/codegen`

### Storage and observability

- [File format readers](https://github.com/apache/impala/tree/master/be/src/exec): `be/src/exec`
- [Data sinks](https://github.com/apache/impala/tree/master/be/src/exec): `be/src/exec`
- **External storage**
- [Query event hooks](https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java): `fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java`
- [Hook API](https://github.com/apache/impala/blob/master/java/query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java): `java/query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java`
- [OpenTelemetry tracing](https://github.com/apache/impala/blob/master/be/src/observe/otel-trace-manager.cc): `be/src/observe/otel-trace-manager.cc`

### Other components

- **SQL user**

## Connections

- SQL user → Impala shell: submits SQL
- Impala shell → Impala service: sends HS2 requests
- Impala service → SQL parser: routes queries
- SQL parser → Query planning: parses for planning
- Query planning → Catalog service: uses metadata
- Query planning → Admission control: submits plan
- Admission control → Request pools: checks pool policy (evidence: `be/src/scheduling/admission-controller.cc`)
- Request pools → Pool configuration: loads allocations (evidence: `fe/src/main/java/org/apache/impala/util/RequestPoolService.java`)
- Admission control → Backend environment: uses backend environment (evidence: `be/src/scheduling/admission-controller.cc`)
- Backend environment → Impala service: exposes server (evidence: `be/src/runtime/exec-env.h`)
- Backend environment → Request pools: exposes pool service (evidence: `be/src/runtime/exec-env.h`)
- Backend environment → Query operators: provides query execution
- Query operators → SQL expressions: evaluates expressions
- Query operators → File format readers: scans data
- Query operators → Data sinks: writes results
- File format readers → External storage: reads supported formats
- Data sinks → External storage: writes data
- Query planning → LLVM code generation: generates query code
- Metastore events → Catalog service: dispatches metadata events (evidence: `fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java`)
- Query event hooks → Hook API: invokes hooks (evidence: `fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java`)
- Impala service → Query event hooks: runs query hooks
- Impala service → OpenTelemetry tracing: traces queries (evidence: `be/src/observe/otel-trace-manager.cc`)
