# pola-rs/polars architecture

> Architecture diagram of the GitHub repository pola-rs/polars, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pola-rs/polars
- Repository: https://github.com/pola-rs/polars

## Overview

Polars is a Rust analytical query engine exposed through DataFrame, SQL, and Python APIs. A typical workflow builds a lazy query from data, expresses transformations, optimizes a shared plan, executes it in parallel, and returns a DataFrame; SQL queries enter that same optimizer and engine. The repository also includes Arrow interchange, streaming execution, disk spilling, and optional GPU and distributed engines. The map combines README-documented capabilities with sampled paths and source-index references; unsampled implementations and their internal call relationships are not inferred.

Main entry points: [`crates/polars-sql/src/context.rs`](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/context.rs) (SQL context), [`py-polars/src/polars/dataframe/frame.py`](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/dataframe/frame.py) (Python DataFrame API), [`crates/polars-lazy/src/frame/mod.rs`](https://github.com/pola-rs/polars/blob/main/crates/polars-lazy/src/frame/mod.rs) (Lazy query API), [`crates/polars-sql/src/subquery.rs`](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/subquery.rs) (SQL subquery rewrites)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["User APIs"]
  node_python_api["Python DataFrame API<br/>[frame.py]"]
  node_python_expr["Python expressions<br/>[datetime.py]"]
  node_python_types["Python type definitions<br/>[_typing.py]"]
  node_python_helpers["Python utilities<br/>[various.py]"]
  node_sql_context["SQL context<br/>[context.rs]"]
  node_sql_expr["SQL expression conversion<br/>[sql_expr.rs]"]
  node_sql_functions["SQL functions<br/>[functions.rs]"]
  node_sql_subqueries["SQL subquery rewrites<br/>[subquery.rs]"]
  node_sql_pullup["Subquery pull-up"]
  node_sql_visitors["SQL AST analysis<br/>[sql_visitors.rs]"]
  node_sql_dynamic_visit["Dynamic AST traversal<br/>[dyn_visit.rs]"]
  node_result["DataFrame result<br/>[query_result.rs]"]
end

subgraph group_query["Query Planning"]
  node_lazyframe["Lazy query API<br/>[mod.rs]"]
  node_optimizer["Query optimizer<br/>[mod.rs]"]
end

subgraph group_runtime["Execution Runtime"]
  node_compute["Vectorized compute<br/>[lib.rs]"]
  node_streaming["Streaming execution"]
  node_spill["Memory and spilling<br/>[memory_manager.rs]"]
end

subgraph group_data["Data and Interchange"]
  node_arrow["Arrow columnar data<br/>[lib.rs]"]
  node_io["Data I/O"]
end

subgraph group_integration["Optional Engines"]
  node_python_binding["Python Rust binding<br/>[_plr.py]"]
  node_engine_config["Engine configuration<br/>[engine_config.py]"]
  node_remote_engine["Remote engine<br/>[engine_remote.py]"]
  node_gpu_engine["GPU engine<br/>[engine.py]"]
end

node_user(("Query author"))

node_user -->|"builds queries"| node_python_api
node_user -->|"writes SQL"| node_sql_context
node_python_api -->|"composes expressions"| node_python_expr
node_python_api -->|"collects lazy query"| node_lazyframe
node_sql_context -->|"converts expressions"| node_sql_expr
node_sql_context -->|"handles functions"| node_sql_functions
node_sql_context -->|"rewrites subqueries"| node_sql_subqueries
node_sql_subqueries -->|"uses pull-up rewrites"| node_sql_pullup
node_sql_context -->|"plans SQL"| node_lazyframe
node_sql_expr -->|"resolves functions"| node_sql_functions
node_sql_context -->|"analyzes SQL AST"| node_sql_visitors
node_sql_visitors -->|"traverses AST"| node_sql_dynamic_visit
node_sql_subqueries -->|"visits subqueries"| node_sql_dynamic_visit
node_lazyframe -->|"optimizes plan"| node_optimizer
node_optimizer -->|"executes plan"| node_compute
node_compute -->|"returns result"| node_result
node_io -->|"uses columnar data"| node_arrow
node_arrow -->|"shares data"| node_python_binding
node_python_binding -->|"exposes Rust API"| node_python_api
node_optimizer -.->|"executes streaming"| node_streaming
node_compute -.->|"spills when needed"| node_spill
node_lazyframe -.->|"can execute remotely"| node_remote_engine
node_lazyframe -.->|"can use GPU"| node_gpu_engine
node_python_api -->|"configures engine"| node_engine_config
node_engine_config -->|"sets engine options"| node_python_binding

click node_python_api "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/dataframe/frame.py"
click node_python_expr "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/expr/datetime.py"
click node_python_types "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/_typing.py"
click node_python_helpers "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/_utils/various.py"
click node_sql_context "https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/context.rs"
click node_sql_expr "https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/sql_expr.rs"
click node_sql_functions "https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/functions.rs"
click node_sql_subqueries "https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/subquery.rs"
click node_sql_pullup "https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/subquery_pull_up.rs"
click node_sql_visitors "https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/sql_visitors.rs"
click node_sql_dynamic_visit "https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/dyn_visit.rs"
click node_lazyframe "https://github.com/pola-rs/polars/blob/main/crates/polars-lazy/src/frame/mod.rs"
click node_optimizer "https://github.com/pola-rs/polars/blob/main/crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs"
click node_compute "https://github.com/pola-rs/polars/blob/main/crates/polars-compute/src/lib.rs"
click node_streaming "https://github.com/pola-rs/polars/blob/main/crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs"
click node_spill "https://github.com/pola-rs/polars/blob/main/crates/polars-ooc/src/memory_manager.rs"
click node_arrow "https://github.com/pola-rs/polars/blob/main/crates/polars-arrow/src/lib.rs"
click node_io "https://github.com/pola-rs/polars/blob/main/crates/polars-io/src/csv/read/schema_inference.rs"
click node_python_binding "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/_plr.py"
click node_engine_config "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/lazyframe/engine_config.py"
click node_remote_engine "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/lazyframe/engine_remote.py"
click node_gpu_engine "https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/lazyframe/engine.py"
click node_result "https://github.com/pola-rs/polars/blob/main/crates/polars-core/src/query_result.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_python_api,node_python_expr,node_python_types,node_python_helpers,node_sql_context,node_sql_expr,node_sql_functions,node_sql_subqueries,node_sql_pullup,node_sql_visitors,node_sql_dynamic_visit,node_result toneBlue
class node_lazyframe,node_optimizer toneAmber
class node_compute,node_streaming,node_spill toneMint
class node_arrow,node_io toneRose
class node_python_binding,node_engine_config,node_remote_engine,node_gpu_engine,node_user toneIndigo
```

## Components

### User APIs

- [Python DataFrame API](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/dataframe/frame.py): `py-polars/src/polars/dataframe/frame.py`
- [Python expressions](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/expr/datetime.py): `py-polars/src/polars/expr/datetime.py`
- [Python type definitions](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/_typing.py): `py-polars/src/polars/_typing.py`
- [Python utilities](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/_utils/various.py): `py-polars/src/polars/_utils/various.py`
- [SQL context](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/context.rs): `crates/polars-sql/src/context.rs`
- [SQL expression conversion](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/sql_expr.rs): `crates/polars-sql/src/sql_expr.rs`
- [SQL functions](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/functions.rs): `crates/polars-sql/src/functions.rs`
- [SQL subquery rewrites](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/subquery.rs): `crates/polars-sql/src/subquery.rs`
- [Subquery pull-up](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/subquery_pull_up.rs): `crates/polars-sql/src/subquery_pull_up.rs`
- [SQL AST analysis](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/sql_visitors.rs): `crates/polars-sql/src/sql_visitors.rs`
- [Dynamic AST traversal](https://github.com/pola-rs/polars/blob/main/crates/polars-sql/src/dyn_visit.rs): `crates/polars-sql/src/dyn_visit.rs`
- [DataFrame result](https://github.com/pola-rs/polars/blob/main/crates/polars-core/src/query_result.rs): `crates/polars-core/src/query_result.rs`

### Query Planning

- [Lazy query API](https://github.com/pola-rs/polars/blob/main/crates/polars-lazy/src/frame/mod.rs): `crates/polars-lazy/src/frame/mod.rs`
- [Query optimizer](https://github.com/pola-rs/polars/blob/main/crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs): `crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs`

### Execution Runtime

- [Vectorized compute](https://github.com/pola-rs/polars/blob/main/crates/polars-compute/src/lib.rs): `crates/polars-compute/src/lib.rs`
- [Streaming execution](https://github.com/pola-rs/polars/blob/main/crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs): `crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs`
- [Memory and spilling](https://github.com/pola-rs/polars/blob/main/crates/polars-ooc/src/memory_manager.rs): `crates/polars-ooc/src/memory_manager.rs`

### Data and Interchange

- [Arrow columnar data](https://github.com/pola-rs/polars/blob/main/crates/polars-arrow/src/lib.rs): `crates/polars-arrow/src/lib.rs`
- [Data I/O](https://github.com/pola-rs/polars/blob/main/crates/polars-io/src/csv/read/schema_inference.rs): `crates/polars-io/src/csv/read/schema_inference.rs`

### Optional Engines

- [Python Rust binding](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/_plr.py): `py-polars/src/polars/_plr.py`
- [Engine configuration](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/lazyframe/engine_config.py): `py-polars/src/polars/lazyframe/engine_config.py`
- [Remote engine](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/lazyframe/engine_remote.py): `py-polars/src/polars/lazyframe/engine_remote.py`
- [GPU engine](https://github.com/pola-rs/polars/blob/main/py-polars/src/polars/lazyframe/engine.py): `py-polars/src/polars/lazyframe/engine.py`

### Other components

- **Query author**

## Connections

- Query author → Python DataFrame API: builds queries (evidence: `README.md`)
- Query author → SQL context: writes SQL (evidence: `README.md`)
- Python DataFrame API → Python expressions: composes expressions (evidence: `README.md`)
- Python DataFrame API → Lazy query API: collects lazy query (evidence: `README.md`)
- SQL context → SQL expression conversion: converts expressions (evidence: `crates/polars-sql/src/context.rs`)
- SQL context → SQL functions: handles functions (evidence: `crates/polars-sql/src/context.rs`)
- SQL context → SQL subquery rewrites: rewrites subqueries (evidence: `crates/polars-sql/src/context.rs`)
- SQL subquery rewrites → Subquery pull-up: uses pull-up rewrites (evidence: `crates/polars-sql/src/subquery.rs`)
- SQL context → Lazy query API: plans SQL (evidence: `crates/polars-sql/src/context.rs`)
- SQL expression conversion → SQL functions: resolves functions (evidence: `crates/polars-sql/src/sql_expr.rs`)
- SQL context → SQL AST analysis: analyzes SQL AST (evidence: `crates/polars-sql/src/context.rs`)
- SQL AST analysis → Dynamic AST traversal: traverses AST (evidence: `crates/polars-sql/src/sql_visitors.rs`)
- SQL subquery rewrites → Dynamic AST traversal: visits subqueries (evidence: `crates/polars-sql/src/subquery.rs`)
- Lazy query API → Query optimizer: optimizes plan (evidence: `README.md`)
- Query optimizer → Vectorized compute: executes plan (evidence: `README.md`)
- Vectorized compute → DataFrame result: returns result (evidence: `README.md`)
- Data I/O → Arrow columnar data: uses columnar data (evidence: `README.md`)
- Arrow columnar data → Python Rust binding: shares data (evidence: `README.md`)
- Python Rust binding → Python DataFrame API: exposes Rust API (evidence: `py-polars/src/polars/_plr.py`)
- Query optimizer → Streaming execution: executes streaming (evidence: `README.md`)
- Vectorized compute → Memory and spilling: spills when needed (evidence: `README.md`)
- Lazy query API → Remote engine: can execute remotely (evidence: `README.md`)
- Lazy query API → GPU engine: can use GPU (evidence: `README.md`)
- Python DataFrame API → Engine configuration: configures engine (evidence: `py-polars/src/polars/lazyframe/engine_config.py`)
- Engine configuration → Python Rust binding: sets engine options (evidence: `py-polars/src/polars/lazyframe/engine_config.py`)
