# ladybugdb/ladybug architecture

> Architecture diagram of the GitHub repository ladybugdb/ladybug, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ladybugdb/ladybug
- Repository: https://github.com/ladybugdb/ladybug

## Overview

1. **Overall shape**
   Ladybug is an embedded, in-process graph database. There is no separate database server in this repo; applications link against the core library, or use one of the language bindings, CLI shell, or WebAssembly build. The central runtime is C++ with a CMake-based build, and most subsystems are compiled into the core engine.

2. **External entry points**
   The main public surfaces are the C/C++ API, higher-level language bindings, and the shell. The `src/c_api` and `src/main` layers expose connection/database/query/result abstractions. `tools/shell` provides an interactive Cypher shell, while `tools/python_api`, `tools/nodejs_api`, `tools/rust_api`, `tools/java_api`, and `tools/wasm` are packaging/binding targets around the same engine.

3. **Query processing pipeline**
   Query text enters through the parser, then flows through the binder, planner, and processor. The parser turns Cypher into AST-style parsed statements and expressions. The binder resolves names, types, scopes, and semantics into bound statements. The planner converts bound statements into logical plans. The processor maps logical plans to physical operators and executes them.

4. **Parser and grammar layer**
   Parsing is based on ANTLR-generated Cypher grammar under `src/antlr4` and `src/parser/antlr_parser`. The parser layer also includes transformers and visitors for specific statement kinds such as queries, DDL, transaction commands, copy/import-export, graph statements, and extensions. This layer is responsible for syntactic validity and first-pass statement structure, not storage access.

5. **Binder and semantic analysis**
   The binder is a large semantic layer that resolves graphs, tables, node/rel patterns, properties, macros, functions, and subqueries into bound forms. It also contains query rewrites such as projection normalization, MATCH rewrites, and WITH-clause rewrites. This is where Cypher constructs become engine-specific bound operators and typed expressions.

6. **Planner and optimization**
   The planner creates logical operator trees and applies optimizations before physical execution. The `src/optimizer` subsystem includes pushdown, join ordering, cardinality updates, factorization rewrites, semi-mask optimization, and removal of unnecessary joins/factorization. Planning is particularly important for graph traversals, pattern matching, joins, order-by/limit, and recursive expansion.

7. **Processor and physical operators**
   The processor is the execution engine. It contains vectorized physical operators for scans, filters, joins, projections, aggregates, unwind, order-by, recursive extend, path property probes, DDL/DML, table functions, standalone calls, and transaction control. Execution is chunk-oriented and factorized where possible, with dedicated result structures for flat tuples and factorized tables. This is also where plan mapping from logical to physical occurs.

8. **Storage architecture**
   Storage is columnar and graph-native. Nodes and relationships are represented via node tables, rel tables, chunked node groups, CSR-style adjacency structures, and local/on-disk table variants. Supporting layers include buffer management, page management, compression, overflow storage, statistics, predicates, indexes, and local storage. The storage layer is responsible for persistence, reads/writes, and low-level data layout.

9. **Transactions, durability, and recovery**
   Ladybug supports ACID transactions with write-ahead logging and checkpointing. The transaction subsystem coordinates transaction contexts and managers; the storage WAL subsystem records changes such as inserts, updates, deletes, catalog changes, and extension loads. Checkpointing and WAL replay are the recovery boundary, and many transaction tests in `test/transaction` target rollback, recovery, and concurrent DDL/DML behavior.

10. **Catalog and schema management**
   The catalog tracks graphs, tables, node/rel groups, indexes, sequences, types, functions, and macros. It is the authoritative metadata layer used by binder, planner, and storage. Schema-changing operations in DDL and copy/import/export update the catalog and often generate WAL records so metadata changes are durable and recoverable.

11. **Functions, evaluators, and extensions**
   Built-in functions are organized by domain: arithmetic, comparison, string, list, map, struct, timestamp, UUID, export, table functions, and graph-data-science routines. Expression evaluators execute scalar and lambda logic over vectors during query processing. The extension system allows additional functions, binders, planners, and logical/physical clauses to be installed dynamically, and there are dedicated workflows and tooling around extension packaging and deployment.

12. **Import/export and file format boundary**
   The engine has a substantial ingest/export path for CSV, Parquet, NPY, and Arrow-related integrations. COPY and table-function-based loading flow through parser, binder, planner, and processor, then into storage-specific readers/writers. This boundary is important because it bridges external file formats and the internal columnar graph model.

13. **Tooling, test, and release ecosystem**
   The repo includes extensive end-to-end and subsystem tests, plus benchmark suites and generated answer files. GitHub Actions cover CI, packaging, docs generation, extension builds, Python wheels, WASM, and release artifacts. The `tools` directory contains benchmark, shell, and WAL-dump utilities that exercise the same core engine through different user-facing surfaces.

14. **Dependency and packaging boundary**
   Third-party libraries are vendored under `third_party` and materially shape capabilities: ANTLR for parsing, pybind11 for Python bindings, Arrow/Parquet support, compression libraries, regex and Unicode utilities, logging, and crypto/hash helpers. This repo is therefore both an engine and a distribution workspace: the core C++ library is the center, while bindings, CLI, and platform-specific builds are layered around it.

Main entry points: [`src/main`](https://github.com/ladybugdb/ladybug/tree/main/src/main) (Core API), [`src/planner`](https://github.com/ladybugdb/ladybug/tree/main/src/planner) (Planner), [`src/binder`](https://github.com/ladybugdb/ladybug/tree/main/src/binder) (Binder), [`src/processor`](https://github.com/ladybugdb/ladybug/tree/main/src/processor) (Processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_entry["Entry points"]
  node_n_c_api["C API"]
  node_n_main_api["Core API<br/>engine API"]
  node_n_bindings["Language bindings<br/>packaging"]
  node_n_third_party["Third-party libs<br/>vendored deps"]
end

subgraph group_g_query["Query pipeline"]
  node_n_parser["Parser"]
  node_n_grammar["ANTLR grammar"]
  node_n_binder["Binder<br/>semantic analysis"]
  node_n_planner["Planner"]
  node_n_optimizer["Optimizer<br/>rewrite/estimation"]
  node_n_processor["Processor<br/>execution engine"]
  node_n_copy["COPY I/O<br/>ingest/export path"]
end

subgraph group_g_storage["Storage & tx"]
  node_n_catalog[("Catalog<br/>metadata store")]
  node_n_graph["Graph model<br/>graph metadata"]
  node_n_storage[("Storage engine<br/>columnar storage")]
  node_n_tx["Transactions<br/>ACID manager"]
  node_n_wal["WAL & recovery<br/>durability layer"]
end

subgraph group_g_ext["Functions & extensions"]
  node_n_eval["Evaluators<br/>expression runtime"]
  node_n_functions["Built-ins<br/>function library"]
  node_n_extensions["Extensions<br/>ext framework"]
end

subgraph group_g_tools["Tools"]
  node_n_shell["Shell<br/>CLI frontend"]
  node_n_benchmark["Benchmarks<br/>runner"]
end

node_n_shell -->|"uses"| node_n_main_api
node_n_benchmark -->|"drives"| node_n_main_api
node_n_bindings -->|"wraps"| node_n_main_api
node_n_c_api -->|"adapts"| node_n_main_api
node_n_main_api -->|"submits SQL"| node_n_parser
node_n_parser -->|"uses"| node_n_grammar
node_n_parser -->|"produces AST"| node_n_binder
node_n_binder -->|"produces bound query"| node_n_planner
node_n_planner -->|"optimizes"| node_n_optimizer
node_n_planner -->|"maps to physical"| node_n_processor
node_n_optimizer -->|"feeds rewrites"| node_n_planner
node_n_processor -->|"reads/writes"| node_n_storage
node_n_processor -->|"executes in"| node_n_tx
node_n_tx -->|"logs"| node_n_wal
node_n_wal -->|"replays"| node_n_storage
node_n_catalog -->|"describes"| node_n_storage
node_n_binder -->|"resolves metadata"| node_n_catalog
node_n_planner -->|"consults metadata"| node_n_catalog
node_n_binder -->|"binds expressions"| node_n_eval
node_n_processor -->|"evaluates"| node_n_eval
node_n_functions -->|"implements"| node_n_eval
node_n_extensions -->|"registers"| node_n_catalog
node_n_extensions -->|"extends binding"| node_n_binder
node_n_extensions -->|"extends planning"| node_n_planner
node_n_copy -->|"starts at"| node_n_parser
node_n_copy -->|"executes through"| node_n_processor
node_n_copy -->|"moves data"| node_n_storage
node_n_main_api -->|"manages"| node_n_catalog
node_n_main_api -->|"selects graph"| node_n_graph
node_n_graph -->|"backed by"| node_n_storage
node_n_third_party -->|"supports"| node_n_parser
node_n_third_party -->|"supports"| node_n_bindings

click node_n_c_api "https://github.com/ladybugdb/ladybug/tree/main/src/c_api"
click node_n_main_api "https://github.com/ladybugdb/ladybug/tree/main/src/main"
click node_n_shell "https://github.com/ladybugdb/ladybug/tree/main/tools/shell"
click node_n_benchmark "https://github.com/ladybugdb/ladybug/tree/main/tools/benchmark"
click node_n_parser "https://github.com/ladybugdb/ladybug/tree/main/src/parser"
click node_n_grammar "https://github.com/ladybugdb/ladybug/tree/main/src/antlr4"
click node_n_binder "https://github.com/ladybugdb/ladybug/tree/main/src/binder"
click node_n_planner "https://github.com/ladybugdb/ladybug/tree/main/src/planner"
click node_n_optimizer "https://github.com/ladybugdb/ladybug/tree/main/src/optimizer"
click node_n_processor "https://github.com/ladybugdb/ladybug/tree/main/src/processor"
click node_n_eval "https://github.com/ladybugdb/ladybug/tree/main/src/expression_evaluator"
click node_n_functions "https://github.com/ladybugdb/ladybug/tree/main/src/function"
click node_n_extensions "https://github.com/ladybugdb/ladybug/tree/main/src/extension"
click node_n_catalog "https://github.com/ladybugdb/ladybug/tree/main/src/catalog"
click node_n_graph "https://github.com/ladybugdb/ladybug/tree/main/src/graph"
click node_n_storage "https://github.com/ladybugdb/ladybug/tree/main/src/storage"
click node_n_tx "https://github.com/ladybugdb/ladybug/tree/main/src/transaction"
click node_n_wal "https://github.com/ladybugdb/ladybug/tree/main/src/storage/wal"
click node_n_third_party "https://github.com/ladybugdb/ladybug/tree/main/third_party"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_c_api,node_n_main_api,node_n_bindings,node_n_third_party toneBlue
class node_n_parser,node_n_grammar,node_n_binder,node_n_planner,node_n_optimizer,node_n_processor,node_n_copy toneAmber
class node_n_catalog,node_n_graph,node_n_storage,node_n_tx,node_n_wal toneMint
class node_n_eval,node_n_functions,node_n_extensions toneRose
class node_n_shell,node_n_benchmark toneIndigo
```

## Components

### Entry points

Public surfaces and packaging targets around the embedded engine.

- [C API](https://github.com/ladybugdb/ladybug/tree/main/src/c_api): public API · C and C++-friendly surface for connections, databases, queries, tuples, and results. · `src/c_api`
- [Core API](https://github.com/ladybugdb/ladybug/tree/main/src/main): engine API · Main in-process database, connection, statement, and result abstractions. · `src/main`
- **Language bindings**: packaging · Python, Node.js, Rust, Java, and WASM packaging around the core engine.
- [Third-party libs](https://github.com/ladybugdb/ladybug/tree/main/third_party): vendored deps · ANTLR, pybind11, Arrow/Parquet-related support, compression, regex, hashing, logging, and crypto dependencies. · `third_party`

### Query pipeline

Parse, bind, plan, and execute Cypher queries.

- [Parser](https://github.com/ladybugdb/ladybug/tree/main/src/parser): Cypher parser · Turns Cypher text into parsed statements and expressions. · `src/parser`
- [ANTLR grammar](https://github.com/ladybugdb/ladybug/tree/main/src/antlr4): grammar · Generated Cypher grammar and parser runtime integration. · `src/antlr4`
- [Binder](https://github.com/ladybugdb/ladybug/tree/main/src/binder): semantic analysis · Resolves names, scopes, types, graph patterns, and query rewrites into bound forms. · `src/binder`
- [Planner](https://github.com/ladybugdb/ladybug/tree/main/src/planner): logical planner · Builds logical operator trees and applies planning-time rewrites. · `src/planner`
- [Optimizer](https://github.com/ladybugdb/ladybug/tree/main/src/optimizer): rewrite/estimation · Pushdown, join ordering, cardinality, factorization, and related optimizations. · `src/optimizer`
- [Processor](https://github.com/ladybugdb/ladybug/tree/main/src/processor): execution engine · Maps logical plans to physical operators and executes chunk-oriented queries. · `src/processor`
- **COPY I/O**: ingest/export path · CSV, Parquet, NPY, Arrow-related loading and export boundary into storage.

### Storage & tx

Catalog, storage, transactions, and recovery.

- [Catalog](https://github.com/ladybugdb/ladybug/tree/main/src/catalog): metadata store · Authoritative metadata for graphs, tables, indexes, sequences, types, macros, and functions. · `src/catalog`
- [Graph model](https://github.com/ladybugdb/ladybug/tree/main/src/graph): graph metadata · Graph entries and on-disk graph descriptors used by database attachment and query routing. · `src/graph`
- [Storage engine](https://github.com/ladybugdb/ladybug/tree/main/src/storage): columnar storage · Node/rel tables, chunked node groups, local/on-disk variants, indexes, stats, and buffer/page management. · `src/storage`
- [Transactions](https://github.com/ladybugdb/ladybug/tree/main/src/transaction): ACID manager · Transaction contexts and managers coordinating atomic updates. · `src/transaction`
- [WAL & recovery](https://github.com/ladybugdb/ladybug/tree/main/src/storage/wal): durability layer · Write-ahead logging, checkpointing, record replay, and recovery. · `src/storage/wal`

### Functions & extensions

Built-in functions, evaluators, and extension loading.

- [Evaluators](https://github.com/ladybugdb/ladybug/tree/main/src/expression_evaluator): expression runtime · Scalar, lambda, path, pattern, and reference evaluation during execution. · `src/expression_evaluator`
- [Built-ins](https://github.com/ladybugdb/ladybug/tree/main/src/function): function library · Arithmetic, string, list, map, date, path, GDS, export, table, and utility functions. · `src/function`
- [Extensions](https://github.com/ladybugdb/ladybug/tree/main/src/extension): ext framework · Dynamic extension installation, loading, catalog integration, and custom planner/binder hooks. · `src/extension`

### Tools

CLI, benchmark, and auxiliary utilities.

- [Shell](https://github.com/ladybugdb/ladybug/tree/main/tools/shell): CLI frontend · Interactive Cypher shell on top of the embedded engine. · `tools/shell`
- [Benchmarks](https://github.com/ladybugdb/ladybug/tree/main/tools/benchmark): runner · Benchmark harness and workload execution tools. · `tools/benchmark`

## Connections

- Shell → Core API: uses · Interactive shell invokes the embedded database API.
- Benchmarks → Core API: drives · Benchmarks exercise the same core engine through application APIs.
- Language bindings → Core API: wraps · Language-specific bindings expose the engine to other runtimes.
- C API → Core API: adapts · C API forwards into the core in-process database layer.
- Core API → Parser: submits SQL · Query text enters the parser through the main database surface.
- Parser → ANTLR grammar: uses · Parser is driven by ANTLR-generated Cypher grammar.
- Parser → Binder: produces AST · Parsed statements flow into semantic binding.
- Binder → Planner: produces bound query · Bound statements are handed to logical planning.
- Planner → Optimizer: optimizes · Planner applies join-order and rewrite optimizations before execution.
- Planner → Processor: maps to physical · Logical plans are lowered into executable operators.
- Optimizer → Planner: feeds rewrites · Optimization outputs reshape the logical plan tree.
- Processor → Storage engine: reads/writes · Physical operators access node tables, rel tables, indexes, and storage primitives.
- Processor → Transactions: executes in · Mutating operators run under transaction control.
- Transactions → WAL & recovery: logs · Transactions persist durable changes through WAL records and checkpoints.
- WAL & recovery → Storage engine: replays · Recovery re-applies storage-level changes after failure.
- Catalog → Storage engine: describes · Catalog metadata governs storage layout and object lifecycles.
- Binder → Catalog: resolves metadata · Binder consults catalog entries for graphs, tables, functions, and types.
- Planner → Catalog: consults metadata · Planning uses catalog information for schema-aware decisions.
- Binder → Evaluators: binds expressions · Bound expressions are later evaluated by the runtime expression layer.
- Processor → Evaluators: evaluates · Execution uses evaluators for scalar, lambda, path, and pattern logic.
- Built-ins → Evaluators: implements · Built-in functions are executed through the expression runtime.
- Extensions → Catalog: registers · Extensions add catalog-visible functions, clauses, and metadata.
- Extensions → Binder: extends binding · Extensions can inject binder behavior and custom clauses.
- Extensions → Planner: extends planning · Extensions can contribute logical planner behavior.
- COPY I/O → Parser: starts at · COPY and import/export statements are parsed as part of Cypher processing.
- COPY I/O → Processor: executes through · Load/export paths are realized by physical operators and readers/writers.
- COPY I/O → Storage engine: moves data · Ingest and export bridge external files and internal columnar graph storage.
- Core API → Catalog: manages · Database lifecycle operations create and update metadata.
- Core API → Graph model: selects graph · Connection/database surfaces route work to attached graphs.
- Graph model → Storage engine: backed by · Graph entries reference the underlying table and storage structures.
- Third-party libs → Parser: supports · ANTLR runtime and grammar generation support parsing.
- Third-party libs → Language bindings: supports · pybind11 and packaging dependencies support language bindings.