ladybugdb/ladybug

1. Overall shape

Ladybug is an embedded, in-process graph database. There is no separate database server in this repo; applications link against the core library, or use one of the language bindings, CLI shell, or WebAssembly build. The central runtime is C++ with a CMake-based build, and most subsystems are compiled into the core engine.

2. External entry points

The main public surfaces are the C/C++ API, higher-level language bindings, and the shell. The src/c_api and src/main layers expose connection/database/query/result abstractions. tools/shell provides an interactive Cypher shell, while tools/python_api, tools/nodejs_api, tools/rust_api, tools/java_api, and tools/wasm are packaging/binding targets around the same engine.

3. Query processing pipeline

Query text enters through the parser, then flows through the binder, planner, and processor. The parser turns Cypher into AST-style parsed statements and expressions. The binder resolves names, types, scopes, and semantics into bound statements. The planner converts bound statements into logical plans. The processor maps logical plans to physical operators and executes them.

4. Parser and grammar layer

Parsing is based on ANTLR-generated Cypher grammar under src/antlr4 and src/parser/antlr_parser. The parser layer also includes transformers and visitors for specific statement kinds such as queries, DDL, transaction commands, copy/import-export, graph statements, and extensions. This layer is responsible for syntactic validity and first-pass statement structure, not storage access.

5. Binder and semantic analysis

The binder is a large semantic layer that resolves graphs, tables, node/rel patterns, properties, macros, functions, and subqueries into bound forms. It also contains query rewrites such as projection normalization, MATCH rewrites, and WITH-clause rewrites. This is where Cypher constructs become engine-specific bound operators and typed expressions.

6. Planner and optimization

The planner creates logical operator trees and applies optimizations before physical execution. The src/optimizer subsystem includes pushdown, join ordering, cardinality updates, factorization rewrites, semi-mask optimization, and removal of unnecessary joins/factorization. Planning is particularly important for graph traversals, pattern matching, joins, order-by/limit, and recursive expansion.

7. Processor and physical operators

The processor is the execution engine. It contains vectorized physical operators for scans, filters, joins, projections, aggregates, unwind, order-by, recursive extend, path property probes, DDL/DML, table functions, standalone calls, and transaction control. Execution is chunk-oriented and factorized where possible, with dedicated result structures for flat tuples and factorized tables. This is also where plan mapping from logical to physical occurs.

8. Storage architecture

Storage is columnar and graph-native. Nodes and relationships are represented via node tables, rel tables, chunked node groups, CSR-style adjacency structures, and local/on-disk table variants. Supporting layers include buffer management, page management, compression, overflow storage, statistics, predicates, indexes, and local storage. The storage layer is responsible for persistence, reads/writes, and low-level data layout.

9. Transactions, durability, and recovery

Ladybug supports ACID transactions with write-ahead logging and checkpointing. The transaction subsystem coordinates transaction contexts and managers; the storage WAL subsystem records changes such as inserts, updates, deletes, catalog changes, and extension loads. Checkpointing and WAL replay are the recovery boundary, and many transaction tests in test/transaction target rollback, recovery, and concurrent DDL/DML behavior.

10. Catalog and schema management

The catalog tracks graphs, tables, node/rel groups, indexes, sequences, types, functions, and macros. It is the authoritative metadata layer used by binder, planner, and storage. Schema-changing operations in DDL and copy/import/export update the catalog and often generate WAL records so metadata changes are durable and recoverable.

11. Functions, evaluators, and extensions

Built-in functions are organized by domain: arithmetic, comparison, string, list, map, struct, timestamp, UUID, export, table functions, and graph-data-science routines. Expression evaluators execute scalar and lambda logic over vectors during query processing. The extension system allows additional functions, binders, planners, and logical/physical clauses to be installed dynamically, and there are dedicated workflows and tooling around extension packaging and deployment.

12. Import/export and file format boundary

The engine has a substantial ingest/export path for CSV, Parquet, NPY, and Arrow-related integrations. COPY and table-function-based loading flow through parser, binder, planner, and processor, then into storage-specific readers/writers. This boundary is important because it bridges external file formats and the internal columnar graph model.

13. Tooling, test, and release ecosystem

The repo includes extensive end-to-end and subsystem tests, plus benchmark suites and generated answer files. GitHub Actions cover CI, packaging, docs generation, extension builds, Python wheels, WASM, and release artifacts. The tools directory contains benchmark, shell, and WAL-dump utilities that exercise the same core engine through different user-facing surfaces.

14. Dependency and packaging boundary

Third-party libraries are vendored under third_party and materially shape capabilities: ANTLR for parsing, pybind11 for Python bindings, Arrow/Parquet support, compression libraries, regex and Unicode utilities, logging, and crypto/hash helpers. This repo is therefore both an engine and a distribution workspace: the core C++ library is the center, while bindings, CLI, and platform-specific builds are layered around it.