How nationalsecurityagency/lemongraph works
Main Components
a. C Core Library:
– Show the “lib” directory components (e.g., db.c, lemongraph.c, and their corresponding headers) which implement the core graph engine.
– Emphasize that the C layer handles low-level graph data storage, log management, indexing (node_idx, edge_idx, etc.), transactions, and interactions with LMDB.
b. Python Binding/Wrapper:
– Represent the “LemonGraph” Python package that wraps the C core via CFFI (as indicated by files like lemongraph-cffi.c, cffi_stubs.py, and lg_cffi_setup.py).
– Highlight that it adds user-friendly interfaces, Pythonic transaction management (using the with block), dictionary-style operations for graph components, and query language support.
c. Query Language Engine:
d. REST Service:
– Include the REST service module (found in LemonGraph/server) which provides HTTP endpoints for graph operations and manages an indexed cache.
e. LMDB Dependency:
– Represent LMDB (Symas LMDB) as an underlying transactional key/value store that the C core leverages for storage. Note its role in supporting transactions, MVCC, and sub-index tables.
f. Utility/CLI Tools and Scripts:
– Add smaller components such as dbtool.py, snapshot.py, bulk_create and bulk_delete scripts, benchmarks (bench.py), and tests.
g. External Build/Setup Components:
– Components like setup.py, requirements.txt, and Makefile, which are important for deployment and dependencies.
Relationships and Data Flow
– Show that the C core interacts directly with LMDB for storage (the underlying persistent file-based store).
– Display that the REST service (which has its own spawning of worker processes and sync processes) interfaces with the Python layer and ultimately the C core for processing graph transactions.
– Represent that both the query language engine and transaction management in Python feed into the C layer for executing graph updates (via log writes and index updates).
Architectural Patterns/Design Principles
– Note the separation of concerns between the core database engine (in C) and the higher-level application logic (the Python API, query parser, and REST service).
– Emphasize transaction management central to both layers with “with” block style in Python, and ensuring that LMDB’s requirements (single writer, multiple readers) are respected.
– Outline that the system uses a log-based update mechanism—a key design pattern for managing graph changes, historical views, and snapshots.
Note on Technologies/Frameworks
– C and CFFI for interfacing the core engine with Python.
– LMDB as the underpinning key/value store.
– Standard Python ecosystem (with dependencies such as msgpack, ujson, etc.) for data serialization, query processing, and REST API services.
– Process management (master, worker, and sync process) in the REST service component.