# nationalsecurityagency/lemongraph architecture

> Architecture diagram of the GitHub repository nationalsecurityagency/lemongraph, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nationalsecurityagency/lemongraph
- Repository: https://github.com/nationalsecurityagency/lemongraph

## Overview

##### Main Components
a. C Core Library:
– Show the “lib” directory components (e.g., db.c, lemongraph.c, and their corresponding headers) which implement the core graph engine.
– Emphasize that the C layer handles low-level graph data storage, log management, indexing (node_idx, edge_idx, etc.), transactions, and interactions with LMDB.
b. Python Binding/Wrapper:
– Represent the “LemonGraph” Python package that wraps the C core via CFFI (as indicated by files like lemongraph-cffi.c, cffi_stubs.py, and lg_cffi_setup.py).
– Highlight that it adds user-friendly interfaces, Pythonic transaction management (using the with block), dictionary-style operations for graph components, and query language support.
c. Query Language Engine:
d. REST Service:
– Include the REST service module (found in LemonGraph/server) which provides HTTP endpoints for graph operations and manages an indexed cache.
e. LMDB Dependency:
– Represent LMDB (Symas LMDB) as an underlying transactional key/value store that the C core leverages for storage. Note its role in supporting transactions, MVCC, and sub-index tables.
f. Utility/CLI Tools and Scripts:
– Add smaller components such as dbtool.py, snapshot.py, bulk_create and bulk_delete scripts, benchmarks (bench.py), and tests.
g. External Build/Setup Components:
– Components like setup.py, requirements.txt, and Makefile, which are important for deployment and dependencies.

##### Relationships and Data Flow
– Show that the C core interacts directly with LMDB for storage (the underlying persistent file-based store).
– Display that the REST service (which has its own spawning of worker processes and sync processes) interfaces with the Python layer and ultimately the C core for processing graph transactions.
– Represent that both the query language engine and transaction management in Python feed into the C layer for executing graph updates (via log writes and index updates).

##### Architectural Patterns/Design Principles
– Note the separation of concerns between the core database engine (in C) and the higher-level application logic (the Python API, query parser, and REST service).
– Emphasize transaction management central to both layers with “with” block style in Python, and ensuring that LMDB’s requirements (single writer, multiple readers) are respected.
– Outline that the system uses a log-based update mechanism—a key design pattern for managing graph changes, historical views, and snapshots.

##### Note on Technologies/Frameworks
– C and CFFI for interfacing the core engine with Python.
– LMDB as the underpinning key/value store.
– Standard Python ecosystem (with dependencies such as msgpack, ujson, etc.) for data serialization, query processing, and REST API services.
– Process management (master, worker, and sync process) in the REST service component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Layer
    subgraph "Core Layer"
        C_Core["C Core Library"]:::c_core
        LMDB(("LMDB Key/Value Store")):::lmdb
    end

    %% Application Layer
    subgraph "Application Layer"
        Python_Binding["Python Binding/Wrapper"]:::python_binding
        Query_Engine["Query Language Engine"]:::query_engine
    end

    %% Interface Layer
    subgraph "Interface Layer"
        REST_Service["REST Service"]:::rest_service
        CLI_Utilities["CLI Utilities & Scripts"]:::cli
        Build_Setup["Build/Setup Components"]:::build
    end

    %% Relationships
    REST_Service -->|"REST_request/response"| Python_Binding
    CLI_Utilities -->|"CLI_operation"| Python_Binding
    Build_Setup -->|"setup"| Python_Binding
    Python_Binding -->|"transaction_call"| C_Core
    Query_Engine -->|"query_execution"| C_Core
    C_Core -->|"data_storage"| LMDB

    %% Click Events
    click C_Core "https://github.com/nationalsecurityagency/lemongraph/blob/master/lib/lemongraph.c"
    click Python_Binding "https://github.com/nationalsecurityagency/lemongraph/blob/master/LemonGraph/__init__.py"
    click Query_Engine "https://github.com/nationalsecurityagency/lemongraph/blob/master/LemonGraph/query.py"
    click REST_Service "https://github.com/nationalsecurityagency/lemongraph/blob/master/LemonGraph/server/__init__.py"
    click LMDB "https://github.com/nationalsecurityagency/lemongraph/tree/master/deps/lmdb"
    click CLI_Utilities "https://github.com/nationalsecurityagency/lemongraph/blob/master/LemonGraph/dbtool.py"
    click Build_Setup "https://github.com/nationalsecurityagency/lemongraph/blob/master/setup.py"

    %% Styles
    classDef c_core fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef python_binding fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef query_engine fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef rest_service fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef lmdb fill:#87CEFA,stroke:#000,stroke-width:2px;
    classDef cli fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef build fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
