# svilendobrev/py-xtdb-datomic architecture

> Architecture diagram of the GitHub repository svilendobrev/py-xtdb-datomic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/svilendobrev/py-xtdb-datomic
- Repository: https://github.com/svilendobrev/py-xtdb-datomic

## Overview

##### Main Components

a) Client Layer:
- Separate client implementations for Datomic and XTDB (v1 and v2)
- REST API wrapper components
- Query builders for different query languages (Datalog and XTQL)

b) Base Components:
- Object mapping system
- Schema handling
- Query syntax utilities
- EDN (Extensible Data Notation) handling

c) Storage Layer:
- Multiple backend configurations
- Kafka (with both Kraft and Zookeeper options)
- PostgreSQL
- Hybrid setups (Kafka + PostgreSQL)

**[Client Applications]**
- ↓
**[Python Interface Layer]**
- ↓
**[Database Systems]**

##### Specific Implementation Details

Top Level (Client Applications):
- External applications connecting through HTTP REST API
- JSON/EDN/Transit-JSON data formats

Middle Level (Python Interface Layer):
- Create three parallel tracks for different implementations:
- Datomic track (datomic/)
- XTDB v1 track (xtdb/)
- XTDB v2 track (xtdb2/)
- Shared base components (base/)
- Query builders and syntax handlers

Bottom Level (Database Systems):
- Datomic and XTDB nodes
- Different storage configurations
- Checkpoint/snapshot system

##### Important Relationships

- Data flow directions between components
- API boundaries
- Storage system interactions
- Query transformation pipeline
- Event sourcing flow

##### Additional Elements

Annotations for:
- Version compatibility
- Implementation status (completed vs. TODO)
- External dependencies
- Configuration points

## Diagram (Mermaid)

```mermaid
graph TB
    %% Client Applications Layer
    subgraph CL["Client Applications"]
        CA["External Applications"]
        REST["REST API Interface"]:::api
    end

    %% Python Interface Layer
    subgraph PIL["Python Interface Layer"]
        subgraph Base["Base Components"]
            OM["Object Mapper"]:::base
            SH["Schema Handler"]:::base
            QSU["Query Syntax Utilities"]:::base
            EDN["EDN Handler"]:::base
            RPC["REST API Wrapper"]:::base
        end

        subgraph DC["Datomic Client"]
            DCI["Client Implementation"]:::datomic
            DSH["Schema Handler"]:::datomic
            DQS["Query Syntax"]:::datomic
        end

        subgraph XC1["XTDB v1 Client"]
            XCI1["Client Implementation"]:::xtdb
            XQS1["Query Syntax"]:::xtdb
            LUC["Lucene Integration"]:::xtdb
        end

        subgraph XC2["XTDB v2 Client"]
            XCI2["Client Implementation"]:::xtdb2
            XQS2["Query Syntax"]:::xtdb2
            XQSE["Extended Query Syntax"]:::xtdb2
        end
    end

    %% Storage Layer
    subgraph SL["Storage Layer"]
        subgraph KS["Kafka Storage"]
            KK["Kafka Kraft"]:::storage
            KZ["Kafka Zookeeper"]:::storage
        end
        
        PG["PostgreSQL"]:::storage
        
        subgraph HS["Hybrid Setup"]
            KP["Kafka + PostgreSQL"]:::storage
        end

        CPS["Checkpoint System"]:::utility
    end

    %% Relationships
    CA --> REST
    REST --> RPC
    RPC --> DCI & XCI1 & XCI2
    
    OM --> DCI & XCI1 & XCI2
    SH --> DSH
    QSU --> DQS & XQS1 & XQS2
    EDN --> DCI & XCI1 & XCI2
    
    DCI --> KS & PG
    XCI1 --> KS & PG & HS
    XCI2 --> KS & PG & HS
    
    KS & PG & HS --> CPS

    %% Click Events
    click OM "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/base/objmapper.py"
    click SH "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/base/schema.py"
    click QSU "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/base/qsyntax.py"
    click EDN "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/base/edn.py"
    click RPC "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/base/rpc_json_http.py"
    click DCI "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/datomic/dbclient.py"
    click DSH "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/datomic/schema.py"
    click DQS "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/datomic/qsyntax.py"
    click XCI1 "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb/dbclient.py"
    click XQS1 "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb/qsyntax.py"
    click LUC "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb/lucene.py"
    click XCI2 "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb2/dbclient.py"
    click XQS2 "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb2/qs.py"
    click XQSE "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb2/qs_extra.py"
    click KK "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb/srv-dockermulti/1-txdoc-n-xtdb/docker-composes/kafka_kraft.yml"
    click KZ "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb/srv-dockermulti/1-txdoc-n-xtdb/docker-composes/kafka_zoo.yml"
    click PG "https://github.com/svilendobrev/py-xtdb-datomic/blob/main/xtdb/srv-dockermulti/1-txdoc-n-xtdb/docker-composes/postgresql.yml"
    click KP "https://github.com/svilendobrev/py-xtdb-datomic/tree/main/xtdb/srv-dockermulti/1-txdoc-n-xtdb/tx-kafka-kraft+docs-pg/"
    click CPS "https://github.com/svilendobrev/py-xtdb-datomic/tree/main/xtdb/srv-dockermulti/1-txdoc-n-xtdb/checkpointkeeper/"

    %% Styles
    classDef api fill:#87CEEB
    classDef base fill:#A9A9A9
    classDef datomic fill:#98FB98
    classDef xtdb fill:#FFA500
    classDef xtdb2 fill:#FFD700
    classDef storage fill:#DDA0DD
    classDef utility fill:#B8860B

    %% Legend
    subgraph Legend
        L1["API Components"]:::api
        L2["Base Components"]:::base
        L3["Datomic Components"]:::datomic
        L4["XTDB v1 Components"]:::xtdb
        L5["XTDB v2 Components"]:::xtdb2
        L6["Storage Systems"]:::storage
        L7["Utility Systems"]:::utility
    end
```
