# walbeh/inserter architecture

> Architecture diagram of the GitHub repository walbeh/inserter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/walbeh/inserter
- Repository: https://github.com/walbeh/inserter

## Overview

1. **What this repo is**
   - This is a client-side load generator for **CrateDB**, not a server application.
   - It contains three runnable implementations of the same workload: **Python async** in `crate_write/`, **Rust HTTP** in `rust/`, and **Rust PostgreSQL wire** in `rust-pg/`.
   - The architectural boundary is simple: generate synthetic records locally, batch them, and send them to an external CrateDB cluster over the network.

2. **Common end-to-end data flow**
   - All variants follow the same basic pipeline: **config input → record generation → batch assembly → network insert → optional verification/metrics**.
   - The generated rows follow one shared schema and are inserted into a user-specified table.
   - After the run, the clients can verify success by comparing inserted row count to `SELECT COUNT(*)` against CrateDB.

3. **Python implementation boundary**
   - `crate_write/main.py` is the async Python engine.
   - It uses **asyncio** and **aiohttp** to run many concurrent tasks without threading contention.
   - The Python path is the most self-contained: one event loop, many workers, compressed bulk inserts, and optional load-balancer testing before the run begins.

4. **Rust HTTP implementation boundary**
   - `rust/src/main.rs` is the CLI/orchestrator for the Rust HTTP variant.
   - `rust/src/client.rs` handles HTTP requests, compression, and request execution.
   - `rust/src/generator.rs` produces records, while `rust/src/monitor.rs` tracks counters and sample statistics.
   - `rust/src/config.rs` loads config from file/env, making this variant more config-driven than the Python one.

5. **Rust PostgreSQL wire variant**
   - `rust-pg/src/main.rs` is a separate executable that inserts through the **PostgreSQL protocol** instead of HTTP.
   - `rust-pg/src/generator.rs` provides the same record-generation logic, but the transport and batching are different.
   - This variant is architecturally distinct because its data path bypasses the HTTP stack entirely and talks to CrateDB using the PG-compatible endpoint.

6. **Shared schema and record shape**
   - Both implementations use the same table contract defined in `create_table.sql` and repeated in the README.
   - Core fields include `id`, `timestamp`, `region`, `product_category`, `event_type`, `user_id`, `user_segment`, `amount`, `quantity`, and a dynamic `metadata` object.
   - Optional low-cardinality text columns (`obj_0`, `obj_1`, etc.) are added when `--objects` is used.
   - This shared schema is important for the architecture diagram because it is the common interface between all clients and CrateDB.

7. **Configuration and startup inputs**
   - Runtime configuration comes from a mix of **CLI flags**, **environment variables**, and in Rust HTTP, a **TOML config file**.
   - The main external input is `CRATE_CONNECTION_STRING`, which points to the CrateDB endpoint.
   - Common controls are batch size, batch interval, thread/task count, run duration, table name, and benchmark mode.
   - The Python and Rust implementations are intended to be behaviorally aligned so they can be compared fairly.

8. **Performance-oriented processing boundary**
   - The repo is designed around throughput, so CPU-heavy work and I/O-heavy work are treated as separate concerns.
   - In Rust HTTP, record generation and serialization/compression are pushed off the async I/O path using blocking worker execution.
   - In Python, concurrency is achieved with async tasks and compressed bulk payloads rather than threads.
   - This split is one of the main architectural reasons the project can saturate bandwidth rather than spending time on local synchronization.

9. **Monitoring and benchmark outputs**
   - `rust/src/monitor.rs` and the Python benchmark path collect run statistics such as throughput and percentile samples.
   - Benchmark mode emits **JSONL**-style output for machine parsing and appending to result files.
   - The benchmark output includes cluster metadata from CrateDB system tables, which means the client is not just sending writes but also introspecting the target cluster for context.

10. **Operational checks and load balancer testing**
   - The repo includes a special path to test **5-tuple load balancing** by creating fresh TCP connections and observing distribution across CrateDB nodes.
   - In normal Python runs, this test is performed automatically before the worker loop starts.
   - This is an operational concern rather than a functional one, but it matters for understanding throughput and topology effects in the architecture.

11. **Tooling and packaging boundaries**
   - The Python package is defined in `pyproject.toml` and invoked via `uv`.
   - The Rust HTTP and Rust PG variants are separate Cargo projects with independent binaries and lockfiles.
   - The repo therefore represents three delivery artifacts sharing a conceptual workload, not one monolithic app with multiple modes.

Main entry points: [`rust/src/main.rs`](https://github.com/walbeh/inserter/blob/main/rust/src/main.rs) (Main), [`create_table.sql`](https://github.com/walbeh/inserter/blob/main/create_table.sql) (Table schema), [`crate_write/main.py`](https://github.com/walbeh/inserter/blob/main/crate_write/main.py) (Main), [`rust/src/generator.rs`](https://github.com/walbeh/inserter/blob/main/rust/src/generator.rs) (Generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_shared["Shared workload"]
  node_table_schema["Table schema<br/>sql contract<br/>[create_table.sql]"]
  node_crate_cluster[("CrateDB<br/>external cluster")]
  node_config_inputs["Inputs<br/>cli/env/config"]
end

subgraph group_python["Python async"]
  node_py_main["Main<br/>async orchestrator<br/>[main.py]"]
  node_py_generate["Record gen<br/>async generator<br/>[main.py]"]
  node_py_batch["Batching<br/>bulk assembler<br/>[main.py]"]
  node_py_send["HTTP send<br/>aiohttp writer<br/>[main.py]"]
  node_py_check["Verify run<br/>count check<br/>[main.py]"]
end

subgraph group_rust_http["Rust HTTP"]
  node_rh_main["Main<br/>CLI orchestrator<br/>[main.rs]"]
  node_rh_config["Config<br/>settings loader<br/>[config.rs]"]
  node_rh_gen["Generator<br/>[generator.rs]"]
  node_rh_client["Client<br/>HTTP transport<br/>[client.rs]"]
  node_rh_monitor(("Monitor<br/>metrics tracker<br/>[monitor.rs]"))
end

subgraph group_rust_pg["Rust PG"]
  node_rpg_main["Main<br/>PG orchestrator<br/>[main.rs]"]
  node_rpg_gen["Generator<br/>[generator.rs]"]
  node_rpg_send["PG insert<br/>pg wire client<br/>[main.rs]"]
end

node_config_inputs -->|"configure"| node_py_main
node_config_inputs -->|"load"| node_rh_config
node_rh_config -->|"feed"| node_rh_main
node_config_inputs -->|"configure"| node_rpg_main
node_table_schema -->|"shape"| node_py_generate
node_table_schema -->|"shape"| node_rh_gen
node_table_schema -->|"shape"| node_rpg_gen
node_py_main -->|"build"| node_py_generate
node_py_generate -->|"rows"| node_py_batch
node_py_batch -->|"payloads"| node_py_send
node_py_send -->|"insert"| node_crate_cluster
node_py_main -->|"verify"| node_py_check
node_py_check -->|"query"| node_crate_cluster
node_rh_main -->|"produce"| node_rh_gen
node_rh_main -->|"execute"| node_rh_client
node_rh_gen -->|"serialize"| node_rh_client
node_rh_client -->|"insert"| node_crate_cluster
node_rh_main -->|"track"| node_rh_monitor
node_rh_monitor -->|"context"| node_crate_cluster
node_rpg_main -->|"produce"| node_rpg_gen
node_rpg_gen -->|"batch"| node_rpg_send
node_rpg_send -->|"insert"| node_crate_cluster

click node_table_schema "https://github.com/walbeh/inserter/blob/main/create_table.sql"
click node_py_main "https://github.com/walbeh/inserter/blob/main/crate_write/main.py"
click node_py_generate "https://github.com/walbeh/inserter/blob/main/crate_write/main.py"
click node_py_batch "https://github.com/walbeh/inserter/blob/main/crate_write/main.py"
click node_py_send "https://github.com/walbeh/inserter/blob/main/crate_write/main.py"
click node_py_check "https://github.com/walbeh/inserter/blob/main/crate_write/main.py"
click node_rh_main "https://github.com/walbeh/inserter/blob/main/rust/src/main.rs"
click node_rh_config "https://github.com/walbeh/inserter/blob/main/rust/src/config.rs"
click node_rh_gen "https://github.com/walbeh/inserter/blob/main/rust/src/generator.rs"
click node_rh_client "https://github.com/walbeh/inserter/blob/main/rust/src/client.rs"
click node_rh_monitor "https://github.com/walbeh/inserter/blob/main/rust/src/monitor.rs"
click node_rpg_main "https://github.com/walbeh/inserter/blob/main/rust-pg/src/main.rs"
click node_rpg_gen "https://github.com/walbeh/inserter/blob/main/rust-pg/src/generator.rs"
click node_rpg_send "https://github.com/walbeh/inserter/blob/main/rust-pg/src/main.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_table_schema,node_crate_cluster,node_config_inputs toneBlue
class node_py_main,node_py_generate,node_py_batch,node_py_send,node_py_check toneAmber
class node_rh_main,node_rh_config,node_rh_gen,node_rh_client,node_rh_monitor toneMint
class node_rpg_main,node_rpg_gen,node_rpg_send toneRose
```

## Components

### Shared workload

Common schema and run contract used by all client variants.

- [Table schema](https://github.com/walbeh/inserter/blob/main/create_table.sql): sql contract · Shared target table shape with core columns and optional object columns. · `create_table.sql`
- **CrateDB**: external cluster · Destination cluster receiving inserts and providing count/metadata queries.
- **Inputs**: cli/env/config · Run-time settings such as connection string, batch size, duration, and table name.

### Python async

Async load generator implementation in crate_write/.

- [Main](https://github.com/walbeh/inserter/blob/main/crate_write/main.py): async orchestrator · Python event-loop entrypoint coordinating workers, batching, verification, and preflight checks. · `crate_write/main.py`
- [Record gen](https://github.com/walbeh/inserter/blob/main/crate_write/main.py): async generator · Builds synthetic rows matching the shared schema. · `crate_write/main.py`
- [Batching](https://github.com/walbeh/inserter/blob/main/crate_write/main.py): bulk assembler · Collects rows into compressed insert payloads for throughput. · `crate_write/main.py`
- [HTTP send](https://github.com/walbeh/inserter/blob/main/crate_write/main.py): aiohttp writer · Executes concurrent network inserts against CrateDB. · `crate_write/main.py`
- [Verify run](https://github.com/walbeh/inserter/blob/main/crate_write/main.py): count check · Compares inserted rows with SELECT COUNT(*) and supports load-balancer probing. · `crate_write/main.py`

### Rust HTTP

Rust implementation that writes to CrateDB over HTTP.

- [Main](https://github.com/walbeh/inserter/blob/main/rust/src/main.rs): CLI orchestrator · Rust HTTP entrypoint that wires config, generation, sending, and monitoring. · `rust/src/main.rs`
- [Config](https://github.com/walbeh/inserter/blob/main/rust/src/config.rs): settings loader · Loads flags, environment, and TOML configuration for the Rust HTTP variant. · `rust/src/config.rs`
- [Generator](https://github.com/walbeh/inserter/blob/main/rust/src/generator.rs): record generator · Produces synthetic records aligned with the shared schema. · `rust/src/generator.rs`
- [Client](https://github.com/walbeh/inserter/blob/main/rust/src/client.rs): HTTP transport · Handles HTTP requests, compression, and request execution. · `rust/src/client.rs`
- [Monitor](https://github.com/walbeh/inserter/blob/main/rust/src/monitor.rs): metrics tracker · Collects throughput and sample statistics during the run. · `rust/src/monitor.rs`

### Rust PG

Rust implementation that writes through the PostgreSQL wire protocol.

- [Main](https://github.com/walbeh/inserter/blob/main/rust-pg/src/main.rs): PG orchestrator · Rust PostgreSQL-wire entrypoint for the non-HTTP insert path. · `rust-pg/src/main.rs`
- [Generator](https://github.com/walbeh/inserter/blob/main/rust-pg/src/generator.rs): record generator · Produces the same row shape for the PG transport path. · `rust-pg/src/generator.rs`
- [PG insert](https://github.com/walbeh/inserter/blob/main/rust-pg/src/main.rs): pg wire client · Sends batched inserts through CrateDB's PostgreSQL-compatible endpoint. · `rust-pg/src/main.rs`

## Connections

- Inputs → Main: configure · CLI/env inputs drive the Python run.
- Inputs → Config: load · Rust HTTP resolves runtime settings from inputs.
- Config → Main: feed · Loaded configuration is passed into orchestration.
- Inputs → Main: configure · CLI/env inputs drive the PG-wire run.
- Table schema → Record gen: shape · Shared schema defines the generated row structure.
- Table schema → Generator: shape · Shared schema defines the generated row structure.
- Table schema → Generator: shape · Shared schema defines the generated row structure.
- Main → Record gen: build · Python orchestrator creates synthetic records.
- Record gen → Batching: rows · Generated rows are assembled into batches.
- Batching → HTTP send: payloads · Batches become compressed bulk requests.
- HTTP send → CrateDB: insert · Python sends inserts to the target cluster.
- Main → Verify run: verify · Run completion triggers verification and LB probing.
- Verify run → CrateDB: query · Verification reads counts and cluster context from CrateDB.
- Main → Generator: produce · Rust HTTP orchestrator creates records.
- Main → Client: execute · Orchestrator hands batches to the transport client.
- Generator → Client: serialize · Generated records are prepared for HTTP transmission.
- Client → CrateDB: insert · HTTP transport writes batches to CrateDB.
- Main → Monitor: track · Run progress and sample stats are recorded.
- Monitor → CrateDB: context · Monitoring may include cluster-aware run context.
- Main → Generator: produce · PG orchestrator creates records.
- Generator → PG insert: batch · Generated records are passed to the PG transport path.
- PG insert → CrateDB: insert · PG wire protocol sends inserts to CrateDB.
