How walbeh/inserter works
1. What this repo is
- This is a client-side load generator for CrateDB, not a server application.
- It contains three runnable implementations of the same workload: Python async in crate_write/, Rust HTTP in rust/, and Rust PostgreSQL wire in rust-pg/.
- The architectural boundary is simple: generate synthetic records locally, batch them, and send them to an external CrateDB cluster over the network.
2. Common end-to-end data flow
- All variants follow the same basic pipeline: config input → record generation → batch assembly → network insert → optional verification/metrics.
- The generated rows follow one shared schema and are inserted into a user-specified table.
- After the run, the clients can verify success by comparing inserted row count to SELECT COUNT(*) against CrateDB.
3. Python implementation boundary
- crate_write/main.py is the async Python engine.
- It uses asyncio and aiohttp to run many concurrent tasks without threading contention.
- The Python path is the most self-contained: one event loop, many workers, compressed bulk inserts, and optional load-balancer testing before the run begins.
4. Rust HTTP implementation boundary
- rust/src/main.rs is the CLI/orchestrator for the Rust HTTP variant.
- rust/src/client.rs handles HTTP requests, compression, and request execution.
- rust/src/generator.rs produces records, while rust/src/monitor.rs tracks counters and sample statistics.
- rust/src/config.rs loads config from file/env, making this variant more config-driven than the Python one.
5. Rust PostgreSQL wire variant
- rust-pg/src/main.rs is a separate executable that inserts through the PostgreSQL protocol instead of HTTP.
- rust-pg/src/generator.rs provides the same record-generation logic, but the transport and batching are different.
- This variant is architecturally distinct because its data path bypasses the HTTP stack entirely and talks to CrateDB using the PG-compatible endpoint.
6. Shared schema and record shape
- Both implementations use the same table contract defined in create_table.sql and repeated in the README.
- Core fields include id, timestamp, region, product_category, event_type, user_id, user_segment, amount, quantity, and a dynamic metadata object.
- Optional low-cardinality text columns (obj_0, obj_1, etc.) are added when --objects is used.
- This shared schema is important for the architecture diagram because it is the common interface between all clients and CrateDB.
7. Configuration and startup inputs
- Runtime configuration comes from a mix of CLI flags, environment variables, and in Rust HTTP, a TOML config file.
- The main external input is CRATE_CONNECTION_STRING, which points to the CrateDB endpoint.
- Common controls are batch size, batch interval, thread/task count, run duration, table name, and benchmark mode.
- The Python and Rust implementations are intended to be behaviorally aligned so they can be compared fairly.
8. Performance-oriented processing boundary
- The repo is designed around throughput, so CPU-heavy work and I/O-heavy work are treated as separate concerns.
- In Rust HTTP, record generation and serialization/compression are pushed off the async I/O path using blocking worker execution.
- In Python, concurrency is achieved with async tasks and compressed bulk payloads rather than threads.
- This split is one of the main architectural reasons the project can saturate bandwidth rather than spending time on local synchronization.
9. Monitoring and benchmark outputs
- rust/src/monitor.rs and the Python benchmark path collect run statistics such as throughput and percentile samples.
- Benchmark mode emits JSONL-style output for machine parsing and appending to result files.
- The benchmark output includes cluster metadata from CrateDB system tables, which means the client is not just sending writes but also introspecting the target cluster for context.
10. Operational checks and load balancer testing
- The repo includes a special path to test 5-tuple load balancing by creating fresh TCP connections and observing distribution across CrateDB nodes.
- In normal Python runs, this test is performed automatically before the worker loop starts.
- This is an operational concern rather than a functional one, but it matters for understanding throughput and topology effects in the architecture.
11. Tooling and packaging boundaries
- The Python package is defined in pyproject.toml and invoked via uv.
- The Rust HTTP and Rust PG variants are separate Cargo projects with independent binaries and lockfiles.
- The repo therefore represents three delivery artifacts sharing a conceptual workload, not one monolithic app with multiple modes.
Main entry points: main.rs Main · create_table.sql Table schema · main.py Main · generator.rs Generator
Components 16
Shared workload
Common schema and run contract used by all client variants.
- Table schema · Shared target table shape with core columns and optional object columns.
create_table.sql - CrateDB · Destination cluster receiving inserts and providing count/metadata queries.
- Inputs · Run-time settings such as connection string, batch size, duration, and table name.
Python async
Async load generator implementation in crate_write/.
- Main · Python event-loop entrypoint coordinating workers, batching, verification, and preflight checks.
crate_write/main.py - Record gen · Builds synthetic rows matching the shared schema.
crate_write/main.py - Batching · Collects rows into compressed insert payloads for throughput.
crate_write/main.py - HTTP send · Executes concurrent network inserts against CrateDB.
crate_write/main.py - Verify run · Compares inserted rows with SELECT COUNT(*) and supports load-balancer probing.
crate_write/main.py
Rust HTTP
Rust implementation that writes to CrateDB over HTTP.
- Main · Rust HTTP entrypoint that wires config, generation, sending, and monitoring.
rust/src/main.rs - Config · Loads flags, environment, and TOML configuration for the Rust HTTP variant.
rust/src/config.rs - Generator · Produces synthetic records aligned with the shared schema.
rust/src/generator.rs - Client · Handles HTTP requests, compression, and request execution.
rust/src/client.rs - Monitor · Collects throughput and sample statistics during the run.
rust/src/monitor.rs
Rust PG
Rust implementation that writes through the PostgreSQL wire protocol.
Connections 22
- Inputs → Main: configure · CLI/env inputs drive the Python run.
- Inputs → Config: load · Rust HTTP resolves runtime settings from inputs.
- Config → Main: feed · Loaded configuration is passed into orchestration.
- Inputs → Main: configure · CLI/env inputs drive the PG-wire run.
- Table schema → Record gen: shape · Shared schema defines the generated row structure.
- Table schema → Generator: shape · Shared schema defines the generated row structure.
- Table schema → Generator: shape · Shared schema defines the generated row structure.
- Main → Record gen: build · Python orchestrator creates synthetic records.
- Record gen → Batching: rows · Generated rows are assembled into batches.
- Batching → HTTP send: payloads · Batches become compressed bulk requests.
- HTTP send → CrateDB: insert · Python sends inserts to the target cluster.
- Main → Verify run: verify · Run completion triggers verification and LB probing.
- Verify run → CrateDB: query · Verification reads counts and cluster context from CrateDB.
- Main → Generator: produce · Rust HTTP orchestrator creates records.
- Main → Client: execute · Orchestrator hands batches to the transport client.
- Generator → Client: serialize · Generated records are prepared for HTTP transmission.
- Client → CrateDB: insert · HTTP transport writes batches to CrateDB.
- Main → Monitor: track · Run progress and sample stats are recorded.
- Monitor → CrateDB: context · Monitoring may include cluster-aware run context.
- Main → Generator: produce · PG orchestrator creates records.
- Generator → PG insert: batch · Generated records are passed to the PG transport path.
- PG insert → CrateDB: insert · PG wire protocol sends inserts to CrateDB.