# depsilon/shardloom architecture

> Architecture diagram of the GitHub repository depsilon/shardloom, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/depsilon/shardloom
- Repository: https://github.com/depsilon/shardloom

## Overview

1. **Repository shape**
   - This is a Rust workspace plus a thin Python package, a static website, benchmark harnesses, contract tests, and a large docs/RFC layer.
   - The architecture is intentionally **evidence-first**: claims are backed by certificates, diagnostics, benchmark artifacts, and contract tests, not just runtime code.

2. **Primary product boundary**
   - ShardLoom is a **Vortex-native, no-fallback execution engine**.
   - The core architectural rule is: unsupported work must fail with deterministic diagnostics rather than delegating to another query engine.
   - Vortex is not “the fallback”; it is the native provider family that ShardLoom can admit through policy and certificates.

3. **User-facing entry points**
   - The main runtime surface is the **CLI** in `shardloom-cli/`.
   - The Python package in `python/` is a **thin client** over the CLI JSON protocol, not a native engine binding.
   - The public website under `website/` is a **claim-safe interpretation layer**, not the full docs or product surface.

4. **Core shared model**
   - `shardloom-core/` holds the cross-cutting domain model: capabilities, schema, expressions, input/output contracts, manifests, execution certificates, observability, security, sessions, and engine-mode abstractions.
   - This crate is the conceptual contract layer used by planning, execution, certification, and release logic.

5. **Planning and translation layer**
   - `shardloom-plan/` turns front-door requests into executable plans.
   - It contains plan IR, optimizer logic, estimation, explainability, input bridging, object-store planning, scan planning, and the execution facade.
   - Architecturally, this is where ShardLoom decides whether a request is admissible, how it should be routed, and what evidence must be emitted.

6. **Execution/runtime layer**
   - `shardloom-exec/` provides the generic execution runtime concerns: memory, spill lifecycle, streaming, sizing, and recovery.
   - This is the bounded execution substrate under the engine-specific layers.
   - The repo treats spill, backpressure, and recovery as first-class runtime boundaries rather than incidental implementation details.

7. **Vortex-specific engine layer**
   - `shardloom-vortex/` is the main specialized engine integration layer.
   - It covers encoded reads, metadata pruning, query primitives, local execution, ingest, commit protocol, output payloads, staged manifests, runtime/scheduler/memory bridges, and Vortex compatibility.
   - This crate is where the “compute less, decode later” strategy is realized: metadata-first planning, encoded-columnar execution, and late materialization where possible.

8. **Typical data flow**
   - The canonical path is: **front door → source route → preparation route → execution route → output route → evidence route**.
   - For non-Vortex inputs, data flows through input adapters and source state into **Vortex ingest / prepared state** before native execution.
   - Outputs are then written as artifacts or emitted inline, with certificates and diagnostics attached so the route can be audited.

9. **CLI as orchestration and evidence emitter**
   - `shardloom-cli/` is not just a command shell; it is the repository’s main orchestration boundary.
   - It organizes command families for planning, diagnostics, benchmark evidence, SQL-local-source smoke paths, prepared/native Vortex paths, object-store planning, packaging/deployment, and release readiness checks.
   - The CLI emits typed envelopes, certificates, fallback status, and route/evidence fields that downstream tooling can consume.

10. **Python package as protocol client**
   - `python/src/shardloom/` exposes client, context, query, and model helpers that mirror the CLI protocol.
   - It is designed to preserve typed result refs, artifact refs, certificates, diagnostics, and fallback status from CLI output.
   - Architecturally, Python is a convenience and integration surface, not an independent execution engine.

11. **Correctness and claim gating**
   - `shardloom-contract-tests/` enforces the architecture’s boundaries: no-fallback invariants, terminology mappings, execution certificate contracts, benchmark evidence rules, and supported/unsupported path behavior.
   - These tests act as a “claim gate” for the codebase: if a route cannot prove itself, it should remain blocked or report-only.

12. **Benchmarks and evidence pipeline**
   - `benchmarks/` and `docs/benchmarks/` define workload scenarios, taxonomy, and result handling.
   - Benchmark artifacts are treated as architectural evidence, and the website publishes a curated slice of that evidence in `website/assets/`.
   - This makes performance and compatibility claims part of the repo’s architecture, not just operational output.

13. **Docs, RFCs, and public interpretation**
   - `docs/rfcs/`, `docs/architecture/`, `docs/use-cases/`, and `docs/skills/` provide the long-form contract and design history.
   - The public website is intentionally narrower than the repository docs: it presents a safe, current snapshot of the system rather than the full internal capability matrix.

14. **Packaging, release, and CI boundary**
   - Cargo workspace metadata, Python packaging, conda recipes, GitHub Actions, and release-validation scripts form the delivery boundary.
   - These tools don’t define product behavior, but they strongly affect the architecture because they gate what can be published, verified, and claimed.
   - In practice, the repo is structured so that runtime behavior, evidence generation, and release readiness are all independently checkable.

15. **How to draw the diagram**
   - Put **CLI** and **Python client** as the main ingress nodes.
   - Route both into **core contracts + planning**.
   - From planning, branch into **generic execution runtime** and the **Vortex-specific engine**.
   - Show **ingest/preparation** feeding **prepared Vortex state**, then **encoded/native execution**, then **output/artifact commit**, with **certificates/diagnostics** flowing alongside every stage.
   - Off to the side, show **contract tests**, **benchmarks**, and **website/docs** as evidence and publication subsystems that validate and explain the runtime, but do not execute user workloads.

Main entry points: [`shardloom-plan/src/lib.rs`](https://github.com/depsilon/shardloom/blob/main/shardloom-plan/src/lib.rs) (Plan IR), [`shardloom-vortex/src/lib.rs`](https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/lib.rs) (Vortex engine), [`shardloom-cli/src/main.rs`](https://github.com/depsilon/shardloom/blob/main/shardloom-cli/src/main.rs) (CLI), [`shardloom-core/src/lib.rs`](https://github.com/depsilon/shardloom/blob/main/shardloom-core/src/lib.rs) (Core model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Ingress"]
  node_cli["CLI<br/>orchestrator<br/>[main.rs]"]
  node_py["Python client<br/>thin client<br/>[client.py]"]
end

subgraph group_g2["Core"]
  node_core["Core model<br/>contracts<br/>[lib.rs]"]
  node_arch["Architecture spine<br/>governance"]
  node_cert["Certificates<br/>evidence"]
end

subgraph group_g3["Planning"]
  node_plan["Plan IR<br/>planner<br/>[lib.rs]"]
  node_inputbridge["Input bridge<br/>adapter<br/>[input_bridge.rs]"]
  node_objplan["Object-store plan<br/>planner<br/>[object_store.rs]"]
end

subgraph group_g4["Runtime"]
  node_genericexec["Exec runtime<br/>runtime substrate<br/>[lib.rs]"]
  node_spill["Spill lifecycle<br/>runtime boundary<br/>[spill_lifecycle.rs]"]
  node_vortex["Vortex engine<br/>engine integration<br/>[lib.rs]"]
  node_meta["Metadata prune<br/>planning/execution"]
  node_ingest["Vortex ingest<br/>[vortex_ingest.rs]"]
  node_encoded["Encoded read<br/>query path"]
  node_commit["Output commit<br/>output/commit<br/>[commit_protocol.rs]"]
end

subgraph group_g5["Evidence"]
  node_contract["Contract tests<br/>claim gate<br/>[lib.rs]"]
  node_bench["Benchmarks<br/>evidence pipeline<br/>[run.py]"]
  node_docs["Docs & RFCs<br/>design history"]
  node_site["Website<br/>public snapshot<br/>[index.html]"]
end

subgraph group_g6["Delivery"]
  node_release["Release gates<br/>delivery"]
end

node_cli -->|"uses contracts"| node_core
node_py -->|"calls protocol"| node_cli
node_cli -->|"builds plans"| node_plan
node_plan -->|"reads model"| node_core
node_plan -->|"bridges inputs"| node_inputbridge
node_plan -->|"routes object store"| node_objplan
node_plan -->|"exec facade"| node_genericexec
node_genericexec -->|"manages spill"| node_spill
node_plan -->|"specializes for Vortex"| node_vortex
node_vortex -->|"prunes metadata"| node_meta
node_vortex -->|"prepares state"| node_ingest
node_vortex -->|"executes encoded"| node_encoded
node_vortex -->|"commits output"| node_commit
node_vortex -->|"emits evidence"| node_cert
node_cli -->|"surfaces evidence"| node_cert
node_contract -->|"guards invariants"| node_core
node_contract -->|"checks envelopes"| node_cli
node_bench -->|"produces artifacts"| node_cert
node_bench -->|"publishes slice"| node_site
node_docs -->|"specifies model"| node_core
node_docs -->|"specifies flow"| node_plan
node_site -->|"curates"| node_docs
node_release -->|"gates claims"| node_contract
node_release -->|"gates evidence"| node_bench
node_release -->|"validates publish"| node_site

click node_cli "https://github.com/depsilon/shardloom/blob/main/shardloom-cli/src/main.rs"
click node_py "https://github.com/depsilon/shardloom/blob/main/python/src/shardloom/client.py"
click node_core "https://github.com/depsilon/shardloom/blob/main/shardloom-core/src/lib.rs"
click node_arch "https://github.com/depsilon/shardloom/blob/main/shardloom-core/src/architecture_spine.rs"
click node_plan "https://github.com/depsilon/shardloom/blob/main/shardloom-plan/src/lib.rs"
click node_inputbridge "https://github.com/depsilon/shardloom/blob/main/shardloom-plan/src/input_bridge.rs"
click node_objplan "https://github.com/depsilon/shardloom/blob/main/shardloom-plan/src/object_store.rs"
click node_genericexec "https://github.com/depsilon/shardloom/blob/main/shardloom-exec/src/lib.rs"
click node_spill "https://github.com/depsilon/shardloom/blob/main/shardloom-exec/src/spill_lifecycle.rs"
click node_vortex "https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/lib.rs"
click node_meta "https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/metadata_pruning.rs"
click node_ingest "https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/vortex_ingest.rs"
click node_encoded "https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/encoded_read_executor.rs"
click node_commit "https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/commit_protocol.rs"
click node_cert "https://github.com/depsilon/shardloom/blob/main/shardloom-core/src/execution_certificate.rs"
click node_contract "https://github.com/depsilon/shardloom/blob/main/shardloom-contract-tests/src/lib.rs"
click node_bench "https://github.com/depsilon/shardloom/blob/main/benchmarks/traditional_analytics/run.py"
click node_docs "https://github.com/depsilon/shardloom/blob/main/docs/architecture/global-architecture-review.md"
click node_site "https://github.com/depsilon/shardloom/blob/main/website/index.html"
click node_release "https://github.com/depsilon/shardloom/blob/main/scripts/check_release_readiness.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_py toneBlue
class node_core,node_arch,node_cert toneAmber
class node_plan,node_inputbridge,node_objplan toneMint
class node_genericexec,node_spill,node_vortex,node_meta,node_ingest,node_encoded,node_commit toneRose
class node_contract,node_bench,node_docs,node_site toneIndigo
class node_release toneTeal
```

## Components

### Ingress

User-facing entry points and protocol surface

- [CLI](https://github.com/depsilon/shardloom/blob/main/shardloom-cli/src/main.rs): orchestrator · Main command boundary; emits typed envelopes, certificates, diagnostics, and route evidence · `shardloom-cli/src/main.rs`
- [Python client](https://github.com/depsilon/shardloom/blob/main/python/src/shardloom/client.py): thin client · Protocol client over the CLI JSON surface; preserves typed refs, artifacts, and certificates · `python/src/shardloom/client.py`

### Core

Shared contracts and architecture model

- [Core model](https://github.com/depsilon/shardloom/blob/main/shardloom-core/src/lib.rs): contracts · Cross-cutting domain model for capabilities, schema, manifests, certificates, observability, security, and sessions · `shardloom-core/src/lib.rs`
- [Architecture spine](https://github.com/depsilon/shardloom/blob/main/shardloom-core/src/architecture_spine.rs): governance · Terminology, claim gating, and architecture-wide policy boundary · `shardloom-core/src/architecture_spine.rs`
- [Certificates](https://github.com/depsilon/shardloom/blob/main/shardloom-core/src/execution_certificate.rs): evidence · Execution certificates, diagnostics, and claim fields attached to runtime outcomes · `shardloom-core/src/execution_certificate.rs`

### Planning

Admissibility, routing, and plan construction

- [Plan IR](https://github.com/depsilon/shardloom/blob/main/shardloom-plan/src/lib.rs): planner · Front-door request shaping, admissibility, routing, optimizer, explainability, and execution facade · `shardloom-plan/src/lib.rs`
- [Input bridge](https://github.com/depsilon/shardloom/blob/main/shardloom-plan/src/input_bridge.rs): adapter · Bridges source state and external inputs into planning · `shardloom-plan/src/input_bridge.rs`
- [Object-store plan](https://github.com/depsilon/shardloom/blob/main/shardloom-plan/src/object_store.rs): planner · Object-store routing and scan planning for distributed or remote inputs · `shardloom-plan/src/object_store.rs`

### Runtime

Generic execution substrate and Vortex-specific engine

- [Exec runtime](https://github.com/depsilon/shardloom/blob/main/shardloom-exec/src/lib.rs): runtime substrate · Memory, spill, streaming, sizing, and recovery substrate under engine-specific layers · `shardloom-exec/src/lib.rs`
- [Spill lifecycle](https://github.com/depsilon/shardloom/blob/main/shardloom-exec/src/spill_lifecycle.rs): runtime boundary · First-class spill reservation, payload, and recovery lifecycle · `shardloom-exec/src/spill_lifecycle.rs`
- [Vortex engine](https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/lib.rs): engine integration · Specialized encoded-read, ingest, commit, metadata-pruning, and runtime bridge layer · `shardloom-vortex/src/lib.rs`
- [Metadata prune](https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/metadata_pruning.rs): planning/execution · Metadata-first pruning, summary, and read planning before decoding · `shardloom-vortex/src/metadata_pruning.rs`
- [Vortex ingest](https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/vortex_ingest.rs): ingest · Preparation and ingest path that turns source-backed inputs into prepared Vortex state · `shardloom-vortex/src/vortex_ingest.rs`
- [Encoded read](https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/encoded_read_executor.rs): query path · Encoded-columnar execution with late materialization and primitive kernels · `shardloom-vortex/src/encoded_read_executor.rs`
- [Output commit](https://github.com/depsilon/shardloom/blob/main/shardloom-vortex/src/commit_protocol.rs): output/commit · Finalization, staged manifests, output payloads, and commit protocol · `shardloom-vortex/src/commit_protocol.rs`

### Evidence

Tests, benchmarks, docs, and publication surfaces

- [Contract tests](https://github.com/depsilon/shardloom/blob/main/shardloom-contract-tests/src/lib.rs): claim gate · No-fallback invariants, correctness contracts, diagnostics stability, and supported-path behavior · `shardloom-contract-tests/src/lib.rs`
- [Benchmarks](https://github.com/depsilon/shardloom/blob/main/benchmarks/traditional_analytics/run.py): evidence pipeline · Workload scenarios, taxonomy, and benchmark artifact handling · `benchmarks/traditional_analytics/run.py`
- [Docs & RFCs](https://github.com/depsilon/shardloom/blob/main/docs/architecture/global-architecture-review.md): design history · Long-form architecture, RFCs, use cases, and skills guidance · `docs/architecture/global-architecture-review.md`
- [Website](https://github.com/depsilon/shardloom/blob/main/website/index.html): public snapshot · Claim-safe public interpretation layer and curated evidence publication surface · `website/index.html`

### Delivery

Packaging, release, and CI gates

- [Release gates](https://github.com/depsilon/shardloom/blob/main/scripts/check_release_readiness.py): delivery · Packaging, CI, release validation, and publication checks · `scripts/check_release_readiness.py`

## Connections

- CLI → Core model: uses contracts · CLI validates and serializes outcomes against shared domain contracts
- Python client → CLI: calls protocol · Python delegates work through the CLI JSON protocol
- CLI → Plan IR: builds plans · CLI front door routes requests into plan construction and routing
- Plan IR → Core model: reads model · Planning depends on shared capabilities, schema, manifests, and certificates
- Plan IR → Input bridge: bridges inputs · Planning admits source state and external inputs through adapters
- Plan IR → Object-store plan: routes object store · Planning creates object-store scans and remote-input routes
- Plan IR → Exec runtime: exec facade · Plan facade hands executable work into the runtime substrate
- Exec runtime → Spill lifecycle: manages spill · Runtime treats spill as a core lifecycle boundary
- Plan IR → Vortex engine: specializes for Vortex · Admitted requests are lowered into the Vortex engine path when compatible
- Vortex engine → Metadata prune: prunes metadata · Vortex chooses metadata-first execution and deferred decoding
- Vortex engine → Vortex ingest: prepares state · Source-backed inputs are ingested into prepared Vortex state
- Vortex engine → Encoded read: executes encoded · Native execution uses encoded reads and query primitives
- Vortex engine → Output commit: commits output · Execution finalizes through staged manifests and output payload commit
- Vortex engine → Certificates: emits evidence · Runtime attaches certificates and diagnostics to outputs
- CLI → Certificates: surfaces evidence · CLI exposes typed envelopes, fallback status, diagnostics, and certificates
- Contract tests → Core model: guards invariants · Contract tests enforce architecture claims and supported/unsupported behavior
- Contract tests → CLI: checks envelopes · Contract tests snapshot CLI protocol and claim-gate behavior
- Benchmarks → Certificates: produces artifacts · Benchmarks generate evidence artifacts and result records
- Benchmarks → Website: publishes slice · Curated benchmark evidence is published to the public site
- Docs & RFCs → Core model: specifies model · Docs and RFCs define the long-form contract and terminology
- Docs & RFCs → Plan IR: specifies flow · Architecture docs describe admissibility, translation, and execution flow
- Website → Docs & RFCs: curates · Website presents a safe, narrower interpretation of the repository docs
- Release gates → Contract tests: gates claims · Release validation depends on contract and readiness checks
- Release gates → Benchmarks: gates evidence · Packaging and release gates consume benchmark evidence and artifacts
- Release gates → Website: validates publish · Website publication is checked as part of release readiness
