How depsilon/shardloom works
1. Repository shape
- This is a Rust workspace plus a thin Python package, a static website, benchmark harnesses, contract tests, and a large docs/RFC layer.
- The architecture is intentionally evidence-first: claims are backed by certificates, diagnostics, benchmark artifacts, and contract tests, not just runtime code.
2. Primary product boundary
- ShardLoom is a Vortex-native, no-fallback execution engine.
- The core architectural rule is: unsupported work must fail with deterministic diagnostics rather than delegating to another query engine.
- Vortex is not “the fallback”; it is the native provider family that ShardLoom can admit through policy and certificates.
3. User-facing entry points
- The main runtime surface is the CLI in shardloom-cli/.
- The Python package in python/ is a thin client over the CLI JSON protocol, not a native engine binding.
- The public website under website/ is a claim-safe interpretation layer, not the full docs or product surface.
4. Core shared model
- shardloom-core/ holds the cross-cutting domain model: capabilities, schema, expressions, input/output contracts, manifests, execution certificates, observability, security, sessions, and engine-mode abstractions.
- This crate is the conceptual contract layer used by planning, execution, certification, and release logic.
5. Planning and translation layer
- shardloom-plan/ turns front-door requests into executable plans.
- It contains plan IR, optimizer logic, estimation, explainability, input bridging, object-store planning, scan planning, and the execution facade.
- Architecturally, this is where ShardLoom decides whether a request is admissible, how it should be routed, and what evidence must be emitted.
6. Execution/runtime layer
- shardloom-exec/ provides the generic execution runtime concerns: memory, spill lifecycle, streaming, sizing, and recovery.
- This is the bounded execution substrate under the engine-specific layers.
- The repo treats spill, backpressure, and recovery as first-class runtime boundaries rather than incidental implementation details.
7. Vortex-specific engine layer
- shardloom-vortex/ is the main specialized engine integration layer.
- It covers encoded reads, metadata pruning, query primitives, local execution, ingest, commit protocol, output payloads, staged manifests, runtime/scheduler/memory bridges, and Vortex compatibility.
- This crate is where the “compute less, decode later” strategy is realized: metadata-first planning, encoded-columnar execution, and late materialization where possible.
8. Typical data flow
- The canonical path is: front door → source route → preparation route → execution route → output route → evidence route.
- For non-Vortex inputs, data flows through input adapters and source state into Vortex ingest / prepared state before native execution.
- Outputs are then written as artifacts or emitted inline, with certificates and diagnostics attached so the route can be audited.
9. CLI as orchestration and evidence emitter
- shardloom-cli/ is not just a command shell; it is the repository’s main orchestration boundary.
- It organizes command families for planning, diagnostics, benchmark evidence, SQL-local-source smoke paths, prepared/native Vortex paths, object-store planning, packaging/deployment, and release readiness checks.
- The CLI emits typed envelopes, certificates, fallback status, and route/evidence fields that downstream tooling can consume.
10. Python package as protocol client
- python/src/shardloom/ exposes client, context, query, and model helpers that mirror the CLI protocol.
- It is designed to preserve typed result refs, artifact refs, certificates, diagnostics, and fallback status from CLI output.
- Architecturally, Python is a convenience and integration surface, not an independent execution engine.
11. Correctness and claim gating
- shardloom-contract-tests/ enforces the architecture’s boundaries: no-fallback invariants, terminology mappings, execution certificate contracts, benchmark evidence rules, and supported/unsupported path behavior.
- These tests act as a “claim gate” for the codebase: if a route cannot prove itself, it should remain blocked or report-only.
12. Benchmarks and evidence pipeline
- benchmarks/ and docs/benchmarks/ define workload scenarios, taxonomy, and result handling.
- Benchmark artifacts are treated as architectural evidence, and the website publishes a curated slice of that evidence in website/assets/.
- This makes performance and compatibility claims part of the repo’s architecture, not just operational output.
13. Docs, RFCs, and public interpretation
- docs/rfcs/, docs/architecture/, docs/use-cases/, and docs/skills/ provide the long-form contract and design history.
- The public website is intentionally narrower than the repository docs: it presents a safe, current snapshot of the system rather than the full internal capability matrix.
14. Packaging, release, and CI boundary
- Cargo workspace metadata, Python packaging, conda recipes, GitHub Actions, and release-validation scripts form the delivery boundary.
- These tools don’t define product behavior, but they strongly affect the architecture because they gate what can be published, verified, and claimed.
- In practice, the repo is structured so that runtime behavior, evidence generation, and release readiness are all independently checkable.
15. How to draw the diagram
- Put CLI and Python client as the main ingress nodes.
- Route both into core contracts + planning.
- From planning, branch into generic execution runtime and the Vortex-specific engine.
- Show ingest/preparation feeding prepared Vortex state, then encoded/native execution, then output/artifact commit, with certificates/diagnostics flowing alongside every stage.
- Off to the side, show contract tests, benchmarks, and website/docs as evidence and publication subsystems that validate and explain the runtime, but do not execute user workloads.
Main entry points: src/lib.rs Plan IR · src/lib.rs Vortex engine · main.rs CLI · src/lib.rs Core model
Components 20
Ingress
User-facing entry points and protocol surface
- CLI · Main command boundary; emits typed envelopes, certificates, diagnostics, and route evidence
shardloom-cli/src/main.rs - Python client · Protocol client over the CLI JSON surface; preserves typed refs, artifacts, and certificates
python/src/shardloom/client.py
Core
Shared contracts and architecture model
- Core model · Cross-cutting domain model for capabilities, schema, manifests, certificates, observability, security, and sessions
shardloom-core/src/lib.rs - Architecture spine · Terminology, claim gating, and architecture-wide policy boundary
shardloom-core/src/architecture_spine.rs - Certificates · Execution certificates, diagnostics, and claim fields attached to runtime outcomes
shardloom-core/src/execution_certificate.rs
Planning
Admissibility, routing, and plan construction
- Plan IR · Front-door request shaping, admissibility, routing, optimizer, explainability, and execution facade
shardloom-plan/src/lib.rs - Input bridge · Bridges source state and external inputs into planning
shardloom-plan/src/input_bridge.rs - Object-store plan · Object-store routing and scan planning for distributed or remote inputs
shardloom-plan/src/object_store.rs
Runtime
Generic execution substrate and Vortex-specific engine
- Exec runtime · Memory, spill, streaming, sizing, and recovery substrate under engine-specific layers
shardloom-exec/src/lib.rs - Spill lifecycle · First-class spill reservation, payload, and recovery lifecycle
shardloom-exec/src/spill_lifecycle.rs - Vortex engine · Specialized encoded-read, ingest, commit, metadata-pruning, and runtime bridge layer
shardloom-vortex/src/lib.rs - Metadata prune · Metadata-first pruning, summary, and read planning before decoding
shardloom-vortex/src/metadata_pruning.rs - Vortex ingest · Preparation and ingest path that turns source-backed inputs into prepared Vortex state
shardloom-vortex/src/vortex_ingest.rs - Encoded read · Encoded-columnar execution with late materialization and primitive kernels
shardloom-vortex/src/encoded_read_executor.rs - Output commit · Finalization, staged manifests, output payloads, and commit protocol
shardloom-vortex/src/commit_protocol.rs
Evidence
Tests, benchmarks, docs, and publication surfaces
- Contract tests · No-fallback invariants, correctness contracts, diagnostics stability, and supported-path behavior
shardloom-contract-tests/src/lib.rs - Benchmarks · Workload scenarios, taxonomy, and benchmark artifact handling
benchmarks/traditional_analytics/run.py - Docs & RFCs · Long-form architecture, RFCs, use cases, and skills guidance
docs/architecture/global-architecture-review.md - Website · Claim-safe public interpretation layer and curated evidence publication surface
website/index.html
Delivery
Packaging, release, and CI gates
- Release gates · Packaging, CI, release validation, and publication checks
scripts/check_release_readiness.py
Connections 25
- CLI → Core model: uses contracts · CLI validates and serializes outcomes against shared domain contracts
- Python client → CLI: calls protocol · Python delegates work through the CLI JSON protocol
- CLI → Plan IR: builds plans · CLI front door routes requests into plan construction and routing
- Plan IR → Core model: reads model · Planning depends on shared capabilities, schema, manifests, and certificates
- Plan IR → Input bridge: bridges inputs · Planning admits source state and external inputs through adapters
- Plan IR → Object-store plan: routes object store · Planning creates object-store scans and remote-input routes
- Plan IR → Exec runtime: exec facade · Plan facade hands executable work into the runtime substrate
- Exec runtime → Spill lifecycle: manages spill · Runtime treats spill as a core lifecycle boundary
- Plan IR → Vortex engine: specializes for Vortex · Admitted requests are lowered into the Vortex engine path when compatible
- Vortex engine → Metadata prune: prunes metadata · Vortex chooses metadata-first execution and deferred decoding
- Vortex engine → Vortex ingest: prepares state · Source-backed inputs are ingested into prepared Vortex state
- Vortex engine → Encoded read: executes encoded · Native execution uses encoded reads and query primitives
- Vortex engine → Output commit: commits output · Execution finalizes through staged manifests and output payload commit
- Vortex engine → Certificates: emits evidence · Runtime attaches certificates and diagnostics to outputs
- CLI → Certificates: surfaces evidence · CLI exposes typed envelopes, fallback status, diagnostics, and certificates
- Contract tests → Core model: guards invariants · Contract tests enforce architecture claims and supported/unsupported behavior
- Contract tests → CLI: checks envelopes · Contract tests snapshot CLI protocol and claim-gate behavior
- Benchmarks → Certificates: produces artifacts · Benchmarks generate evidence artifacts and result records
- Benchmarks → Website: publishes slice · Curated benchmark evidence is published to the public site
- Docs & RFCs → Core model: specifies model · Docs and RFCs define the long-form contract and terminology
- Docs & RFCs → Plan IR: specifies flow · Architecture docs describe admissibility, translation, and execution flow
- Website → Docs & RFCs: curates · Website presents a safe, narrower interpretation of the repository docs
- Release gates → Contract tests: gates claims · Release validation depends on contract and readiness checks
- Release gates → Benchmarks: gates evidence · Packaging and release gates consume benchmark evidence and artifacts
- Release gates → Website: validates publish · Website publication is checked as part of release readiness