# saminkhan1/context-compression architecture

> Architecture diagram of the GitHub repository saminkhan1/context-compression, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saminkhan1/context-compression
- Repository: https://github.com/saminkhan1/context-compression

## Overview

1. Core purpose  
This repository implements a deterministic “context selector” that optimizes structured files before an AI agent reads them. The key architectural constraint is losslessness: the selected representation must round-trip back to the parsed source value, and the source file itself is never rewritten.

2. Main runtime flow  
The central flow is: an agent or hook detects a file read, calls the selector, the selector evaluates candidate encodings, verifies round-trip safety, and returns a `context-selector/v1` report with a trusted `read_path`. Adapters then substitute that verified path for the original file when safe.

3. Selector engine  
`selector.py` is the core decision engine and CLI. It parses supported inputs, generates candidate representations, counts tokens with the active model tokenizer, applies safety checks, and chooses the lowest-token valid candidate subject to savings thresholds. The report includes source path, selected format, token counts, hashes, and verification status.

4. Supported data boundary  
The selector is intentionally limited to structured tabular/text data: JSON, JSONL, CSV, and TSV. Unsupported or semantically risky shell-style reads stay raw. The repo’s boundary is “structured file compression,” not general prompt compression or summarization.

5. Tokenizer and model resolution  
Model metadata is resolved from adapter payloads, Codex config, environment variables, and a bundled `model-catalog.snapshot.json`. Tokenization is handled with `tiktoken` for OpenAI-compatible models, or a Hugging Face `tokenizer.json` when configured. If exact tokenization is unavailable, fallback counts are explicitly treated as estimates.

6. Adapter layer  
`adapters/` contains thin runtime integrations for different agent hosts: Claude Code, Pi, Hermes Agent, MCP, OpenClaw, and a generic contract document. These adapters do not implement compression logic themselves; they all delegate to the same selector/verifier contract and only vary in how they intercept reads or expose a tool.

7. Codex hook runtime  
`hook.py` and `run-hook.sh` implement the conservative Codex hook path. The hook rewrites only safe whole-file Bash reads, writes optimized sidecars under `.codex/context-cache/`, and avoids adding explanatory text to the conversation. The README notes that some hook surfaces, like `UserPromptSubmit`, are intentionally no-ops because they cannot safely substitute pasted text or attachments.

8. Verification boundary  
`verify_selector_report.py` and the test suite enforce the contract that adapters must trust only the verified `read_path`, not any raw output path. Round-trip safety, hash/auditability, and deterministic decision making are treated as first-class invariants.

9. Benchmark and evidence pipeline  
`benchmark.py` builds corpora, runs token comparisons, and can compare against external codecs via reproducible baselines. Checked-in reports under `reports/` and `EVIDENCE.md` capture the current claim surface: deterministic savings and round-trip correctness, with answer-parity evidence left as an optional evaluation path rather than a production guarantee.

10. Optional quality evals  
The `evals/` directory contains scripts to build paired raw/optimized datasets and verify dataset integrity for later Inspect AI runs. This is a separate research/evaluation lane, not part of the runtime selector path, and it is explicitly gated from being treated as answer-parity proof until model runs are completed.

11. Tests and operational checks  
`tests/` covers adapters, selector behavior, hooks, benchmark logic, usage stats, and smoke paths. `scripts/` contains operational checks such as clean-install verification, harness contract verification, evidence verification, and usage-stat exports. This keeps the repository split between runtime logic and validation tooling.

12. Artifacts and local state  
Generated content is expected to live outside versioned source: `.codex/context-cache/`, `data/benchmark-corpus/`, `logs/`, and virtualenv/cache directories. The architecture assumes source files remain immutable and all optimization is expressed through verified sidecars and report surfaces.

Main entry points: [`selector.py`](https://github.com/saminkhan1/context-compression/blob/main/selector.py) (Selector), [`adapters/CONTRACT.md`](https://github.com/saminkhan1/context-compression/blob/main/adapters/CONTRACT.md) (Adapter Contract), [`verify_selector_report.py`](https://github.com/saminkhan1/context-compression/blob/main/verify_selector_report.py) (Report Verify), [`adapters/claude-code/context_selector_hook.py`](https://github.com/saminkhan1/context-compression/blob/main/adapters/claude-code/context_selector_hook.py) (Claude Code)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_selector["Selector<br/>decision engine<br/>[selector.py]"]
  node_hook["Codex Hook<br/>runtime hook<br/>[hook.py]"]
  node_run_hook["Hook Runner<br/>shell entrypoint<br/>[run-hook.sh]"]
end

subgraph group_adapters["Adapters"]
  node_adapters_contract["Adapter Contract<br/>integration contract<br/>[CONTRACT.md]"]
  node_claude_adapter["Claude Code<br/>host adapter"]
  node_pi_adapter["Pi Adapter<br/>host adapter"]
  node_hermes_adapter["Hermes Plugin<br/>host adapter<br/>[__init__.py]"]
  node_mcp_adapter["MCP Server<br/>host adapter"]
  node_openclaw_adapter["OpenClaw<br/>host adapter<br/>[index.ts]"]
end

subgraph group_validation["Validation"]
  node_report_schema["Report Schema<br/>contract schema"]
  node_verify_report["Report Verify<br/>contract verifier"]
  node_usage_stats["Usage Stats<br/>metrics export"]
  node_clean_install["Clean Install<br/>ops check"]
  node_harness_checks["Harness Checks<br/>ops check"]
end

subgraph group_research["Research"]
  node_benchmark["Benchmark<br/>comparison pipeline<br/>[benchmark.py]"]
  node_evals_build["Eval Build<br/>dataset builder"]
  node_evals_check["Eval Check<br/>dataset verifier"]
  node_benchmark_report["Benchmark Report<br/>evidence artifact"]
end

subgraph group_artifacts["Artifacts"]
end

node_hook -->|"invokes"| node_selector
node_run_hook -->|"starts"| node_hook
node_selector -->|"emits"| node_report_schema
node_verify_report -->|"validates"| node_report_schema
node_verify_report -->|"checks"| node_selector
node_claude_adapter -->|"follows"| node_adapters_contract
node_pi_adapter -->|"follows"| node_adapters_contract
node_hermes_adapter -->|"follows"| node_adapters_contract
node_mcp_adapter -->|"follows"| node_adapters_contract
node_openclaw_adapter -->|"follows"| node_adapters_contract
node_claude_adapter -->|"delegates"| node_selector
node_pi_adapter -->|"delegates"| node_selector
node_hermes_adapter -->|"delegates"| node_selector
node_mcp_adapter -->|"delegates"| node_selector
node_openclaw_adapter -->|"delegates"| node_selector
node_benchmark -->|"writes"| node_benchmark_report
node_evals_build -->|"feeds"| node_evals_check
node_harness_checks -->|"tests"| node_verify_report
node_clean_install -->|"checks"| node_selector
node_usage_stats -->|"summarizes"| node_benchmark_report

click node_selector "https://github.com/saminkhan1/context-compression/blob/main/selector.py"
click node_hook "https://github.com/saminkhan1/context-compression/blob/main/hook.py"
click node_run_hook "https://github.com/saminkhan1/context-compression/blob/main/run-hook.sh"
click node_report_schema "https://github.com/saminkhan1/context-compression/blob/main/docs/schemas/context-selector-v1.schema.json"
click node_verify_report "https://github.com/saminkhan1/context-compression/blob/main/verify_selector_report.py"
click node_adapters_contract "https://github.com/saminkhan1/context-compression/blob/main/adapters/CONTRACT.md"
click node_claude_adapter "https://github.com/saminkhan1/context-compression/blob/main/adapters/claude-code/context_selector_hook.py"
click node_pi_adapter "https://github.com/saminkhan1/context-compression/blob/main/adapters/pi/context-selector-tool.ts"
click node_hermes_adapter "https://github.com/saminkhan1/context-compression/blob/main/adapters/hermes-plugin/__init__.py"
click node_mcp_adapter "https://github.com/saminkhan1/context-compression/blob/main/adapters/mcp/context_selector_server.py"
click node_openclaw_adapter "https://github.com/saminkhan1/context-compression/blob/main/adapters/openclaw/index.ts"
click node_benchmark "https://github.com/saminkhan1/context-compression/blob/main/benchmark.py"
click node_evals_build "https://github.com/saminkhan1/context-compression/blob/main/evals/build_context_quality_dataset.py"
click node_evals_check "https://github.com/saminkhan1/context-compression/blob/main/evals/verify_context_quality_dataset.py"
click node_usage_stats "https://github.com/saminkhan1/context-compression/blob/main/scripts/export_usage_stats.py"
click node_clean_install "https://github.com/saminkhan1/context-compression/blob/main/scripts/verify_clean_install.py"
click node_harness_checks "https://github.com/saminkhan1/context-compression/blob/main/scripts/verify_harness_contracts.py"
click node_benchmark_report "https://github.com/saminkhan1/context-compression/blob/main/reports/benchmark-report.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_selector,node_hook,node_run_hook toneBlue
class node_adapters_contract,node_claude_adapter,node_pi_adapter,node_hermes_adapter,node_mcp_adapter,node_openclaw_adapter toneAmber
class node_report_schema,node_verify_report,node_usage_stats,node_clean_install,node_harness_checks toneMint
class node_benchmark,node_evals_build,node_evals_check,node_benchmark_report toneRose
```

## Components

### Runtime

Core selector execution and hook integration

- [Selector](https://github.com/saminkhan1/context-compression/blob/main/selector.py): decision engine · Parses structured files, generates candidate encodings, verifies round-trip safety, and selects the lowest-token valid representation. · `selector.py`
- [Codex Hook](https://github.com/saminkhan1/context-compression/blob/main/hook.py): runtime hook · Conservative file-read interception path for safe whole-file substitutions and sidecar writes. · `hook.py`
- [Hook Runner](https://github.com/saminkhan1/context-compression/blob/main/run-hook.sh): shell entrypoint · Shell wrapper that launches the hook runtime. · `run-hook.sh`

### Adapters

Host-specific integrations that delegate to the selector contract

- [Adapter Contract](https://github.com/saminkhan1/context-compression/blob/main/adapters/CONTRACT.md): integration contract · Shared contract document for thin host adapters. · `adapters/CONTRACT.md`
- [Claude Code](https://github.com/saminkhan1/context-compression/blob/main/adapters/claude-code/context_selector_hook.py): host adapter · Claude Code hook integration that delegates to the selector contract. · `adapters/claude-code/context_selector_hook.py`
- [Pi Adapter](https://github.com/saminkhan1/context-compression/blob/main/adapters/pi/context-selector-tool.ts): host adapter · TypeScript adapter for Pi host integration. · `adapters/pi/context-selector-tool.ts`
- [Hermes Plugin](https://github.com/saminkhan1/context-compression/blob/main/adapters/hermes-plugin/__init__.py): host adapter · Hermes Agent plugin wrapper around the selector contract. · `adapters/hermes-plugin/__init__.py`
- [MCP Server](https://github.com/saminkhan1/context-compression/blob/main/adapters/mcp/context_selector_server.py): host adapter · MCP server exposing selector functionality as a tool. · `adapters/mcp/context_selector_server.py`
- [OpenClaw](https://github.com/saminkhan1/context-compression/blob/main/adapters/openclaw/index.ts): host adapter · OpenClaw integration that exposes the selector through a plugin package. · `adapters/openclaw/index.ts`

### Validation

Verification, tests, and operational checks

- [Report Schema](https://github.com/saminkhan1/context-compression/blob/main/docs/schemas/context-selector-v1.schema.json): contract schema · Defines the context-selector/v1 report shape and trusted read-path contract. · `docs/schemas/context-selector-v1.schema.json`
- [Report Verify](https://github.com/saminkhan1/context-compression/blob/main/verify_selector_report.py): contract verifier · Checks selector reports and enforces trust in verified read paths only. · `verify_selector_report.py`
- [Usage Stats](https://github.com/saminkhan1/context-compression/blob/main/scripts/export_usage_stats.py): metrics export · Exports usage telemetry and operational statistics. · `scripts/export_usage_stats.py`
- [Clean Install](https://github.com/saminkhan1/context-compression/blob/main/scripts/verify_clean_install.py): ops check · Verifies reproducible clean-install behavior. · `scripts/verify_clean_install.py`
- [Harness Checks](https://github.com/saminkhan1/context-compression/blob/main/scripts/verify_harness_contracts.py): ops check · Verifies harness contracts and smoke paths. · `scripts/verify_harness_contracts.py`

### Research

Benchmarking, evaluation, and evidence artifacts

- [Benchmark](https://github.com/saminkhan1/context-compression/blob/main/benchmark.py): comparison pipeline · Builds corpora and measures token savings and codec baselines. · `benchmark.py`
- [Eval Build](https://github.com/saminkhan1/context-compression/blob/main/evals/build_context_quality_dataset.py): dataset builder · Creates paired raw and optimized datasets for later quality runs. · `evals/build_context_quality_dataset.py`
- [Eval Check](https://github.com/saminkhan1/context-compression/blob/main/evals/verify_context_quality_dataset.py): dataset verifier · Verifies dataset integrity and supports context-quality smoke paths. · `evals/verify_context_quality_dataset.py`
- [Benchmark Report](https://github.com/saminkhan1/context-compression/blob/main/reports/benchmark-report.json): evidence artifact · Checked-in benchmark results and summary claims. · `reports/benchmark-report.json`

## Connections

- Codex Hook → Selector: invokes · Hook delegates file-read decisions to the selector engine.
- Hook Runner → Codex Hook: starts · Shell wrapper launches the hook runtime.
- Selector → Report Schema: emits · Selector returns a context-selector/v1 report matching the schema.
- Report Verify → Report Schema: validates · Verifier enforces the report contract and trusted read-path semantics.
- Report Verify → Selector: checks · Verification asserts deterministic, round-trip-safe selector behavior.
- Claude Code → Adapter Contract: follows · Adapter implements the shared host contract.
- Pi Adapter → Adapter Contract: follows · Adapter implements the shared host contract.
- Hermes Plugin → Adapter Contract: follows · Adapter implements the shared host contract.
- MCP Server → Adapter Contract: follows · Adapter implements the shared host contract.
- OpenClaw → Adapter Contract: follows · Adapter implements the shared host contract.
- Claude Code → Selector: delegates · Host integration relies on the same selection engine.
- Pi Adapter → Selector: delegates · Host integration relies on the same selection engine.
- Hermes Plugin → Selector: delegates · Host integration relies on the same selection engine.
- MCP Server → Selector: delegates · Host integration relies on the same selection engine.
- OpenClaw → Selector: delegates · Host integration relies on the same selection engine.
- Benchmark → Benchmark Report: writes · Benchmark pipeline produces checked-in evidence outputs.
- Eval Build → Eval Check: feeds · Dataset builder output is verified before evaluation use.
- Harness Checks → Report Verify: tests · Operational checks exercise the report verification boundary.
- Clean Install → Selector: checks · Sanity checks cover the runtime installation path.
- Usage Stats → Benchmark Report: summarizes · Usage metrics inform evidence and reporting surfaces.