saminkhan1/context-compression

1. Core purpose

This repository implements a deterministic “context selector” that optimizes structured files before an AI agent reads them. The key architectural constraint is losslessness: the selected representation must round-trip back to the parsed source value, and the source file itself is never rewritten.

2. Main runtime flow

The central flow is: an agent or hook detects a file read, calls the selector, the selector evaluates candidate encodings, verifies round-trip safety, and returns a context-selector/v1 report with a trusted read_path. Adapters then substitute that verified path for the original file when safe.

3. Selector engine

selector.py is the core decision engine and CLI. It parses supported inputs, generates candidate representations, counts tokens with the active model tokenizer, applies safety checks, and chooses the lowest-token valid candidate subject to savings thresholds. The report includes source path, selected format, token counts, hashes, and verification status.

4. Supported data boundary

The selector is intentionally limited to structured tabular/text data: JSON, JSONL, CSV, and TSV. Unsupported or semantically risky shell-style reads stay raw. The repo’s boundary is “structured file compression,” not general prompt compression or summarization.

5. Tokenizer and model resolution

Model metadata is resolved from adapter payloads, Codex config, environment variables, and a bundled model-catalog.snapshot.json. Tokenization is handled with tiktoken for OpenAI-compatible models, or a Hugging Face tokenizer.json when configured. If exact tokenization is unavailable, fallback counts are explicitly treated as estimates.

6. Adapter layer

adapters/ contains thin runtime integrations for different agent hosts: Claude Code, Pi, Hermes Agent, MCP, OpenClaw, and a generic contract document. These adapters do not implement compression logic themselves; they all delegate to the same selector/verifier contract and only vary in how they intercept reads or expose a tool.

7. Codex hook runtime

hook.py and run-hook.sh implement the conservative Codex hook path. The hook rewrites only safe whole-file Bash reads, writes optimized sidecars under .codex/context-cache/, and avoids adding explanatory text to the conversation. The README notes that some hook surfaces, like UserPromptSubmit, are intentionally no-ops because they cannot safely substitute pasted text or attachments.

8. Verification boundary

verify_selector_report.py and the test suite enforce the contract that adapters must trust only the verified read_path, not any raw output path. Round-trip safety, hash/auditability, and deterministic decision making are treated as first-class invariants.

9. Benchmark and evidence pipeline

benchmark.py builds corpora, runs token comparisons, and can compare against external codecs via reproducible baselines. Checked-in reports under reports/ and EVIDENCE.md capture the current claim surface: deterministic savings and round-trip correctness, with answer-parity evidence left as an optional evaluation path rather than a production guarantee.

10. Optional quality evals

The evals/ directory contains scripts to build paired raw/optimized datasets and verify dataset integrity for later Inspect AI runs. This is a separate research/evaluation lane, not part of the runtime selector path, and it is explicitly gated from being treated as answer-parity proof until model runs are completed.

11. Tests and operational checks

tests/ covers adapters, selector behavior, hooks, benchmark logic, usage stats, and smoke paths. scripts/ contains operational checks such as clean-install verification, harness contract verification, evidence verification, and usage-stat exports. This keeps the repository split between runtime logic and validation tooling.

12. Artifacts and local state

Generated content is expected to live outside versioned source: .codex/context-cache/, data/benchmark-corpus/, logs/, and virtualenv/cache directories. The architecture assumes source files remain immutable and all optimization is expressed through verified sidecars and report surfaces.