How mathews-tom/archex works
1. What the system is
archex is a local repository-context engine for agents. It does not answer questions itself; it builds a ranked, token-budgeted context bundle plus a receipt that explains freshness, completeness, skipped candidates, and omitted edges.
2. Primary entry surfaces
The same core capabilities are exposed through several front doors: the CLI under src/archex/cli/, the Python API in src/archex/api.py, the MCP server in src/archex/integrations/mcp.py, and Docker images built from docker/. These are delivery surfaces, not separate implementations.
3. Repo-local state boundary
All generated state lives under .archex/ in the target repository. That boundary holds settings, metadata, SQLite index data, optional vector artifacts, graph artifacts, and dogfood/baseline files. The project is designed to run locally by default with no hosted dependency.
4. Repository acquisition
Input repositories are discovered and normalized by src/archex/acquire/:
- local.py for the current checkout
- git.py for git-based sources
- discovery.py for repo detection and setup
This is the ingestion boundary between a filesystem/repo and the rest of the pipeline.
5. Language parsing and structural extraction
src/archex/parse/ is the structural extraction layer. It contains language adapters for full-tier languages, structured markup/config languages, and chunk-only languages. The parser extracts symbols, imports, and cross-file references where the tier supports them, and falls back to chunking for lower-fidelity formats.
6. Analysis layer
src/archex/analyze/ turns parsed structure into higher-level signals:
- interfaces.py for interface and type relationships
- modules.py for module-level structure
- patterns.py for architectural pattern detection
- decisions.py for inferred design decisions
This layer feeds both retrieval quality and architecture-oriented commands.
7. Indexing and retrieval core
src/archex/index/ is the main retrieval subsystem. It combines:
- BM25-style lexical search
- optional vector retrieval and embeddings
- sparse/rerank components
- graph-based relationships
- artifact/store management and delta handling
This is where candidate files, symbols, and chunks are scored and merged into a final ranking.
8. Query pipeline and bundle assembly
src/archex/pipeline/ and top-level query-related modules such as graph_query.py, explain.py, scout.py, and impact.py assemble the final response. The pipeline expands the initial retrieval set with graph/type context, applies token budgets, and emits the bundle the agent will actually consume.
9. Serving and rendering the context
src/archex/serve/ is the prompt-facing presentation layer. It handles context packing, modality/intent handling, compression, and output rendering to JSON, Markdown, or XML. This is the final transformation from ranked internal evidence into a consumable context envelope.
10. Receipts, status, and trust signals
src/archex/receipt.py, status.py, doctor.py, and related modules define the operational trust surface. Receipts explain what was included or skipped; doctor/status report index health, grammar availability, and freshness; these outputs are central to the repository’s “safe-to-act” story.
11. Metrics and privacy boundary
src/archex/metrics/ is a separate, opt-in local telemetry substitute: it records anonymous usage counters on the machine only, with no source code, query text, or prompt bodies in event rows. It is intentionally isolated from the core retrieval flow and from any hosted upload path.
12. Benchmarking and regression control
src/archex/benchmark/ plus the checked-in benchmarks/ corpus form a full evaluation harness. It drives task suites, baseline comparisons, gate checks, head-to-head reports, delta runs, and bundle-only evaluator flows. This is the main quality-control loop for retrieval, ranking, compression, and language promotion decisions.
13. Integrations and extension points
src/archex/integrations/ contains optional adapters for LangChain, LlamaIndex, LSAP/LSP-style enrichment, and MCP. The parser and language layers are also extensible via adapter registration, which is how new languages enter the system.
14. Tests and fixtures as architecture evidence
The large tests/ tree and tests/fixtures/ corpus are not just verification; they encode the intended boundaries of the system. The fixtures cover many languages and repository shapes, mirroring the parser tiers, graph behavior, benchmark tasks, and CLI lifecycle commands.
15. End-to-end flow to diagram
The main architecture flow is: repository source -> acquisition -> parse/analyze -> index/store in .archex/ -> retrieval/ranking/graph expansion -> budgeted bundle + receipt -> CLI/API/MCP/Docker consumer. The benchmark and metrics subsystems sit alongside this flow as validation and observability, not as part of the primary retrieval path.
Main entry points: serve Serve · index Index · pipeline Pipeline · benchmark Benchmark
Components 18
Core engine
Primary local repository-context pipeline.
- Repo source · Target repository discovered from the local filesystem or git source.
- Acquire · Normalizes local, git, and discovery inputs into a usable repo context.
src/archex/acquire - Parse · Language adapters and parsing engine for symbols, imports, and chunking.
src/archex/parse - Analyze · Derives interfaces, module structure, patterns, and design decisions.
src/archex/analyze - Index · BM25, vector, graph, fusion, rerank, artifact, and store management.
src/archex/index - Pipeline · Expands candidates, applies graph context, and enforces token budgets.
src/archex/pipeline - Serve · Packages intent-aware context and renders JSON, Markdown, or XML.
src/archex/serve - Receipt · Explains freshness, completeness, skipped candidates, and omitted edges.
src/archex/receipt.py - Graph query · Graph-based lookup and multi-hop expansion used by retrieval and impact flows.
src/archex/graph_query.py - Languages · Language tiering and adapter registration for parser extensibility.
src/archex/languages.py
Delivery surfaces
User-facing entry points and integrations.
- CLI · Primary command-line front door for indexing, querying, analysis, and status.
src/archex/cli - Python API · Programmatic entry surface for embedding archex in other tools.
src/archex/api.py - MCP server · Model Context Protocol serving surface for external agents.
src/archex/integrations/mcp.py - Docker · Container images for local deployment and portability.
docker
Validation and observability
Benchmarks, metrics, and trust signals.
- Metrics · Opt-in anonymous machine-local usage counters and health reporting.
src/archex/metrics - Benchmark · Task suites, baselines, gates, and regression reporting for quality control.
src/archex/benchmark - Doctor/status · Index health, grammar availability, freshness, and readiness reporting.
src/archex/doctor.py
Local state
Repository-local artifacts under .archex/.
- .archex · Local settings, metadata, SQLite index data, vector artifacts, and graph artifacts.
.archex
Connections 21
- Repo source → Acquire: discover · Repository inputs are discovered and normalized before processing.
- Acquire → Parse: parse · Acquired repositories are structurally extracted by language adapters.
- Parse → Analyze: derive signals · Parsed symbols, imports, and chunks feed higher-level analysis.
- Analyze → Index: index · Analysis enriches ranking, graph relationships, and stored artifacts.
- Index → Graph query: expand · Graph relationships are used to broaden and refine candidates.
- Index → Pipeline: ranked candidates · Search and rerank produce scored candidates for assembly.
- Graph query → Pipeline: context enrich · Graph/type context is merged into the query pipeline.
- Pipeline → Serve: bundle · Token-budgeted context bundles are prepared for presentation.
- Serve → Receipt: emit receipt · The final envelope includes provenance and omission details.
- Pipeline → Receipt: track omissions · Skipped items and budget decisions are recorded in the receipt.
- Serve → CLI: render · CLI exposes the packed context and report outputs.
- Serve → Python API: serve · Python callers consume the same context assembly pipeline.
- Serve → MCP server: serve · MCP exposes the same retrieval and rendering capabilities.
- Serve → Docker: package · Docker images deliver the runtime as a portable local tool.
- Index → .archex: persist · Indexes, metadata, and artifacts are stored in repo-local state.
- Metrics → Doctor/status: report · Metrics and health outputs contribute to operational visibility.
- Benchmark → Index: evaluate · Benchmarks measure retrieval and ranking quality of the core index.
- Benchmark → Pipeline: evaluate · Benchmark tasks also exercise bundle assembly and query flow.
- Benchmark → Doctor/status: gate · Readiness and regression gates summarize trust signals.
- Languages → Parse: register adapters · Language adapters are extended through registry-style hooks.
- Doctor/status → CLI: surface · Health and status checks are primarily exposed through the CLI.