How elara-labs/code-context-engine works
1. Overall shape
- This repository is a Python 3.11+ CLI/runtime for “Code Context Engine,” a local code indexing and retrieval system that integrates with AI coding tools through MCP and editor-specific config files.
- The core architectural boundary is between project-local indexing/search state and external clients such as Claude Code, Cursor, VS Code/Copilot, Gemini CLI, Codex, OpenCode, and Tabnine.
2. Primary execution surfaces
- The main user entrypoint is the cce CLI in src/context_engine/cli.py.
- Secondary surfaces are:
- an MCP server for tool access,
- a local HTTP dashboard/server,
- and editor/session hooks that capture context and keep the index current.
3. Indexing subsystem
- src/context_engine/indexer/ is the ingestion pipeline for source files.
- It handles file discovery, ignore rules, secret redaction, chunking, embedding, manifest tracking, git-aware indexing, and filesystem watching.
- The important data flow is: repository files → semantic chunks → embeddings + metadata → persisted index state.
4. Chunking and semantic parsing
- chunker.py and related indexer modules suggest AST-aware chunking for supported languages, with fallback line-based chunking for other text files.
- README indicates Tree-sitter is used for semantic boundaries such as functions, classes, and modules.
- This is the first major architecture boundary: raw files are transformed into smaller searchable units before any retrieval happens.
5. Embedding and cache layer
- embedder.py and embedding_cache.py indicate embeddings are generated per chunk and cached by content hash.
- The cache is central to fast re-indexing: unchanged chunks skip recomputation, which is why incremental updates are fast.
- The repository supports both local embedding execution and Ollama-backed embedding via compression/ollama_client.py and README-described local vs Ollama modes.
6. Storage subsystem
- src/context_engine/storage/ is the persistence boundary.
- It contains separate stores for:
- vector search,
- full-text search,
- graph relationships,
- and backend selection for local vs remote operation.
- README states the system uses SQLite-based storage, including vector and FTS capabilities, with graph relationships persisted alongside the index. This implies the index is not just embeddings but a multi-index bundle.
7. Retrieval subsystem
- src/context_engine/retrieval/ turns a natural-language query into search candidates and ranks them.
- Key pieces include query parsing, retrieval orchestration, and confidence scoring.
- The retrieval path is hybrid: vector similarity + BM25/keyword matching, then graph expansion for related code. This is the main “search instead of re-read files” data flow exposed to agents.
8. Graph expansion and code relationships
- storage/graph_store.py and retrieval modules imply explicit relationships such as imports and calls.
- This is a distinct boundary from pure similarity search: once a chunk is found, related code can be pulled in through graph edges to give agents surrounding context without indexing entire files.
9. Compression subsystem
- src/context_engine/compression/ reduces the size of retrieved or generated text.
- It includes rules, prompts, quality checks, and an Ollama client for LLM-assisted compression when available.
- The output of retrieval is therefore not just “which chunks,” but “which chunks after compression,” which materially affects token economics and tool responses.
10. Memory and session persistence
- src/context_engine/memory/ stores cross-session knowledge such as decisions, code areas, and session summaries.
- It uses its own database, grammar-based compression, extractive logic, migration support, and hook integration.
- This forms a second persistence lane beside the code index: it remembers project decisions and prior work, not just source code.
11. Integration layer for AI tools
- src/context_engine/integration/ bridges the engine to external AI workflows.
- mcp_server.py exposes the tool surface to MCP clients; bootstrap.py, git_context.py, and session_capture.py support initialization and session-aware behavior.
- The README’s listed tools (context_search, expand_chunk, related_context, session_recall, record_decision, etc.) map to this layer.
12. Editor and project bootstrap
- editors.py, project_commands.py, and config.py support auto-detecting editors and writing the correct per-editor configuration files.
- cce init is the orchestration point that installs hooks, registers MCP servers, and writes instructions/config for the target editor(s).
- This is where the repository crosses from “engine” into “workspace integration.”
13. Dashboard and service hosting
- src/context_engine/dashboard/ and serve_http.py provide a local status/dashboard UI.
- The dashboard consumes index, session, and savings data to show freshness, history, and token savings.
- It is a read-only observability surface over the same underlying index and memory stores.
14. Operational boundaries and safety
- indexer/secrets.py and memory redaction logic show that sensitive data is filtered before persistence.
- ignorefile.py, git hooks, and watcher support keep indexing scoped to relevant files and current changes.
- event_bus.py, services.py, and utils.py likely provide internal coordination across these subsystems without exposing them directly to clients.
15. Non-runtime parts of the repo
- docs-src/, docs/, and benchmarks/ are substantial but ancillary: they support documentation, site generation, and reproducible benchmark reporting.
- benchmarks/run_benchmark.py and the saved results show the project’s claim that retrieval/compression efficiency is a core product concern, not just a development detail.
16. Diagramming summary
- If drawing the architecture, the main boxes are:
- CLI/bootstrap
- Indexing pipeline
- Storage backends
- Retrieval/search
- Compression
- Memory/session store
- MCP/editor integration
- Dashboard/observability
- The key arrows are:
- repository files → indexer → storage,
- user query → retrieval → compressed context → MCP client,
- sessions/hooks → memory store → recall tools,
- and CLI/init → editor configs + hook installation.
Main entry points: retriever.py Retriever · storage Storage · integration Integration · indexer Indexer
Components 20
Entry & integration
CLI, MCP, editor bootstrap, and local serving surfaces.
- CCE CLI · Main command-line interface and orchestration surface.
src/context_engine/cli.py - Integration · Bridges the engine to MCP clients, editor setup, and session capture.
src/context_engine/integration - MCP server · Exposes search, expansion, and memory tools to AI clients.
src/context_engine/integration/mcp_server.py - Bootstrap · Initializes project integrations, git context, and session wiring.
src/context_engine/integration/bootstrap.py - Project commands · Writes editor configs and installs project-local command hooks.
src/context_engine/project_commands.py
Indexing pipeline
Turns repository files into chunks, embeddings, and index state.
- Indexer · Repository file discovery, chunking, embedding, and persistence.
src/context_engine/indexer - Chunker · AST-aware and fallback chunk boundaries for source and text files.
src/context_engine/indexer/chunker.py - Embedder · Generates embeddings for chunks and coordinates embedding backends.
src/context_engine/indexer/embedder.py - Embed cache · Caches embeddings by content hash to accelerate incremental re-indexing.
src/context_engine/indexer/embedding_cache.py
Storage backends
Persistence for vector, text, graph, and backend selection.
- Storage · Backend selection and store implementations for index data.
src/context_engine/storage - Vector store · Persists vector search data for semantic retrieval.
src/context_engine/storage/vector_store.py - FTS store · Persists full-text search data for BM25 and lexical matching.
src/context_engine/storage/fts_store.py - Graph store · Persists code relationships such as imports and call links.
src/context_engine/storage/graph_store.py
Retrieval & compression
Query handling, ranking, expansion, and context reduction.
- Retriever · Combines query parsing, hybrid ranking, and graph expansion.
src/context_engine/retrieval/retriever.py - Query parser · Interprets natural-language search requests and retrieval intent.
src/context_engine/retrieval/query_parser.py - Compressor · Shrinks retrieved or generated context to fit token budgets.
src/context_engine/compression/compressor.py
Memory & sessions
Cross-session decisions, summaries, and hook-based capture.
- Memory · Persists decisions, summaries, and cross-session knowledge.
src/context_engine/memory - Hook server · Receives editor and agent events for memory capture workflows.
src/context_engine/memory/hook_server.py
Observability
Local dashboard and service endpoints for status and savings.
- HTTP server · Serves the local dashboard and related HTTP endpoints.
src/context_engine/serve_http.py - Dashboard · Read-only view over index freshness, history, and savings.
src/context_engine/dashboard
Connections 24
- CCE CLI → Indexer: starts indexing · The CLI orchestrates repository indexing and maintenance.
- CCE CLI → Integration: boots integrations · The CLI sets up MCP, hooks, and editor-facing integrations.
- CCE CLI → Project commands: writes config · Project commands install editor configs and local project commands.
- Integration → MCP server: exposes tools · Integration layer publishes engine capabilities through MCP.
- Integration → Bootstrap: initializes workspace · Bootstrap wires project context and session-aware setup.
- Integration → Hook server: captures sessions · Integration forwards session events into memory capture.
- HTTP server → Dashboard: serves ui · HTTP service hosts the dashboard and related views.
- Indexer → Chunker: chunks files · Source files are segmented into searchable units.
- Indexer → Embedder: embeds chunks · Chunks are transformed into embeddings.
- Embedder → Embed cache: reuses cache · Embedding computation skips unchanged content via cache.
- Indexer → Storage: persists index · Ingested chunks and metadata are written to storage.
- Storage → Vector store: stores vectors · Vector data is persisted for semantic search.
- Storage → FTS store: stores text · Lexical index data is persisted for keyword search.
- Storage → Graph store: stores graph · Relationship edges are persisted alongside the index.
- Retriever → Query parser: parses query · Search requests are interpreted before retrieval execution.
- Retriever → Vector store: scores semantics · Retriever consults vector similarity for candidate search.
- Retriever → FTS store: matches keywords · Retriever consults full-text search for lexical candidates.
- Retriever → Graph store: expands graph · Retriever pulls related code through graph edges.
- Retriever → Compressor: compresses context · Retrieved context is reduced before returning to clients.
- Compressor → MCP server: feeds tools · Compressed context is returned through tool surfaces.
- Hook server → Memory: writes memory · Captured sessions and decisions are persisted for recall.
- Memory → MCP server: serves recall · Memory is surfaced to clients through MCP tools.
- Dashboard → Memory: reads metrics · Dashboard consumes session and savings information.
- Dashboard → Storage: reads status · Dashboard reflects index and storage state.