# atomicstrata/llm-wiki-compiler architecture

> Architecture diagram of the GitHub repository atomicstrata/llm-wiki-compiler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/atomicstrata/llm-wiki-compiler
- Repository: https://github.com/atomicstrata/llm-wiki-compiler

## Overview

1. **Overall shape**
   - This is a Node.js CLI/compiler for building a persistent markdown wiki from raw sources.
   - The core architectural boundary is a workspace on disk with three important areas: `sources/` for inputs, `wiki/` for compiled output, and `.llmwiki/` for state, schema, review candidates, and locks.

2. **Primary pipeline**
   - The main flow is: ingest raw content → detect changes by hashing → extract concepts and dependencies → generate or update wiki pages → resolve wikilinks → regenerate `index.md`.
   - The compile path is intentionally two-phase: first discover concepts from all sources, then synthesize pages. That avoids order dependence and lets shared concepts merge across sources.

3. **Source ingestion subsystem**
   - `src/ingest` handles multiple source types: files, web URLs, PDFs, images, transcripts, and session exports from Claude/Codex/Cursor.
   - Ingestion normalizes everything into source files under `sources/`, with metadata such as truncation status and original size preserved in frontmatter or source state.

4. **Incremental compilation subsystem**
   - `src/compiler` is the central transformation layer.
   - It includes hashing, dependency resolution, provenance tracking, prompt budgeting, source-state tracking, page rendering, orphan detection, and wiki/index generation.
   - The compiler’s main boundary is “source state vs generated output”: only changed sources are reprocessed, while unchanged ones are skipped via hash-based comparison.

5. **Wiki page model**
   - Compiled output is Obsidian-compatible markdown with YAML frontmatter.
   - Pages are grouped into concept pages and query-result pages, with generated links and metadata such as `sources`, `confidence`, `provenanceState`, and `contradictedBy`.
   - The index and link resolution are part of the compile output, not a separate runtime service.

6. **Provenance and trust metadata**
   - The compiler carries source attribution into page content using paragraph markers and claim-level line citations.
   - Provenance-related modules support line-range validation, merged-source metadata reconciliation, confidence scores, and contradiction tracking.
   - This makes source traceability a first-class architectural concern, not an afterthought.

7. **Query subsystem**
   - `llmwiki query` is a separate retrieval-and-answer pipeline over the compiled wiki.
   - It uses semantic retrieval and embeddings when available, falls back when necessary, and can save answers back into the wiki as new pages.
   - That “save” path closes the loop: query output becomes future source material, so the wiki compounds over time.

8. **Provider abstraction**
   - `src/providers` and `src/adapters` isolate model/vendor differences behind a common interface.
   - The repo supports Anthropic, OpenAI-compatible endpoints, Ollama, Copilot, and other adapter flows, with env-based config and provider-specific fallbacks.
   - This means the compile/query pipelines depend on a provider boundary, not a specific vendor SDK.

9. **Review workflow boundary**
   - `compile --review` does not write directly into `wiki/`; it emits candidate records into `.llmwiki/candidates/`.
   - Separate review commands approve, reject, or inspect candidates, and approval refreshes the wiki/index while holding a lock to serialize changes.
   - This creates a human-in-the-loop gate between generation and publication.

10. **Schema and linting**
   - The optional `.llmwiki/schema.json` layer defines page kinds and cross-link expectations beyond plain concepts.
   - `src/linter` enforces structural and semantic rules: broken links, orphans, low confidence, contradictions, uncited paragraphs, and schema violations.
   - Schema and linting are validation boundaries around the generated knowledge graph.

11. **Export and interoperability**
   - `src/export` turns the compiled wiki into portable formats: `llms.txt`, JSON, JSON-LD, GraphML, and Marp slides.
   - This subsystem is downstream of compilation and reads the wiki as the source of truth rather than re-running generation logic.

12. **MCP server boundary**
   - `src/mcp` exposes the system as tools/resources for agents through Model Context Protocol.
   - The server provides read-only access, ingest, compile, query, lint, and status operations, plus resources like the index, individual pages, and compilation state.
   - This is the external integration boundary for AI clients such as Claude Desktop, Cursor, and similar tools.

13. **Operational concerns**
   - The repo includes file locking, prompt budgeting, output-language control, and provider timeouts to keep long-running compile/query jobs predictable.
   - Watch mode and CI scripts indicate the compiler is meant to run repeatedly against a changing workspace, not just as a one-off generator.

14. **How to diagram it**
   - A useful architecture diagram would show four layers: raw sources in, compilation/retrieval core in the middle, generated wiki and exported artifacts out, and MCP/CLI as the control surface.
   - The main data stores are filesystem directories, while the main external dependencies are model providers and, optionally, session export formats and web/PDF/image sources.

Main entry points: [`src/commands/compile.ts`](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/compile.ts) (Compile), [`src/cli.ts`](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/cli.ts) (CLI), [`src/commands/query.ts`](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/query.ts) (Query), [`src/mcp/server.ts`](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/mcp/server.ts) (MCP server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control surface"]
  node_cli["CLI<br/>entrypoint<br/>[cli.ts]"]
  node_compile_cmd["Compile<br/>command<br/>[compile.ts]"]
  node_query_cmd["Query<br/>command<br/>[query.ts]"]
  node_review_cmds["Review flow<br/>commands<br/>[review-approve.ts]"]
  node_mcp["MCP server<br/>tooling<br/>[server.ts]"]
end

subgraph group_ingest["Ingestion"]
  node_ingest["Ingest core<br/>normalizer<br/>[shared.ts]"]
  node_session_ingest["Session ingest<br/>adapter<br/>[ingest-session.ts]"]
  node_file_web_media["File/Web/Media<br/>adapters<br/>[file.ts]"]
end

subgraph group_compile["Compilation core"]
  node_hasher_state["Hash state<br/>incremental-state<br/>[hasher.ts]"]
  node_deps_provenance["Deps + provenance<br/>analysis<br/>[deps.ts]"]
  node_prompt_budget["Prompt budget<br/>planner<br/>[prompt-budget.ts]"]
  node_page_render["Page render<br/>renderer<br/>[page-renderer.ts]"]
  node_index_links["Index + links<br/>postprocess<br/>[indexgen.ts]"]
  node_candidates["Candidates<br/>review-artifacts<br/>[candidates.ts]"]
end

subgraph group_output["Output and validation"]
  node_schema_lint["Schema + lint<br/>validation<br/>[index.ts]"]
  node_export["Exports<br/>serialization<br/>[collect.ts]"]
end

subgraph group_integrations["Providers and retrieval"]
  node_query_core["Retrieval<br/>search<br/>[retrieval.ts]"]
  node_embeddings["Embeddings<br/>vectorization<br/>[embeddings.ts]"]
  node_providers["Providers<br/>llm-abstraction<br/>[registry.ts]"]
end

subgraph group_state["Workspace state"]
  node_workspace[("Workspace<br/>filesystem boundary")]
end

node_cli -->|"dispatches"| node_compile_cmd
node_cli -->|"dispatches"| node_query_cmd
node_cli -->|"dispatches"| node_review_cmds
node_cli -->|"serves"| node_mcp
node_compile_cmd -->|"reads/writes"| node_workspace
node_query_cmd -->|"reads"| node_workspace
node_review_cmds -->|"updates"| node_workspace
node_mcp -->|"reads"| node_workspace
node_session_ingest -->|"normalizes to"| node_ingest
node_file_web_media -->|"normalizes to"| node_ingest
node_ingest -->|"writes sources"| node_workspace
node_compile_cmd -->|"starts with"| node_hasher_state
node_hasher_state -->|"feeds"| node_deps_provenance
node_deps_provenance -->|"packages for"| node_prompt_budget
node_prompt_budget -->|"drives"| node_page_render
node_page_render -->|"postprocesses"| node_index_links
node_index_links -->|"writes wiki"| node_workspace
node_compile_cmd -->|"emits"| node_candidates
node_candidates -->|"consumed by"| node_review_cmds
node_compile_cmd -->|"validates with"| node_schema_lint
node_schema_lint -->|"reads wiki"| node_workspace
node_compile_cmd -->|"can trigger"| node_export
node_query_cmd -->|"uses"| node_query_core
node_query_core -->|"optionally uses"| node_embeddings
node_query_core -->|"asks"| node_providers
node_compile_cmd -->|"asks"| node_providers
node_mcp -.->|"uses"| node_providers
node_query_cmd -.->|"saves results"| node_workspace

click node_cli "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/cli.ts"
click node_compile_cmd "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/compile.ts"
click node_query_cmd "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/query.ts"
click node_review_cmds "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/review-approve.ts"
click node_mcp "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/mcp/server.ts"
click node_ingest "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/ingest/shared.ts"
click node_session_ingest "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/ingest-session.ts"
click node_file_web_media "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/ingest/file.ts"
click node_hasher_state "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/hasher.ts"
click node_deps_provenance "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/deps.ts"
click node_prompt_budget "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/prompt-budget.ts"
click node_page_render "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/page-renderer.ts"
click node_index_links "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/indexgen.ts"
click node_candidates "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/candidates.ts"
click node_schema_lint "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/schema/index.ts"
click node_export "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/export/collect.ts"
click node_query_core "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/utils/retrieval.ts"
click node_embeddings "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/utils/embeddings.ts"
click node_providers "https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/adapters/registry.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_compile_cmd,node_query_cmd,node_review_cmds,node_mcp toneBlue
class node_ingest,node_session_ingest,node_file_web_media toneAmber
class node_hasher_state,node_deps_provenance,node_prompt_budget,node_page_render,node_index_links,node_candidates toneMint
class node_schema_lint,node_export toneRose
class node_query_core,node_embeddings,node_providers toneIndigo
class node_workspace toneTeal
```

## Components

### Control surface

CLI entrypoints and external agent interfaces

- [CLI](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/cli.ts): entrypoint · Main command router for compile, ingest, query, lint, review, export, and watch · `src/cli.ts`
- [Compile](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/compile.ts): command · Runs the main two-phase compile pipeline · `src/commands/compile.ts`
- [Query](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/query.ts): command · Retrieval-and-answer workflow over the compiled wiki · `src/commands/query.ts`
- [Review flow](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/review-approve.ts): commands · Approve, reject, list, and inspect generated candidates · `src/commands/review-approve.ts`
- [MCP server](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/mcp/server.ts): tooling · Model Context Protocol interface for agents and external clients · `src/mcp/server.ts`

### Ingestion

Normalizes raw inputs into workspace sources

- [Ingest core](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/ingest/shared.ts): normalizer · Normalizes raw inputs into source files with metadata · `src/ingest/shared.ts`
- [Session ingest](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/commands/ingest-session.ts): adapter · Imports Claude, Codex, and Cursor session exports · `src/commands/ingest-session.ts`
- [File/Web/Media](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/ingest/file.ts): adapters · File, web, PDF, image, and transcript ingestion paths · `src/ingest/file.ts`

### Compilation core

Incremental source-to-wiki pipeline and provenance tracking

- [Hash state](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/hasher.ts): incremental-state · Detects changes and persists source state for skipped recompiles · `src/compiler/hasher.ts`
- [Deps + provenance](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/deps.ts): analysis · Resolves dependencies, tracks claims, confidence, and contradictions · `src/compiler/deps.ts`
- [Prompt budget](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/prompt-budget.ts): planner · Controls context size and prompt construction for synthesis · `src/compiler/prompt-budget.ts`
- [Page render](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/page-renderer.ts): renderer · Synthesizes Obsidian-style markdown pages with frontmatter · `src/compiler/page-renderer.ts`
- [Index + links](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/indexgen.ts): postprocess · Resolves wikilinks, generates index.md, and handles orphan logic · `src/compiler/indexgen.ts`
- [Candidates](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/compiler/candidates.ts): review-artifacts · Emits reviewable candidate records instead of writing directly · `src/compiler/candidates.ts`

### Output and validation

Generated wiki, exports, schema checks, and linting

- [Schema + lint](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/schema/index.ts): validation · Enforces page kinds, link structure, orphans, confidence, and citations · `src/schema/index.ts`
- [Exports](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/export/collect.ts): serialization · Produces llms.txt, JSON, JSON-LD, GraphML, and Marp · `src/export/collect.ts`

### Providers and retrieval

Model backends, embeddings, and query retrieval

- [Retrieval](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/utils/retrieval.ts): search · Semantic retrieval and fallback answering over wiki content · `src/utils/retrieval.ts`
- [Embeddings](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/utils/embeddings.ts): vectorization · Embedding support used by retrieval and semantic search · `src/utils/embeddings.ts`
- [Providers](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/src/adapters/registry.ts): llm-abstraction · Common interface over Anthropic, OpenAI-compatible, Ollama, Copilot, and others · `src/adapters/registry.ts`

### Workspace state

Filesystem-backed sources, wiki output, and .llmwiki state

- **Workspace**: filesystem boundary · Disk-backed sources, wiki output, and .llmwiki state

## Connections

- CLI → Compile: dispatches · Routes CLI compile requests
- CLI → Query: dispatches · Routes query requests
- CLI → Review flow: dispatches · Routes review lifecycle commands
- CLI → MCP server: serves · Exposes the MCP runtime as an external control surface
- Compile → Workspace: reads/writes · Consumes sources and writes wiki state
- Query → Workspace: reads · Queries compiled wiki and saved answers
- Review flow → Workspace: updates · Approves or rejects candidate records and refreshes output
- MCP server → Workspace: reads · Provides tool and resource access to workspace state
- Session ingest → Ingest core: normalizes to · Transforms session exports into source material
- File/Web/Media → Ingest core: normalizes to · Converts raw files and media into source material
- Ingest core → Workspace: writes sources · Stores normalized inputs under sources/
- Compile → Hash state: starts with · Uses hash comparison to skip unchanged sources
- Hash state → Deps + provenance: feeds · Only changed or relevant sources continue downstream
- Deps + provenance → Prompt budget: packages for · Prepares bounded synthesis context
- Prompt budget → Page render: drives · Controls page generation from discovered concepts
- Page render → Index + links: postprocesses · Adds links, index, and orphan handling
- Index + links → Workspace: writes wiki · Materializes compiled markdown into wiki/
- Compile → Candidates: emits · Review mode writes candidate records instead of publishing directly
- Candidates → Review flow: consumed by · Review commands inspect and approve pending output
- Compile → Schema + lint: validates with · Checks generated pages against schema and lint rules
- Schema + lint → Workspace: reads wiki · Inspects compiled pages and state for violations
- Compile → Exports: can trigger · Downstream export reads the wiki as source of truth
- Query → Retrieval: uses · Answers questions through retrieval
- Retrieval → Embeddings: optionally uses · Uses embeddings when available, falls back otherwise
- Retrieval → Providers: asks · Retrieval and answering depend on model backends
- Compile → Providers: asks · Synthesis and extraction depend on provider abstraction
- MCP server → Providers: uses · Tool-backed operations can reach model providers through the same abstraction
- Query → Workspace: saves results · Query answers can be persisted back as future source material