How atomicstrata/llm-wiki-compiler works
1. Overall shape
- This is a Node.js CLI/compiler for building a persistent markdown wiki from raw sources.
- The core architectural boundary is a workspace on disk with three important areas: sources/ for inputs, wiki/ for compiled output, and .llmwiki/ for state, schema, review candidates, and locks.
2. Primary pipeline
- The main flow is: ingest raw content → detect changes by hashing → extract concepts and dependencies → generate or update wiki pages → resolve wikilinks → regenerate index.md.
- The compile path is intentionally two-phase: first discover concepts from all sources, then synthesize pages. That avoids order dependence and lets shared concepts merge across sources.
3. Source ingestion subsystem
- src/ingest handles multiple source types: files, web URLs, PDFs, images, transcripts, and session exports from Claude/Codex/Cursor.
- Ingestion normalizes everything into source files under sources/, with metadata such as truncation status and original size preserved in frontmatter or source state.
4. Incremental compilation subsystem
- src/compiler is the central transformation layer.
- It includes hashing, dependency resolution, provenance tracking, prompt budgeting, source-state tracking, page rendering, orphan detection, and wiki/index generation.
- The compiler’s main boundary is “source state vs generated output”: only changed sources are reprocessed, while unchanged ones are skipped via hash-based comparison.
5. Wiki page model
- Compiled output is Obsidian-compatible markdown with YAML frontmatter.
- Pages are grouped into concept pages and query-result pages, with generated links and metadata such as sources, confidence, provenanceState, and contradictedBy.
- The index and link resolution are part of the compile output, not a separate runtime service.
6. Provenance and trust metadata
- The compiler carries source attribution into page content using paragraph markers and claim-level line citations.
- Provenance-related modules support line-range validation, merged-source metadata reconciliation, confidence scores, and contradiction tracking.
- This makes source traceability a first-class architectural concern, not an afterthought.
7. Query subsystem
- llmwiki query is a separate retrieval-and-answer pipeline over the compiled wiki.
- It uses semantic retrieval and embeddings when available, falls back when necessary, and can save answers back into the wiki as new pages.
- That “save” path closes the loop: query output becomes future source material, so the wiki compounds over time.
8. Provider abstraction
- src/providers and src/adapters isolate model/vendor differences behind a common interface.
- The repo supports Anthropic, OpenAI-compatible endpoints, Ollama, Copilot, and other adapter flows, with env-based config and provider-specific fallbacks.
- This means the compile/query pipelines depend on a provider boundary, not a specific vendor SDK.
9. Review workflow boundary
- compile --review does not write directly into wiki/; it emits candidate records into .llmwiki/candidates/.
- Separate review commands approve, reject, or inspect candidates, and approval refreshes the wiki/index while holding a lock to serialize changes.
- This creates a human-in-the-loop gate between generation and publication.
10. Schema and linting
- The optional .llmwiki/schema.json layer defines page kinds and cross-link expectations beyond plain concepts.
- src/linter enforces structural and semantic rules: broken links, orphans, low confidence, contradictions, uncited paragraphs, and schema violations.
- Schema and linting are validation boundaries around the generated knowledge graph.
11. Export and interoperability
- src/export turns the compiled wiki into portable formats: llms.txt, JSON, JSON-LD, GraphML, and Marp slides.
- This subsystem is downstream of compilation and reads the wiki as the source of truth rather than re-running generation logic.
12. MCP server boundary
- src/mcp exposes the system as tools/resources for agents through Model Context Protocol.
- The server provides read-only access, ingest, compile, query, lint, and status operations, plus resources like the index, individual pages, and compilation state.
- This is the external integration boundary for AI clients such as Claude Desktop, Cursor, and similar tools.
13. Operational concerns
- The repo includes file locking, prompt budgeting, output-language control, and provider timeouts to keep long-running compile/query jobs predictable.
- Watch mode and CI scripts indicate the compiler is meant to run repeatedly against a changing workspace, not just as a one-off generator.
14. How to diagram it
- A useful architecture diagram would show four layers: raw sources in, compilation/retrieval core in the middle, generated wiki and exported artifacts out, and MCP/CLI as the control surface.
- The main data stores are filesystem directories, while the main external dependencies are model providers and, optionally, session export formats and web/PDF/image sources.
Main entry points: compile.ts Compile · cli.ts CLI · query.ts Query · server.ts MCP server
Components 20
Control surface
CLI entrypoints and external agent interfaces
- CLI · Main command router for compile, ingest, query, lint, review, export, and watch
src/cli.ts - Compile · Runs the main two-phase compile pipeline
src/commands/compile.ts - Query · Retrieval-and-answer workflow over the compiled wiki
src/commands/query.ts - Review flow · Approve, reject, list, and inspect generated candidates
src/commands/review-approve.ts - MCP server · Model Context Protocol interface for agents and external clients
src/mcp/server.ts
Ingestion
Normalizes raw inputs into workspace sources
- Ingest core · Normalizes raw inputs into source files with metadata
src/ingest/shared.ts - Session ingest · Imports Claude, Codex, and Cursor session exports
src/commands/ingest-session.ts - File/Web/Media · File, web, PDF, image, and transcript ingestion paths
src/ingest/file.ts
Compilation core
Incremental source-to-wiki pipeline and provenance tracking
- Hash state · Detects changes and persists source state for skipped recompiles
src/compiler/hasher.ts - Deps + provenance · Resolves dependencies, tracks claims, confidence, and contradictions
src/compiler/deps.ts - Prompt budget · Controls context size and prompt construction for synthesis
src/compiler/prompt-budget.ts - Page render · Synthesizes Obsidian-style markdown pages with frontmatter
src/compiler/page-renderer.ts - Index + links · Resolves wikilinks, generates index.md, and handles orphan logic
src/compiler/indexgen.ts - Candidates · Emits reviewable candidate records instead of writing directly
src/compiler/candidates.ts
Output and validation
Generated wiki, exports, schema checks, and linting
- Schema + lint · Enforces page kinds, link structure, orphans, confidence, and citations
src/schema/index.ts - Exports · Produces llms.txt, JSON, JSON-LD, GraphML, and Marp
src/export/collect.ts
Providers and retrieval
Model backends, embeddings, and query retrieval
- Retrieval · Semantic retrieval and fallback answering over wiki content
src/utils/retrieval.ts - Embeddings · Embedding support used by retrieval and semantic search
src/utils/embeddings.ts - Providers · Common interface over Anthropic, OpenAI-compatible, Ollama, Copilot, and others
src/adapters/registry.ts
Workspace state
Filesystem-backed sources, wiki output, and .llmwiki state
- Workspace · Disk-backed sources, wiki output, and .llmwiki state
Connections 28
- CLI → Compile: dispatches · Routes CLI compile requests
- CLI → Query: dispatches · Routes query requests
- CLI → Review flow: dispatches · Routes review lifecycle commands
- CLI → MCP server: serves · Exposes the MCP runtime as an external control surface
- Compile → Workspace: reads/writes · Consumes sources and writes wiki state
- Query → Workspace: reads · Queries compiled wiki and saved answers
- Review flow → Workspace: updates · Approves or rejects candidate records and refreshes output
- MCP server → Workspace: reads · Provides tool and resource access to workspace state
- Session ingest → Ingest core: normalizes to · Transforms session exports into source material
- File/Web/Media → Ingest core: normalizes to · Converts raw files and media into source material
- Ingest core → Workspace: writes sources · Stores normalized inputs under sources/
- Compile → Hash state: starts with · Uses hash comparison to skip unchanged sources
- Hash state → Deps + provenance: feeds · Only changed or relevant sources continue downstream
- Deps + provenance → Prompt budget: packages for · Prepares bounded synthesis context
- Prompt budget → Page render: drives · Controls page generation from discovered concepts
- Page render → Index + links: postprocesses · Adds links, index, and orphan handling
- Index + links → Workspace: writes wiki · Materializes compiled markdown into wiki/
- Compile → Candidates: emits · Review mode writes candidate records instead of publishing directly
- Candidates → Review flow: consumed by · Review commands inspect and approve pending output
- Compile → Schema + lint: validates with · Checks generated pages against schema and lint rules
- Schema + lint → Workspace: reads wiki · Inspects compiled pages and state for violations
- Compile → Exports: can trigger · Downstream export reads the wiki as source of truth
- Query → Retrieval: uses · Answers questions through retrieval
- Retrieval → Embeddings: optionally uses · Uses embeddings when available, falls back otherwise
- Retrieval → Providers: asks · Retrieval and answering depend on model backends
- Compile → Providers: asks · Synthesis and extraction depend on provider abstraction
- MCP server → Providers: uses · Tool-backed operations can reach model providers through the same abstraction
- Query → Workspace: saves results · Query answers can be persisted back as future source material