# asun28/llm-wiki-flywheel architecture

> Architecture diagram of the GitHub repository asun28/llm-wiki-flywheel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/asun28/llm-wiki-flywheel
- Repository: https://github.com/asun28/llm-wiki-flywheel

## Overview

1. **What this repository is**
   This is a Python 3.12+ knowledge-base compiler for Markdown-first “wiki” generation. The core idea is: source documents go into `raw/`, the system extracts structured knowledge into generated wiki pages under `wiki/`, and then it supports query, lint, graph, review, and publish workflows around that generated knowledge.

2. **Primary data boundary: raw vs generated vs human-authored**
   The architecture centers on three content zones:
   - `raw/`: immutable source inputs such as articles, papers, videos, repos, conversations, datasets
   - `wiki/`: generated markdown pages with YAML frontmatter, organized into entities, concepts, summaries, synthesis, comparisons, and indexes
   - `research/` and `docs/`: human-authored analysis, plans, decisions, and reference material
   This boundary is important because most application logic is about transforming and validating content as it crosses from raw sources into the wiki.

3. **Main runtime surfaces**
   There are two user-facing entry points:
   - a **CLI** in `src/kb/cli.py` for operations like ingest, compile, query, lint, evolve, and rebuild-indexes
   - an **MCP server** in `src/kb/mcp/` and `src/kb/mcp_server.py` exposing the same capabilities as tools for Claude Code and other MCP clients
   The CLI and MCP layers are thin orchestration shells over the same underlying modules.

4. **Ingestion pipeline**
   `src/kb/ingest/` handles source-to-knowledge transformation. It uses template-driven extractors from `templates/` to pull out entities, concepts, claims, evidence, contradictions, and source metadata. The pipeline creates or updates wiki pages, persists provenance, and can trigger retroactive wikilink injection into existing pages that mention newly discovered topics.

5. **Compile and linking subsystem**
   `src/kb/compile/` is the batch compiler. It scans raw sources, detects changes via hashing, increments only what changed, and coordinates:
   - page compilation
   - link resolution and wikilink injection
   - publishing artifacts
   - manifest tracking for idempotence and restart safety
   This is the main “compile, don’t retrieve” engine that turns source changes into wiki updates.

6. **Query subsystem**
   `src/kb/query/` implements retrieval and answer assembly. The search stack combines:
   - BM25 keyword ranking
   - optional vector embeddings / hybrid retrieval
   - deduplication and query rewriting
   - citation generation
   - output formatting for Markdown, HTML, Marp, Jupyter, and chart-like outputs
   Querying is not just search; it assembles grounded context and synthesizes responses from wiki content with source citations.

7. **Knowledge graph subsystem**
   `src/kb/graph/` builds the wikilink graph and computes graph metrics such as PageRank and other centrality measures. It also exports graph representations for visualization and downstream use. This graph is used by both query ranking and quality workflows, so the graph is a core internal index rather than a separate feature.

8. **Quality, lint, and drift control**
   `src/kb/lint/` is a broad health-check subsystem with rule-based checks plus semantic validation. It covers dead links, orphan pages, duplicate slugs, frontmatter correctness, inline callouts, staleness, contradictions, and source fidelity. Related modules in `src/kb/feedback/` track trust/reliability from user feedback, and `src/kb/review/` supports structured page review and refinement.

9. **Gap analysis and self-healing loop**
   `src/kb/evolve/` analyzes the wiki for missing coverage, disconnected topics, and likely connection opportunities. Architecturally, this closes the loop: ingest creates structure, lint finds defects, evolve finds missing knowledge, and review/refine can correct or strengthen pages.

10. **MCP and agent integration boundary**
   The `src/kb/mcp/` package exposes fine-grained tools for query, ingest, capture, lint, evolve, graph visualization, review, refinement, and stats. This boundary is significant because it lets an external agent orchestrate the system without direct API integration, making the repo both a local CLI tool and an LLM-native service.

11. **Cross-cutting utilities and models**
   `src/kb/models/` defines page and frontmatter schemas, while `src/kb/utils/` contains shared concerns: hashing, path safety, markdown/text helpers, I/O, LLM subprocess/API abstraction, and sanitized logging. `src/kb/config.py` centralizes project-root resolution, path layout, and model-tier settings, so many modules depend on it as the configuration anchor.

12. **Persistence and file-system-centric design**
   This is a file-based system, not a database-centric one. The wiki is stored as markdown files; the raw corpus is file-backed; manifests, logs, and verdict history live alongside the project; and generated outputs are written to repository paths or `.data/`-style internal storage. The architecture emphasizes atomic writes, locks, idempotent rebuilds, and path validation because the filesystem is the primary data store.

13. **Publishing and external consumption**
   `src/kb/compile/publish.py` and related code produce machine-readable publication artifacts such as `llms.txt`, `llms-full.txt`, `graph.jsonld`, sitemap, and related page bundles. This is the outward-facing delivery layer for downstream consumers, distinct from the internal wiki authoring model.

14. **Supporting assets and project hygiene**
   The repo includes `demo/` as a worked example of a compiled wiki, `docs/` for architecture/reference/research, `tests/` for extensive behavioral coverage, and GitHub Actions CI for validation. These are not core runtime components, but they show the project is engineered around deterministic file outputs, regression-heavy safeguards, and documented workflows.

Main entry points: [`demo/wiki`](https://github.com/asun28/llm-wiki-flywheel/tree/main/demo/wiki) (demo/wiki/), [`src/kb/compile/compiler.py`](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/compile/compiler.py) (Compiler), [`src/kb/query/engine.py`](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/query/engine.py) (Query), [`src/kb/ingest/pipeline.py`](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/ingest/pipeline.py) (Ingest)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_content["Content zones"]
  node_demo_raw[("demo/raw/<br/>source corpus")]
  node_demo_wiki["demo/wiki/<br/>generated wiki"]
  node_research_docs["research/docs<br/>human notes"]
end

subgraph group_runtime["Runtime entrypoints"]
  node_cli["CLI<br/>command runner<br/>[cli.py]"]
  node_mcp_server["MCP server<br/>tool server<br/>[mcp_server.py]"]
end

subgraph group_core["Core engine"]
  node_ingest_pipeline["Ingest<br/>source-to-knowledge<br/>[pipeline.py]"]
  node_extractors["Extractors<br/>template parsers<br/>[extractors.py]"]
  node_compiler["Compiler<br/>[compiler.py]"]
  node_linker["Linker<br/>wikilink resolver<br/>[linker.py]"]
  node_publisher["Publish<br/>artifact exporter<br/>[publish.py]"]
  node_query_engine["Query<br/>retrieval assembly<br/>[engine.py]"]
  node_retrieval_stack["Retrieval<br/>hybrid search<br/>[hybrid.py]"]
  node_formats["Formats<br/>renderers"]
  node_knowledge_graph(("Graph<br/>[builder.py]"))
end

subgraph group_quality["Quality loop"]
  node_lint_runner["Lint<br/>health checks<br/>[runner.py]"]
  node_evolve["Evolve<br/>gap analysis<br/>[analyzer.py]"]
  node_feedback["Feedback<br/>trust signals<br/>[reliability.py]"]
  node_review["Review<br/>refinement loop<br/>[refiner.py]"]
end

subgraph group_support["Support"]
  node_models_config["Models/config<br/>schemas + config<br/>[frontmatter.py]"]
  node_utils["Utils<br/>shared helpers"]
end

node_cli -->|"ingest"| node_ingest_pipeline
node_cli -->|"compile"| node_compiler
node_cli -->|"query"| node_query_engine
node_cli -->|"lint"| node_lint_runner
node_cli -->|"evolve"| node_evolve
node_mcp_server -->|"tools"| node_ingest_pipeline
node_mcp_server -->|"tools"| node_query_engine
node_mcp_server -->|"tools"| node_lint_runner
node_ingest_pipeline -->|"extract"| node_extractors
node_ingest_pipeline -->|"write"| node_demo_wiki
node_ingest_pipeline -->|"schema"| node_models_config
node_compiler -->|"scan"| node_demo_raw
node_compiler -->|"compile"| node_demo_wiki
node_compiler -->|"link"| node_linker
node_compiler -->|"publish"| node_publisher
node_linker -->|"inject"| node_demo_wiki
node_query_engine -->|"retrieve"| node_retrieval_stack
node_query_engine -->|"render"| node_formats
node_query_engine -->|"answer from"| node_demo_wiki
node_query_engine -->|"rank with"| node_knowledge_graph
node_knowledge_graph -->|"derive from"| node_demo_wiki
node_lint_runner -->|"check"| node_demo_wiki
node_lint_runner -->|"validate"| node_models_config
node_lint_runner -->|"record"| node_feedback
node_evolve -->|"analyze"| node_demo_wiki
node_review -->|"refine"| node_demo_wiki
node_publisher -->|"export from"| node_demo_wiki
node_utils -.->|"support"| node_ingest_pipeline
node_utils -.->|"support"| node_compiler
node_utils -.->|"support"| node_query_engine
node_utils -.->|"support"| node_lint_runner
node_models_config -.->|"anchor"| node_compiler

click node_demo_raw "https://github.com/asun28/llm-wiki-flywheel/tree/main/demo/raw"
click node_demo_wiki "https://github.com/asun28/llm-wiki-flywheel/tree/main/demo/wiki"
click node_research_docs "https://github.com/asun28/llm-wiki-flywheel/tree/main/research"
click node_cli "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/cli.py"
click node_mcp_server "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/mcp_server.py"
click node_ingest_pipeline "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/ingest/pipeline.py"
click node_extractors "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/ingest/extractors.py"
click node_compiler "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/compile/compiler.py"
click node_linker "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/compile/linker.py"
click node_publisher "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/compile/publish.py"
click node_query_engine "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/query/engine.py"
click node_retrieval_stack "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/query/hybrid.py"
click node_formats "https://github.com/asun28/llm-wiki-flywheel/tree/main/src/kb/query/formats"
click node_knowledge_graph "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/graph/builder.py"
click node_lint_runner "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/lint/runner.py"
click node_evolve "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/evolve/analyzer.py"
click node_feedback "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/feedback/reliability.py"
click node_review "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/review/refiner.py"
click node_models_config "https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/models/frontmatter.py"
click node_utils "https://github.com/asun28/llm-wiki-flywheel/tree/main/src/kb/utils"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo_raw,node_demo_wiki,node_research_docs toneBlue
class node_cli,node_mcp_server toneAmber
class node_ingest_pipeline,node_extractors,node_compiler,node_linker,node_publisher,node_query_engine,node_retrieval_stack,node_formats,node_knowledge_graph toneMint
class node_lint_runner,node_evolve,node_feedback,node_review toneRose
class node_models_config,node_utils toneIndigo
```

## Components

### Content zones

Repository data and authored knowledge zones

- [demo/raw/](https://github.com/asun28/llm-wiki-flywheel/tree/main/demo/raw): source corpus · Example source corpus used by the demo wiki · `demo/raw`
- [demo/wiki/](https://github.com/asun28/llm-wiki-flywheel/tree/main/demo/wiki): generated wiki · Example generated markdown knowledge base · `demo/wiki`
- [research/docs](https://github.com/asun28/llm-wiki-flywheel/tree/main/research): human notes · Human-authored analysis, plans, and reference material · `research`

### Runtime entrypoints

User-facing orchestration surfaces

- [CLI](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/cli.py): command runner · Command-line entrypoint for ingest, compile, query, lint, evolve, and rebuild · `src/kb/cli.py`
- [MCP server](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/mcp_server.py): tool server · MCP entrypoint exposing repository capabilities to agents · `src/kb/mcp_server.py`

### Core engine

Primary transformation and indexing subsystems

- [Ingest](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/ingest/pipeline.py): source-to-knowledge · Transforms raw sources into structured wiki content · `src/kb/ingest/pipeline.py`
- [Extractors](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/ingest/extractors.py): template parsers · Template-driven extraction of entities, claims, evidence, and metadata · `src/kb/ingest/extractors.py`
- [Compiler](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/compile/compiler.py): batch compiler · Detects changes, compiles pages incrementally, and manages manifests · `src/kb/compile/compiler.py`
- [Linker](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/compile/linker.py): wikilink resolver · Resolves links and injects wikilinks across pages · `src/kb/compile/linker.py`
- [Publish](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/compile/publish.py): artifact exporter · Produces publication bundles such as llms.txt, graph exports, and sitemaps · `src/kb/compile/publish.py`
- [Query](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/query/engine.py): retrieval assembly · Searches wiki content and assembles grounded answers with citations · `src/kb/query/engine.py`
- [Retrieval](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/query/hybrid.py): hybrid search · BM25, embeddings, deduplication, and query rewriting · `src/kb/query/hybrid.py`
- [Formats](https://github.com/asun28/llm-wiki-flywheel/tree/main/src/kb/query/formats): renderers · Markdown, HTML, Marp, Jupyter, and chart output adapters · `src/kb/query/formats`
- [Graph](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/graph/builder.py): wikilink graph · Builds and scores the wikilink graph for ranking and export · `src/kb/graph/builder.py`

### Quality loop

Validation, review, and drift control

- [Lint](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/lint/runner.py): health checks · Runs structural and semantic validation across the wiki · `src/kb/lint/runner.py`
- [Evolve](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/evolve/analyzer.py): gap analysis · Finds missing coverage and connection opportunities · `src/kb/evolve/analyzer.py`
- [Feedback](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/feedback/reliability.py): trust signals · Stores reliability and user feedback signals · `src/kb/feedback/reliability.py`
- [Review](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/review/refiner.py): refinement loop · Structured page review and refinement support · `src/kb/review/refiner.py`

### Support

Shared models, config, and utilities

- [Models/config](https://github.com/asun28/llm-wiki-flywheel/blob/main/src/kb/models/frontmatter.py): schemas + config · Page schemas, frontmatter models, and project path settings · `src/kb/models/frontmatter.py`
- [Utils](https://github.com/asun28/llm-wiki-flywheel/tree/main/src/kb/utils): shared helpers · Hashing, paths, markdown, IO, LLM, and sanitization helpers · `src/kb/utils`

## Connections

- CLI → Ingest: ingest · CLI invokes source ingestion
- CLI → Compiler: compile · CLI runs incremental compilation
- CLI → Query: query · CLI exposes retrieval and answer assembly
- CLI → Lint: lint · CLI runs validation checks
- CLI → Evolve: evolve · CLI runs gap analysis
- MCP server → Ingest: tools · MCP exposes ingestion tools
- MCP server → Query: tools · MCP exposes query tools
- MCP server → Lint: tools · MCP exposes quality tools
- Ingest → Extractors: extract · Pipeline uses templates to derive structured knowledge
- Ingest → demo/wiki/: write · Creates or updates wiki pages
- Ingest → Models/config: schema · Uses page and frontmatter models
- Compiler → demo/raw/: scan · Scans source corpus for changes
- Compiler → demo/wiki/: compile · Updates generated wiki content
- Compiler → Linker: link · Coordinates link resolution and injection
- Compiler → Publish: publish · Builds outward-facing artifacts
- Linker → demo/wiki/: inject · Mutates pages with wikilinks
- Query → Retrieval: retrieve · Uses search and ranking components
- Query → Formats: render · Formats results for multiple outputs
- Query → demo/wiki/: answer from · Grounds responses in wiki content
- Query → Graph: rank with · Uses graph signals for relevance
- Graph → demo/wiki/: derive from · Builds graph from wikilinks
- Lint → demo/wiki/: check · Validates generated pages and links
- Lint → Models/config: validate · Checks schema and frontmatter rules
- Lint → Feedback: record · Feeds trust and reliability signals
- Evolve → demo/wiki/: analyze · Finds missing coverage and weak connections
- Review → demo/wiki/: refine · Supports page review and rewriting
- Publish → demo/wiki/: export from · Packages generated pages and indexes
- Utils → Ingest: support · Shared helpers used across ingestion
- Utils → Compiler: support · Shared helpers used across compilation
- Utils → Query: support · Shared helpers used across querying
- Utils → Lint: support · Shared helpers used across validation
- Models/config → Compiler: anchor · Central configuration and schema anchor