asun28/llm-wiki-flywheel

1. What this repository is

This is a Python 3.12+ knowledge-base compiler for Markdown-first “wiki” generation. The core idea is: source documents go into raw/, the system extracts structured knowledge into generated wiki pages under wiki/, and then it supports query, lint, graph, review, and publish workflows around that generated knowledge.

2. Primary data boundary: raw vs generated vs human-authored

The architecture centers on three content zones:

- raw/: immutable source inputs such as articles, papers, videos, repos, conversations, datasets

- wiki/: generated markdown pages with YAML frontmatter, organized into entities, concepts, summaries, synthesis, comparisons, and indexes

- research/ and docs/: human-authored analysis, plans, decisions, and reference material

This boundary is important because most application logic is about transforming and validating content as it crosses from raw sources into the wiki.

3. Main runtime surfaces

There are two user-facing entry points:

- a CLI in src/kb/cli.py for operations like ingest, compile, query, lint, evolve, and rebuild-indexes

- an MCP server in src/kb/mcp/ and src/kb/mcp_server.py exposing the same capabilities as tools for Claude Code and other MCP clients

The CLI and MCP layers are thin orchestration shells over the same underlying modules.

4. Ingestion pipeline

src/kb/ingest/ handles source-to-knowledge transformation. It uses template-driven extractors from templates/ to pull out entities, concepts, claims, evidence, contradictions, and source metadata. The pipeline creates or updates wiki pages, persists provenance, and can trigger retroactive wikilink injection into existing pages that mention newly discovered topics.

5. Compile and linking subsystem

src/kb/compile/ is the batch compiler. It scans raw sources, detects changes via hashing, increments only what changed, and coordinates:

- page compilation

- link resolution and wikilink injection

- publishing artifacts

- manifest tracking for idempotence and restart safety

This is the main “compile, don’t retrieve” engine that turns source changes into wiki updates.

6. Query subsystem

src/kb/query/ implements retrieval and answer assembly. The search stack combines:

- BM25 keyword ranking

- optional vector embeddings / hybrid retrieval

- deduplication and query rewriting

- citation generation

- output formatting for Markdown, HTML, Marp, Jupyter, and chart-like outputs

Querying is not just search; it assembles grounded context and synthesizes responses from wiki content with source citations.

7. Knowledge graph subsystem

src/kb/graph/ builds the wikilink graph and computes graph metrics such as PageRank and other centrality measures. It also exports graph representations for visualization and downstream use. This graph is used by both query ranking and quality workflows, so the graph is a core internal index rather than a separate feature.

8. Quality, lint, and drift control

src/kb/lint/ is a broad health-check subsystem with rule-based checks plus semantic validation. It covers dead links, orphan pages, duplicate slugs, frontmatter correctness, inline callouts, staleness, contradictions, and source fidelity. Related modules in src/kb/feedback/ track trust/reliability from user feedback, and src/kb/review/ supports structured page review and refinement.

9. Gap analysis and self-healing loop

src/kb/evolve/ analyzes the wiki for missing coverage, disconnected topics, and likely connection opportunities. Architecturally, this closes the loop: ingest creates structure, lint finds defects, evolve finds missing knowledge, and review/refine can correct or strengthen pages.

10. MCP and agent integration boundary

The src/kb/mcp/ package exposes fine-grained tools for query, ingest, capture, lint, evolve, graph visualization, review, refinement, and stats. This boundary is significant because it lets an external agent orchestrate the system without direct API integration, making the repo both a local CLI tool and an LLM-native service.

11. Cross-cutting utilities and models

src/kb/models/ defines page and frontmatter schemas, while src/kb/utils/ contains shared concerns: hashing, path safety, markdown/text helpers, I/O, LLM subprocess/API abstraction, and sanitized logging. src/kb/config.py centralizes project-root resolution, path layout, and model-tier settings, so many modules depend on it as the configuration anchor.

12. Persistence and file-system-centric design

This is a file-based system, not a database-centric one. The wiki is stored as markdown files; the raw corpus is file-backed; manifests, logs, and verdict history live alongside the project; and generated outputs are written to repository paths or .data/-style internal storage. The architecture emphasizes atomic writes, locks, idempotent rebuilds, and path validation because the filesystem is the primary data store.

13. Publishing and external consumption

src/kb/compile/publish.py and related code produce machine-readable publication artifacts such as llms.txt, llms-full.txt, graph.jsonld, sitemap, and related page bundles. This is the outward-facing delivery layer for downstream consumers, distinct from the internal wiki authoring model.

14. Supporting assets and project hygiene

The repo includes demo/ as a worked example of a compiled wiki, docs/ for architecture/reference/research, tests/ for extensive behavioral coverage, and GitHub Actions CI for validation. These are not core runtime components, but they show the project is engineered around deterministic file outputs, regression-heavy safeguards, and documented workflows.