niharshrotri/llm-wiki

1. Repository shape

- This is a local-first knowledge-base compiler: it takes documents from a raw source area, converts them into an Obsidian-style wiki, and then exposes that wiki through CLI, search/query, linting, and a web UI.

- The central architectural idea is that the wiki is a persistent derived artifact, not a hand-edited source of truth.

2. Core data boundary: raw vs. generated vs. internal state

- raw/ is the immutable input boundary: original PDFs, Markdown, HTML, DOCX, and text files are read here and never modified by the system.

- wiki/ is the generated output boundary: markdown pages with YAML frontmatter and [[wikilinks]], organized into page-type folders like sources/, entities/, concepts/, and synthesis/.

- .wiki/ is internal runtime state: database, ingest history, and search/index artifacts. It is implementation detail, not user content.

3. CLI as the primary control plane

- src/llm_wiki/cli.py is the main entrypoint and command router.

- It exposes project lifecycle commands such as init, add, ingest, query, lint, reindex, status, and serve.

- Most user workflows start here, so the CLI is the orchestration layer over the rest of the system.

4. Project scaffolding and conventions

- scaffold.py and src/llm_wiki/templates/ build the initial wiki vault structure and seed files like index.md, log.md, and Obsidian graph configs.

- schema/AGENTS.md is a key architectural boundary: it defines the rules the LLM follows for formatting, merging, linking, and citing pages.

- In practice, the schema file acts like a policy/config contract between the human-maintained repo and the model-driven content generation.

5. Source ingestion pipeline

- ingest_raw.py handles importing files into the raw/source registry and registering them for processing.

- The ingest flow is multi-stage and source-centric: parse document → extract structure and entities/concepts → draft or merge wiki pages → write source provenance → update index/log/search state.

- jobs.py supports long-running ingest work, especially for the web UI, so ingest runs are tracked as durable jobs rather than ephemeral requests.

6. Document parsing subsystem

- src/llm_wiki/parsers/ is the file-format adapter layer.

- Separate parser implementations for pdf.py, docx.py, html.py, and text.py normalize heterogeneous inputs into a common text representation.

- parsers/base.py defines the shared interface, which keeps format-specific extraction isolated from the rest of the ingest logic.

7. LLM-driven compilation layer

- ingest_llm.py, llm.py, prompts.py, page_writer.py, and intent.py form the model orchestration layer.

- The LLM is used for structured extraction, page drafting/merging, contradiction handling, and synthesis generation.

- intent.py short-circuits trivial chatty queries so the system does not waste retrieval/LLM work on non-knowledge questions.

- page_writer.py is the boundary where extracted knowledge becomes durable wiki markdown.

8. Search and query path

- search.py and query.py implement query-time retrieval over the compiled wiki.

- The README describes a hybrid search stack: BM25 + vector similarity + LLM reranking, delegated to QMD.

- Query flow is: classify intent → retrieve top pages from the wiki corpus → hydrate page contents → synthesize a cited markdown answer → optionally save it back as a synthesis/ page.

- This is distinct from classic RAG: the system queries the compiled wiki first, not the raw documents.

9. Quality control and maintenance

- lint.py runs structural health checks across the wiki: broken links, malformed frontmatter, orphans, noise, and optional deep contradiction detection.

- Lint can auto-fix many formatting/link issues, and it also triggers index refresh so corrected content becomes searchable immediately.

- This makes lint part of the operational lifecycle, not just a static validation tool.

10. Persistent storage and metadata

- db.py is the persistence layer for source records, ingest status, and job/history metadata.

- The internal SQLite-backed state is important for resumability and for the web UI’s durable job tracking.

- slugify.py supports stable page naming and identifier generation, which matters because page identity is used for links, merges, and provenance tracking.

11. Web application boundary

- src/llm_wiki/webapp/main.py is a separate HTTP interface, organized as a FastAPI-style application with route modules under webapp/routes/.

- Routes are split by concern: dashboard, sources, ingest, jobs, query, lint, and graph.

- The web layer is primarily a presentation/orchestration shell over the same core ingest/query/lint subsystems used by the CLI, not a second implementation.

- webapp/templates/ and webapp/static/ provide the server-rendered UI and styling.

12. External runtime dependencies that shape the architecture

- Ollama hosts the local LLM used for reading, writing, and synthesis.

- QMD is the search backend and handles hybrid retrieval plus reranking.

- Obsidian is the consumer of the generated wiki folder; its backlink/graph model strongly influences the markdown structure and wikilink conventions.

- The stack is intentionally local-only: there are no cloud APIs in the core architecture.

13. What a diagram should emphasize

- The main flow is: raw documents → parsers → LLM compilation → wiki markdown → search/query/lint/web UI.

- The important boundaries are:

- immutable input vs generated knowledge base,

- model orchestration vs page writing,

- query-time retrieval vs ingest-time compilation,

- CLI/core logic vs web presentation.

- The repository is best drawn as a pipeline with a persistent wiki artifact in the middle, not as a typical CRUD app or as a simple retrieval system.