# niharshrotri/llm-wiki architecture

> Architecture diagram of the GitHub repository niharshrotri/llm-wiki, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/niharshrotri/llm-wiki
- Repository: https://github.com/niharshrotri/llm-wiki

## Overview

1. **Repository shape**
   - This is a **local-first knowledge-base compiler**: it takes documents from a raw source area, converts them into an Obsidian-style wiki, and then exposes that wiki through CLI, search/query, linting, and a web UI.
   - The central architectural idea is that the wiki is a **persistent derived artifact**, not a hand-edited source of truth.

2. **Core data boundary: raw vs. generated vs. internal state**
   - **`raw/`** is the immutable input boundary: original PDFs, Markdown, HTML, DOCX, and text files are read here and never modified by the system.
   - **`wiki/`** is the generated output boundary: markdown pages with YAML frontmatter and `[[wikilinks]]`, organized into page-type folders like `sources/`, `entities/`, `concepts/`, and `synthesis/`.
   - **`.wiki/`** is internal runtime state: database, ingest history, and search/index artifacts. It is implementation detail, not user content.

3. **CLI as the primary control plane**
   - `src/llm_wiki/cli.py` is the main entrypoint and command router.
   - It exposes project lifecycle commands such as init, add, ingest, query, lint, reindex, status, and serve.
   - Most user workflows start here, so the CLI is the orchestration layer over the rest of the system.

4. **Project scaffolding and conventions**
   - `scaffold.py` and `src/llm_wiki/templates/` build the initial wiki vault structure and seed files like `index.md`, `log.md`, and Obsidian graph configs.
   - `schema/AGENTS.md` is a key architectural boundary: it defines the rules the LLM follows for formatting, merging, linking, and citing pages.
   - In practice, the schema file acts like a policy/config contract between the human-maintained repo and the model-driven content generation.

5. **Source ingestion pipeline**
   - `ingest_raw.py` handles importing files into the raw/source registry and registering them for processing.
   - The ingest flow is multi-stage and source-centric: parse document → extract structure and entities/concepts → draft or merge wiki pages → write source provenance → update index/log/search state.
   - `jobs.py` supports long-running ingest work, especially for the web UI, so ingest runs are tracked as durable jobs rather than ephemeral requests.

6. **Document parsing subsystem**
   - `src/llm_wiki/parsers/` is the file-format adapter layer.
   - Separate parser implementations for `pdf.py`, `docx.py`, `html.py`, and `text.py` normalize heterogeneous inputs into a common text representation.
   - `parsers/base.py` defines the shared interface, which keeps format-specific extraction isolated from the rest of the ingest logic.

7. **LLM-driven compilation layer**
   - `ingest_llm.py`, `llm.py`, `prompts.py`, `page_writer.py`, and `intent.py` form the model orchestration layer.
   - The LLM is used for structured extraction, page drafting/merging, contradiction handling, and synthesis generation.
   - `intent.py` short-circuits trivial chatty queries so the system does not waste retrieval/LLM work on non-knowledge questions.
   - `page_writer.py` is the boundary where extracted knowledge becomes durable wiki markdown.

8. **Search and query path**
   - `search.py` and `query.py` implement query-time retrieval over the compiled wiki.
   - The README describes a **hybrid search stack**: BM25 + vector similarity + LLM reranking, delegated to QMD.
   - Query flow is: classify intent → retrieve top pages from the wiki corpus → hydrate page contents → synthesize a cited markdown answer → optionally save it back as a `synthesis/` page.
   - This is distinct from classic RAG: the system queries the **compiled wiki** first, not the raw documents.

9. **Quality control and maintenance**
   - `lint.py` runs structural health checks across the wiki: broken links, malformed frontmatter, orphans, noise, and optional deep contradiction detection.
   - Lint can auto-fix many formatting/link issues, and it also triggers index refresh so corrected content becomes searchable immediately.
   - This makes lint part of the operational lifecycle, not just a static validation tool.

10. **Persistent storage and metadata**
   - `db.py` is the persistence layer for source records, ingest status, and job/history metadata.
   - The internal SQLite-backed state is important for resumability and for the web UI’s durable job tracking.
   - `slugify.py` supports stable page naming and identifier generation, which matters because page identity is used for links, merges, and provenance tracking.

11. **Web application boundary**
   - `src/llm_wiki/webapp/main.py` is a separate HTTP interface, organized as a FastAPI-style application with route modules under `webapp/routes/`.
   - Routes are split by concern: `dashboard`, `sources`, `ingest`, `jobs`, `query`, `lint`, and `graph`.
   - The web layer is primarily a presentation/orchestration shell over the same core ingest/query/lint subsystems used by the CLI, not a second implementation.
   - `webapp/templates/` and `webapp/static/` provide the server-rendered UI and styling.

12. **External runtime dependencies that shape the architecture**
   - **Ollama** hosts the local LLM used for reading, writing, and synthesis.
   - **QMD** is the search backend and handles hybrid retrieval plus reranking.
   - **Obsidian** is the consumer of the generated wiki folder; its backlink/graph model strongly influences the markdown structure and wikilink conventions.
   - The stack is intentionally local-only: there are no cloud APIs in the core architecture.

13. **What a diagram should emphasize**
   - The main flow is: **raw documents → parsers → LLM compilation → wiki markdown → search/query/lint/web UI**.
   - The important boundaries are:
     - immutable input vs generated knowledge base,
     - model orchestration vs page writing,
     - query-time retrieval vs ingest-time compilation,
     - CLI/core logic vs web presentation.
   - The repository is best drawn as a **pipeline with a persistent wiki artifact in the middle**, not as a typical CRUD app or as a simple retrieval system.

Main entry points: [`src/llm_wiki/cli.py`](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/cli.py) (CLI), [`src/llm_wiki/webapp/routes`](https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/webapp/routes) (Routes), [`src/llm_wiki/query.py`](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/query.py) (Query), [`src/llm_wiki/ingest_llm.py`](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/ingest_llm.py) (Compile)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_boundary["Data boundaries"]
  node_raw[("raw/<br/>source boundary")]
  node_wiki["wiki/<br/>generated wiki"]
  node_state[(".wiki/<br/>runtime state")]
end

subgraph group_core["Core pipeline"]
  node_cli["CLI<br/>command router<br/>[cli.py]"]
  node_ingest_raw["Import<br/>source registry<br/>[ingest_raw.py]"]
  node_parsers["Parsers<br/>format adapters"]
  node_ingest_llm["Compile<br/>LLM pipeline<br/>[ingest_llm.py]"]
  node_page_writer["Page writer<br/>wiki emitter<br/>[page_writer.py]"]
  node_llm["LLM<br/>model runtime<br/>[llm.py]"]
  node_prompts["Prompts<br/>prompt library<br/>[prompts.py]"]
  node_intent["Intent<br/>query gate<br/>[intent.py]"]
  node_search["Search<br/>retrieval<br/>[search.py]"]
  node_query["Query<br/>cited answer flow<br/>[query.py]"]
  node_lint["Lint<br/>wiki health<br/>[lint.py]"]
  node_db[("DB<br/>metadata store<br/>[db.py]")]
  node_slugify["Slugify<br/>identity helper<br/>[slugify.py]"]
end

subgraph group_web["Web app"]
  node_jobs["Jobs<br/>durable work queue<br/>[jobs.py]"]
  node_webmain["Web app<br/>HTTP app<br/>[main.py]"]
  node_routes["Routes"]
  node_webui["UI assets<br/>templates and static"]
end

subgraph group_scaffold["Scaffolding"]
  node_scaffold_py["Scaffold<br/>vault bootstrap<br/>[scaffold.py]"]
  node_templates["Templates<br/>seed vault files"]
  node_agentrules["AGENTS.md<br/>LLM policy<br/>[AGENTS.md]"]
end

node_cli -->|"init"| node_scaffold_py
node_scaffold_py -->|"seed"| node_templates
node_scaffold_py -->|"policy"| node_agentrules
node_cli -->|"ingest"| node_ingest_raw
node_ingest_raw -->|"reads"| node_raw
node_ingest_raw -->|"dispatch"| node_parsers
node_parsers -->|"normalized text"| node_ingest_llm
node_ingest_llm -->|"model calls"| node_llm
node_ingest_llm -->|"guides"| node_prompts
node_ingest_llm -->|"drafts pages"| node_page_writer
node_page_writer -->|"writes"| node_wiki
node_page_writer -->|"records"| node_db
node_page_writer -->|"names"| node_slugify
node_cli -->|"query"| node_query
node_query -->|"classify"| node_intent
node_query -->|"retrieve"| node_search
node_search -->|"index over"| node_wiki
node_query -->|"synthesize"| node_llm
node_query -->|"save synthesis"| node_wiki
node_cli -->|"lint"| node_lint
node_lint -->|"analyze"| node_wiki
node_lint -->|"refresh index"| node_search
node_cli -->|"status"| node_db
node_db -->|"persists"| node_state
node_cli -->|"serve"| node_webmain
node_webmain -->|"mounts"| node_routes
node_webmain -->|"renders"| node_webui
node_routes -->|"tracks"| node_jobs
node_routes -->|"trigger ingest"| node_ingest_raw
node_routes -->|"run query"| node_query
node_routes -->|"run lint"| node_lint
node_routes -->|"graph/search"| node_search

click node_cli "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/cli.py"
click node_scaffold_py "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/scaffold.py"
click node_templates "https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/templates"
click node_agentrules "https://github.com/niharshrotri/llm-wiki/blob/main/schema/AGENTS.md"
click node_ingest_raw "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/ingest_raw.py"
click node_parsers "https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/parsers"
click node_ingest_llm "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/ingest_llm.py"
click node_page_writer "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/page_writer.py"
click node_llm "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/llm.py"
click node_prompts "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/prompts.py"
click node_intent "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/intent.py"
click node_search "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/search.py"
click node_query "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/query.py"
click node_lint "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/lint.py"
click node_db "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/db.py"
click node_slugify "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/slugify.py"
click node_jobs "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/jobs.py"
click node_webmain "https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/webapp/main.py"
click node_routes "https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/webapp/routes"
click node_webui "https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/webapp/templates"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw,node_wiki,node_state toneBlue
class node_cli,node_ingest_raw,node_parsers,node_ingest_llm,node_page_writer,node_llm,node_prompts,node_intent,node_search,node_query,node_lint,node_db,node_slugify toneAmber
class node_jobs,node_webmain,node_routes,node_webui toneMint
class node_scaffold_py,node_templates,node_agentrules toneRose
```

## Components

### Data boundaries

Immutable input, generated wiki output, and internal runtime state.

- **raw/**: source boundary · Immutable input documents imported for processing.
- **wiki/**: generated wiki · Persistent derived Obsidian-style markdown vault.
- **.wiki/**: runtime state · Internal database, ingest history, and search artifacts.

### Core pipeline

CLI, ingest, parsing, compilation, search, and maintenance.

- [CLI](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/cli.py): command router · Primary control plane for init, ingest, query, lint, reindex, status, and serve. · `src/llm_wiki/cli.py`
- [Import](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/ingest_raw.py): source registry · Registers raw files and prepares them for ingestion. · `src/llm_wiki/ingest_raw.py`
- [Parsers](https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/parsers): format adapters · Normalizes PDF, DOCX, HTML, and text inputs into common text. · `src/llm_wiki/parsers`
- [Compile](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/ingest_llm.py): LLM pipeline · Extracts structure, entities, and synthesis from parsed content. · `src/llm_wiki/ingest_llm.py`
- [Page writer](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/page_writer.py): wiki emitter · Turns extracted knowledge into durable wiki markdown pages. · `src/llm_wiki/page_writer.py`
- [LLM](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/llm.py): model runtime · Local model backend used for reading, drafting, merging, and synthesis. · `src/llm_wiki/llm.py`
- [Prompts](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/prompts.py): prompt library · Structured instructions that steer extraction and writing. · `src/llm_wiki/prompts.py`
- [Intent](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/intent.py): query gate · Short-circuits trivial chatty requests before retrieval or synthesis. · `src/llm_wiki/intent.py`
- [Search](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/search.py): retrieval · Hybrid retrieval over the compiled wiki corpus. · `src/llm_wiki/search.py`
- [Query](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/query.py): cited answer flow · Retrieves wiki pages, hydrates content, and synthesizes cited answers. · `src/llm_wiki/query.py`
- [Lint](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/lint.py): wiki health · Checks links, frontmatter, orphans, noise, and contradictions; can auto-fix. · `src/llm_wiki/lint.py`
- [DB](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/db.py): metadata store · Persists sources, ingest status, and job/history metadata. · `src/llm_wiki/db.py`
- [Slugify](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/slugify.py): identity helper · Generates stable page names and identifiers for links and provenance. · `src/llm_wiki/slugify.py`

### Web app

HTTP presentation and durable job orchestration.

- [Jobs](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/jobs.py): durable work queue · Tracks long-running ingest and UI-driven background work. · `src/llm_wiki/jobs.py`
- [Web app](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/webapp/main.py): HTTP app · FastAPI-style application serving the UI and route modules. · `src/llm_wiki/webapp/main.py`
- [Routes](https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/webapp/routes): presentation routes · Dashboard, sources, ingest, jobs, query, lint, and graph endpoints. · `src/llm_wiki/webapp/routes`
- [UI assets](https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/webapp/templates): templates and static · Server-rendered HTML templates and CSS for the web interface. · `src/llm_wiki/webapp/templates`

### Scaffolding

Project bootstrap and wiki template conventions.

- [Scaffold](https://github.com/niharshrotri/llm-wiki/blob/main/src/llm_wiki/scaffold.py): vault bootstrap · Creates the initial wiki structure and seed files. · `src/llm_wiki/scaffold.py`
- [Templates](https://github.com/niharshrotri/llm-wiki/tree/main/src/llm_wiki/templates): seed vault files · Initial pages and Obsidian config templates. · `src/llm_wiki/templates`
- [AGENTS.md](https://github.com/niharshrotri/llm-wiki/blob/main/schema/AGENTS.md): LLM policy · Formatting, merging, linking, and citation rules for generated pages. · `schema/AGENTS.md`

## Connections

- CLI → Scaffold: init · Bootstraps a new wiki vault.
- Scaffold → Templates: seed · Copies starter pages and config into the vault.
- Scaffold → AGENTS.md: policy · Applies repository conventions for generated content.
- CLI → Import: ingest · Starts source registration and import.
- Import → raw/: reads · Consumes immutable source documents.
- Import → Parsers: dispatch · Sends files to the appropriate format adapter.
- Parsers → Compile: normalized text · Provides common extracted text for compilation.
- Compile → LLM: model calls · Uses the local model for extraction and drafting.
- Compile → Prompts: guides · Uses prompt templates to shape model behavior.
- Compile → Page writer: drafts pages · Produces structured page content for the wiki.
- Page writer → wiki/: writes · Materializes persistent markdown pages.
- Page writer → DB: records · Stores provenance and source metadata.
- Page writer → Slugify: names · Generates stable page identifiers and slugs.
- CLI → Query: query · Runs retrieval and answer synthesis.
- Query → Intent: classify · Detects trivial chatty requests before search.
- Query → Search: retrieve · Finds relevant pages in the compiled wiki.
- Search → wiki/: index over · Searches the derived wiki corpus first.
- Query → LLM: synthesize · Produces cited markdown answers.
- Query → wiki/: save synthesis · Optionally writes answers back as synthesis pages.
- CLI → Lint: lint · Checks and repairs wiki structure.
- Lint → wiki/: analyze · Scans generated content for issues.
- Lint → Search: refresh index · Reindexes content after fixes.
- CLI → DB: status · Reads persistent project and ingest state.
- DB → .wiki/: persists · Stores internal runtime metadata and history.
- CLI → Web app: serve · Starts the web interface.
- Web app → Routes: mounts · Registers route modules for each concern.
- Web app → UI assets: renders · Serves HTML templates and static assets.
- Routes → Jobs: tracks · Surfaces durable background work in the UI.
- Routes → Import: trigger ingest · Invokes the same ingest pipeline as the CLI.
- Routes → Query: run query · Shares the core query flow with the CLI.
- Routes → Lint: run lint · Shares the core lint flow with the CLI.
- Routes → Search: graph/search · Exposes repository and wiki views through the UI.