muhammadsaqlainaslam/my-llm-wiki

Purpose and Entry Points

This repository is an AI/ML research wiki: Obsidian-compatible Markdown notes become a static, searchable, citation-aware web knowledge graph. Main operational entry points are agent.py (research ingestion/maintenance), build_wiki.py (PDF-to-note generation), and build.py (site/data generation). The user-facing artifacts are docs/index.html and docs/graph.html.

Canonical Knowledge Store

wiki/ is the source of truth: each Markdown file is a paper or concept note with YAML metadata, prose, and Obsidian [[wikilinks]]. Navigation/index surfaces live alongside content in wiki/Home.md, wiki/000 Index.md, and wiki/Citation Map.md. The graph boundary is explicit: Markdown links encode relationships; frontmatter encodes searchable/citation metadata.

Research Ingestion Agent

agent.py is the autonomous-but-manually-triggered acquisition layer. Its modes search topical content, monitor arXiv/GitHub/blog sources, discover citing papers, refresh citation counts, and audit citation identities. It uses Claude for relevance evaluation and note generation decisions, while citation refresh and arXiv verification use deterministic external API calls. Run history is persisted in agent_log.json; Claude-facing operational guidance is in skills/wiki-agent/SKILL.md.

PDF-to-Note Generation

build_wiki.py transforms source papers into structured notes, using PyMuPDF for extraction and the Anthropic Claude API for summarization and wiki-note authoring. Its output enters wiki/, rather than directly modifying the site, preserving a reviewable content boundary. Runtime credentials and optional vault location are configured through .env.example.

Static Build Pipeline

build.py compiles wiki/*.md into browser-consumable data and rendered note HTML. It parses YAML, resolves wikilinks, protects/restores LaTex math, converts Markdown/callouts/embeds, and emits metadata including links, citations, arXiv IDs, and rendered body content. Primary generated payload: docs/notes.json; presentation artifacts: docs/index.html and docs/graph.html.

Browser Application Surfaces

The site is a dependency-light static client: docs/index.html provides browsing, search, filtering, timeline, citation exploration, note rendering, backlinks, comparisons, and local reading state. docs/graph.html loads the generated notes data into a D3 v7 force-directed relationship graph. KaTeX renders math client-side; browser localStorage stores appearance and reading-progress state. No application server or database is present.

Deployment and External Services

GitHub Actions deploys generated docs/ content to GitHub Pages on pushes, defined in .github/workflows/deploy.yml. External integration boundaries are Anthropic/Vertex Claude for generation, arXiv for paper search/identity data, Semantic Scholar for citations, GitHub APIs for repository content, and optional YouTube transcript extraction. setup_cron.sh exists for optional scheduling, but the intended control model is manual agent invocation followed by generated-content commits.

End-to-End Control Flow

New content flows from PDFs or agent-discovered sources → build_wiki.py/agent.py → versioned Markdown in wiki/ → build.py → static docs/ assets → GitHub Actions → GitHub Pages. Citation maintenance follows Semantic Scholar → YAML frontmatter in wiki/ → rebuild of docs/notes.json. Wikilinks and metadata are therefore authored once in the vault and consumed consistently by browse, search, citation, and graph views.