# lulu38m/spider-crawler architecture

> Architecture diagram of the GitHub repository lulu38m/spider-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lulu38m/spider-crawler
- Repository: https://github.com/lulu38m/spider-crawler

## Overview

Spider Crawler turns a file or link into a document model, analyzes its text, then stages the actual highlights and keyword catches as a procedural animated crawl and presents a summary. The README also documents OCR, browser-extension crawling, terms and finance modes, optional AI answers, and video/export features. Some paths are documented but their implementation was not sampled, so those links are dashed where the runtime wiring is not evidenced.

Main entry points: [`src/app.ts`](https://github.com/lulu38m/spider-crawler/blob/main/src/app.ts) (App workflow), [`src/summarize/index.ts`](https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/index.ts) (Summary engine), [`src/scene/scene.ts`](https://github.com/lulu38m/spider-crawler/blob/main/src/scene/scene.ts) (Scene renderer), [`src/live/crawl.ts`](https://github.com/lulu38m/spider-crawler/blob/main/src/live/crawl.ts) (Live page crawl)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Document intake"]
  node_document["Document loading<br/>[load.ts]"]
  node_link_parse["Link parsing<br/>[url.ts]"]
  node_inbox["Extension inbox<br/>[inbox.ts]"]
  node_live_crawl["Live page crawl<br/>[crawl.ts]"]
  node_live_extract["Page extraction<br/>[extract.ts]"]
  node_server["Fetch server<br/>[api.mjs]"]
end

subgraph group_analysis["Text analysis"]
  node_summarizer["Summary engine<br/>[index.ts]"]
  node_code_mode["Code analysis<br/>[code.ts]"]
  node_finance_mode["Finance extraction<br/>[finance.ts]"]
  node_terms_mode["Terms extraction<br/>[terms.ts]"]
  node_keywords["Keyword extraction<br/>[keywords.ts]"]
  node_ai["AI answers<br/>[answer.ts]"]
end

subgraph group_experience["Crawl experience"]
  node_scene["Scene renderer<br/>[scene.ts]"]
  node_director["Crawl director<br/>[director.ts]"]
  node_spider["Spider rig<br/>[spider.ts]"]
  node_skins["Spider skins<br/>[skins.ts]"]
  node_web["Silk web<br/>[web.ts]"]
  node_sound["Sound effects<br/>[sfx.ts]"]
end

subgraph group_outputs["Results and integrations"]
  node_export["Summary export<br/>[export.ts]"]
  node_recorder["Video recorder<br/>[recorder.ts]"]
  node_live_panel["Live results panel<br/>[panel.ts]"]
end

subgraph group_delivery["Entry points"]
  node_reader["Reader<br/>[main.ts]"]
  node_app["App workflow<br/>[app.ts]"]
  node_extension["Page extension<br/>[page.js]"]
  node_extension_worker["Extension worker<br/>[background.js]"]
end

node_user(("User"))
node_public_web(("Public web"))
node_ai_provider(("AI provider"))

node_user -->|"opens app"| node_reader
node_reader -->|"starts"| node_app
node_user -->|"provides input"| node_app
node_app -->|"loads file"| node_document
node_app -->|"parses link"| node_link_parse
node_app -->|"opens extension input"| node_inbox
node_inbox -->|"loads payload"| node_document
node_app -->|"summarizes text"| node_summarizer
node_summarizer -->|"dispatches code"| node_code_mode
node_summarizer -->|"extracts figures"| node_finance_mode
node_summarizer -->|"finds clauses"| node_terms_mode
node_summarizer -->|"extracts keywords"| node_keywords
node_app -.->|"asks question"| node_ai
node_ai -.->|"requests answer"| node_ai_provider
node_app -->|"presents crawl"| node_scene
node_scene -->|"creates choreography"| node_director
node_director -->|"moves spider"| node_spider
node_scene -->|"animates"| node_spider
node_scene -->|"uses selected skin"| node_skins
node_scene -->|"emits events"| node_sound
node_app -->|"exports summary"| node_export
node_app -.->|"records video"| node_recorder
node_app -.->|"fetches linked pages"| node_server
node_server -->|"fetches public page"| node_public_web
node_user -->|"activates extension"| node_extension_worker
node_extension_worker -->|"injects page crawler"| node_extension
node_extension -.->|"starts crawl"| node_live_crawl
node_live_crawl -.->|"extracts visible text"| node_live_extract
node_live_crawl -.->|"shows results"| node_live_panel

click node_reader "https://github.com/lulu38m/spider-crawler/blob/main/src/main.ts"
click node_app "https://github.com/lulu38m/spider-crawler/blob/main/src/app.ts"
click node_extension "https://github.com/lulu38m/spider-crawler/blob/main/extension/page.js"
click node_extension_worker "https://github.com/lulu38m/spider-crawler/blob/main/extension/background.js"
click node_document "https://github.com/lulu38m/spider-crawler/blob/main/src/doc/load.ts"
click node_link_parse "https://github.com/lulu38m/spider-crawler/blob/main/src/doc/url.ts"
click node_inbox "https://github.com/lulu38m/spider-crawler/blob/main/src/inbox.ts"
click node_live_crawl "https://github.com/lulu38m/spider-crawler/blob/main/src/live/crawl.ts"
click node_live_extract "https://github.com/lulu38m/spider-crawler/blob/main/src/live/extract.ts"
click node_server "https://github.com/lulu38m/spider-crawler/blob/main/server/api.mjs"
click node_summarizer "https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/index.ts"
click node_code_mode "https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/code.ts"
click node_finance_mode "https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/finance.ts"
click node_terms_mode "https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/terms.ts"
click node_keywords "https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/keywords.ts"
click node_ai "https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/answer.ts"
click node_scene "https://github.com/lulu38m/spider-crawler/blob/main/src/scene/scene.ts"
click node_director "https://github.com/lulu38m/spider-crawler/blob/main/src/scene/director.ts"
click node_spider "https://github.com/lulu38m/spider-crawler/blob/main/src/spider/spider.ts"
click node_skins "https://github.com/lulu38m/spider-crawler/blob/main/src/spider/skins.ts"
click node_web "https://github.com/lulu38m/spider-crawler/blob/main/src/scene/web.ts"
click node_sound "https://github.com/lulu38m/spider-crawler/blob/main/src/audio/sfx.ts"
click node_export "https://github.com/lulu38m/spider-crawler/blob/main/src/ui/export.ts"
click node_recorder "https://github.com/lulu38m/spider-crawler/blob/main/src/ui/recorder.ts"
click node_live_panel "https://github.com/lulu38m/spider-crawler/blob/main/src/live/panel.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_document,node_link_parse,node_inbox,node_live_crawl,node_live_extract,node_server,node_user toneBlue
class node_summarizer,node_code_mode,node_finance_mode,node_terms_mode,node_keywords,node_ai toneAmber
class node_scene,node_director,node_spider,node_skins,node_web,node_sound toneMint
class node_export,node_recorder,node_live_panel toneRose
class node_reader,node_app,node_extension,node_extension_worker,node_public_web,node_ai_provider toneIndigo
```

## Components

### Document intake

- [Document loading](https://github.com/lulu38m/spider-crawler/blob/main/src/doc/load.ts): `src/doc/load.ts`
- [Link parsing](https://github.com/lulu38m/spider-crawler/blob/main/src/doc/url.ts): `src/doc/url.ts`
- [Extension inbox](https://github.com/lulu38m/spider-crawler/blob/main/src/inbox.ts): `src/inbox.ts`
- [Live page crawl](https://github.com/lulu38m/spider-crawler/blob/main/src/live/crawl.ts): `src/live/crawl.ts`
- [Page extraction](https://github.com/lulu38m/spider-crawler/blob/main/src/live/extract.ts): `src/live/extract.ts`
- [Fetch server](https://github.com/lulu38m/spider-crawler/blob/main/server/api.mjs): `server/api.mjs`

### Text analysis

- [Summary engine](https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/index.ts): `src/summarize/index.ts`
- [Code analysis](https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/code.ts): `src/summarize/code.ts`
- [Finance extraction](https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/finance.ts): `src/summarize/finance.ts`
- [Terms extraction](https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/terms.ts): `src/summarize/terms.ts`
- [Keyword extraction](https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/keywords.ts): `src/summarize/keywords.ts`
- [AI answers](https://github.com/lulu38m/spider-crawler/blob/main/src/summarize/answer.ts): `src/summarize/answer.ts`

### Crawl experience

- [Scene renderer](https://github.com/lulu38m/spider-crawler/blob/main/src/scene/scene.ts): `src/scene/scene.ts`
- [Crawl director](https://github.com/lulu38m/spider-crawler/blob/main/src/scene/director.ts): `src/scene/director.ts`
- [Spider rig](https://github.com/lulu38m/spider-crawler/blob/main/src/spider/spider.ts): `src/spider/spider.ts`
- [Spider skins](https://github.com/lulu38m/spider-crawler/blob/main/src/spider/skins.ts): `src/spider/skins.ts`
- [Silk web](https://github.com/lulu38m/spider-crawler/blob/main/src/scene/web.ts): `src/scene/web.ts`
- [Sound effects](https://github.com/lulu38m/spider-crawler/blob/main/src/audio/sfx.ts): `src/audio/sfx.ts`

### Results and integrations

- [Summary export](https://github.com/lulu38m/spider-crawler/blob/main/src/ui/export.ts): `src/ui/export.ts`
- [Video recorder](https://github.com/lulu38m/spider-crawler/blob/main/src/ui/recorder.ts): `src/ui/recorder.ts`
- [Live results panel](https://github.com/lulu38m/spider-crawler/blob/main/src/live/panel.ts): `src/live/panel.ts`

### Entry points

- [Reader](https://github.com/lulu38m/spider-crawler/blob/main/src/main.ts): `src/main.ts`
- [App workflow](https://github.com/lulu38m/spider-crawler/blob/main/src/app.ts): `src/app.ts`
- [Page extension](https://github.com/lulu38m/spider-crawler/blob/main/extension/page.js): `extension/page.js`
- [Extension worker](https://github.com/lulu38m/spider-crawler/blob/main/extension/background.js): `extension/background.js`

### Other components

- **User**
- **Public web**
- **AI provider**

## Connections

- User → Reader: opens app
- Reader → App workflow: starts (evidence: `src/main.ts`)
- User → App workflow: provides input
- App workflow → Document loading: loads file (evidence: `src/app.ts`)
- App workflow → Link parsing: parses link (evidence: `src/app.ts`)
- App workflow → Extension inbox: opens extension input (evidence: `src/app.ts`)
- Extension inbox → Document loading: loads payload (evidence: `src/inbox.ts`)
- App workflow → Summary engine: summarizes text (evidence: `src/app.ts`)
- Summary engine → Code analysis: dispatches code (evidence: `src/summarize/index.ts`)
- Summary engine → Finance extraction: extracts figures (evidence: `src/summarize/index.ts`)
- Summary engine → Terms extraction: finds clauses (evidence: `src/summarize/index.ts`)
- Summary engine → Keyword extraction: extracts keywords (evidence: `src/summarize/index.ts`)
- App workflow → AI answers: asks question (evidence: `src/app.ts`)
- AI answers → AI provider: requests answer
- App workflow → Scene renderer: presents crawl (evidence: `src/app.ts`)
- Scene renderer → Crawl director: creates choreography (evidence: `src/scene/scene.ts`)
- Crawl director → Spider rig: moves spider (evidence: `src/scene/director.ts`)
- Scene renderer → Spider rig: animates (evidence: `src/scene/scene.ts`)
- Scene renderer → Spider skins: uses selected skin (evidence: `src/scene/scene.ts`)
- Scene renderer → Sound effects: emits events (evidence: `src/scene/scene.ts`)
- App workflow → Summary export: exports summary (evidence: `src/app.ts`)
- App workflow → Video recorder: records video (evidence: `src/app.ts`)
- App workflow → Fetch server: fetches linked pages
- Fetch server → Public web: fetches public page (evidence: `server/api.mjs`)
- User → Extension worker: activates extension
- Extension worker → Page extension: injects page crawler (evidence: `extension/background.js`)
- Page extension → Live page crawl: starts crawl
- Live page crawl → Page extraction: extracts visible text
- Live page crawl → Live results panel: shows results
