How lulu38m/spider-crawler works
Spider Crawler turns a file or link into a document model, analyzes its text, then stages the actual highlights and keyword catches as a procedural animated crawl and presents a summary. The README also documents OCR, browser-extension crawling, terms and finance modes, optional AI answers, and video/export features. Some paths are documented but their implementation was not sampled, so those links are dashed where the runtime wiring is not evidenced.
Main entry points: app.ts App workflow · index.ts Summary engine · scene.ts Scene renderer · crawl.ts Live page crawl
Components 28
Document intake
- Document loading
src/doc/load.ts - Link parsing
src/doc/url.ts - Extension inbox
src/inbox.ts - Live page crawl
src/live/crawl.ts - Page extraction
src/live/extract.ts - Fetch server
server/api.mjs
Text analysis
- Summary engine
src/summarize/index.ts - Code analysis
src/summarize/code.ts - Finance extraction
src/summarize/finance.ts - Terms extraction
src/summarize/terms.ts - Keyword extraction
src/summarize/keywords.ts - AI answers
src/summarize/answer.ts
Crawl experience
- Scene renderer
src/scene/scene.ts - Crawl director
src/scene/director.ts - Spider rig
src/spider/spider.ts - Spider skins
src/spider/skins.ts - Silk web
src/scene/web.ts - Sound effects
src/audio/sfx.ts
Results and integrations
- Summary export
src/ui/export.ts - Video recorder
src/ui/recorder.ts - Live results panel
src/live/panel.ts
Entry points
- Reader
src/main.ts - App workflow
src/app.ts - Page extension
extension/page.js - Extension worker
extension/background.js
Other components
- User
- Public web
- AI provider
Connections 29
- User → Reader: opens app
- Reader → App workflow: starts
src/main.ts - User → App workflow: provides input
- App workflow → Document loading: loads file
src/app.ts - App workflow → Link parsing: parses link
src/app.ts - App workflow → Extension inbox: opens extension input
src/app.ts - Extension inbox → Document loading: loads payload
src/inbox.ts - App workflow → Summary engine: summarizes text
src/app.ts - Summary engine → Code analysis: dispatches code
src/summarize/index.ts - Summary engine → Finance extraction: extracts figures
src/summarize/index.ts - Summary engine → Terms extraction: finds clauses
src/summarize/index.ts - Summary engine → Keyword extraction: extracts keywords
src/summarize/index.ts - App workflow → AI answers: asks question
src/app.ts - AI answers → AI provider: requests answer
- App workflow → Scene renderer: presents crawl
src/app.ts - Scene renderer → Crawl director: creates choreography
src/scene/scene.ts - Crawl director → Spider rig: moves spider
src/scene/director.ts - Scene renderer → Spider rig: animates
src/scene/scene.ts - Scene renderer → Spider skins: uses selected skin
src/scene/scene.ts - Scene renderer → Sound effects: emits events
src/scene/scene.ts - App workflow → Summary export: exports summary
src/app.ts - App workflow → Video recorder: records video
src/app.ts - App workflow → Fetch server: fetches linked pages
- Fetch server → Public web: fetches public page
server/api.mjs - User → Extension worker: activates extension
- Extension worker → Page extension: injects page crawler
extension/background.js - Page extension → Live page crawl: starts crawl
- Live page crawl → Page extraction: extracts visible text
- Live page crawl → Live results panel: shows results