How amaghzaz-y/webscout works
Project type and purpose
• This is a full-stack, serverless search engine monorepo (WebScout) composed of three sub-projects:
• core (Rust library + TS binding) – implements indexing, tokenization, query logic
• api (TypeScript/Node.js – Cloudflare Workers) – provides HTTP routes (crawl, parse, index, search, project) and background consumers (crawler, parser, indexer)
• dash (Next.js + React + Tailwind CSS) – user-facing SPA/SSR frontend
File structure
• Top-level “api”, “core”, and “dash” folders map to the main architecture components.
• “api/src/routes” holds REST endpoints; “api/src/consumers” are background jobs triggered via a queue.
• “api/src/lib” abstracts Cloudflare KV, Queue bindings, and the spider logic.
• “core/src” contains Rust modules (document, index, tokenizer, query) and “core/tokenizers” houses language packs.
• “dash/app” contains Next.js pages (/, /search, /auth) and components.
• Configs: tsconfig.json in each TS project, Cargo.toml/Cargo.lock in core, turbo.json orchestrates the monorepo.
Key insights from README
• Designed for Cloudflare Workers (edge), uses KV for index storage, R2 for document storage, and a queue for task orchestration.
• The search algorithm is Rust-based (compiled to WASM or Workers native).
• Lightweight, serverless, distributed – optimized for low latency.
Main components & interactions
a. Frontend (dash)
• Next.js runs on Cloudflare Pages or any edge platform.
• UI pages call the API’s /search route via HTTPS.
b. API layer (api)
• Exposes HTTP endpoints: crawl, parse, index, search, project.
• Pushes tasks to a Cloudflare Queue (consumer pattern).
• Uses core’s WASM functions (index/query) via TS bindings or native Rust Workers.
• Reads/writes indexes in KV; stores raw documents in R2.
c. Consumers
• Crawler: fetches URLs, writes raw HTML to R2, enqueues parse jobs.
• Parser: reads HTML from R2, extracts text, enqueues index jobs.
• Indexer: tokenizes with core library, updates postings in KV.
d. Storage & External Services
• Cloudflare KV – inverted index shards, caches
• R2 Buckets – raw/processed content storage
• Queue – task orchestration
• (Optional) Cloudflare Workers environment
Architectural patterns & design principles
• “Function-per-responsibility”: separate consumers for crawl, parse, index
• Serverless micro-services: each route/consumer can be a separate Worker
• Publish-subscribe or queue-based decoupling for resilience and horizontal scalability
• Shared core library in Rust for performance, compiled once and used by multiple Workers