How browsercash/teracrawl works
Teracrawl exposes direct URL scraping, search-only proxying, and a search-then-scrape workflow that returns Markdown. The crawl handler obtains search results, filters URLs, acquires pooled browser sessions, and invokes the scraper; scraping tries a fast pass before a slower rendering fallback. HTML extraction and optional PDF conversion are separate content paths. The sampled sources support these request and processing relationships; unsampled modules are not inferred beyond documented capabilities.
Main entry points: crawl.ts Crawl and scrape routes · index.ts HTTP server · scraper.ts URL scraper · pool.ts Browser session pool
Components 16
API entry
- HTTP server
src/index.ts - Crawl and scrape routes
src/crawl.ts - Search proxy route
src/serp.ts - Health endpoint
src/index.ts
Search and crawl
- Search-and-crawl orchestration
src/crawl.ts - SERP query client
src/serp.ts - SERP service
- URL scraper
src/scraper.ts
Content processing
- HTML to Markdown
src/markdown.ts - PDF conversion
src/pdf.ts - Datalab Marker API
Browser runtime
- Browser session pool
src/pool.ts - Browser.cash browsers
- Environment configuration
src/env.ts - Request performance logging
src/perf.ts
Other components
- API client
Connections 23
- API client → HTTP server: sends requests
- HTTP server → Crawl and scrape routes: registers routes
- HTTP server → Search proxy route: registers routes
- HTTP server → Health endpoint: serves health
- Crawl and scrape routes → Search-and-crawl orchestration: handles crawl
- Crawl and scrape routes → URL scraper: scrapes URL
- Search-and-crawl orchestration → SERP query client: queries search
- Search proxy route → SERP query client: proxies query
- SERP query client → SERP service: requests results
- Search-and-crawl orchestration → Browser session pool: acquires sessions
- Search-and-crawl orchestration → URL scraper: scrapes result URLs
- URL scraper → HTML to Markdown: converts HTML
- URL scraper → PDF conversion: converts PDF
- PDF conversion → Datalab Marker API: submits and polls
- Browser session pool → Browser.cash browsers: manages sessions
- HTTP server → Browser session pool: initializes and shuts down
- Browser session pool → Environment configuration: loads settings
- Crawl and scrape routes → Environment configuration: loads crawl settings
- URL scraper → Environment configuration: loads scrape settings
- PDF conversion → Environment configuration: loads API key
- Crawl and scrape routes → Request performance logging: records timings
- Search proxy route → API client: returns search results
- Crawl and scrape routes → API client: returns scrape results