# browsercash/teracrawl architecture

> Architecture diagram of the GitHub repository browsercash/teracrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/browsercash/teracrawl
- Repository: https://github.com/browsercash/teracrawl

## Overview

Teracrawl exposes direct URL scraping, search-only proxying, and a search-then-scrape workflow that returns Markdown. The crawl handler obtains search results, filters URLs, acquires pooled browser sessions, and invokes the scraper; scraping tries a fast pass before a slower rendering fallback. HTML extraction and optional PDF conversion are separate content paths. The sampled sources support these request and processing relationships; unsampled modules are not inferred beyond documented capabilities.

Main entry points: [`src/crawl.ts`](https://github.com/browsercash/teracrawl/blob/main/src/crawl.ts) (Crawl and scrape routes), [`src/index.ts`](https://github.com/browsercash/teracrawl/blob/main/src/index.ts) (HTTP server), [`src/scraper.ts`](https://github.com/browsercash/teracrawl/blob/main/src/scraper.ts) (URL scraper), [`src/pool.ts`](https://github.com/browsercash/teracrawl/blob/main/src/pool.ts) (Browser session pool)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API entry"]
  node_server["HTTP server<br/>[index.ts]"]
  node_crawl_routes["Crawl and scrape routes<br/>[crawl.ts]"]
  node_serp_routes["Search proxy route<br/>[serp.ts]"]
  node_health["Health endpoint<br/>[index.ts]"]
end

subgraph group_search_crawl["Search and crawl"]
  node_crawl_flow["Search-and-crawl orchestration<br/>[crawl.ts]"]
  node_serp_query["SERP query client<br/>[serp.ts]"]
  node_serp_service{{"SERP service"}}
  node_scraper["URL scraper<br/>[scraper.ts]"]
end

subgraph group_content["Content processing"]
  node_markdown["HTML to Markdown<br/>[markdown.ts]"]
  node_pdf_conversion["PDF conversion<br/>[pdf.ts]"]
  node_datalab{{"Datalab Marker API"}}
end

subgraph group_runtime["Browser runtime"]
  node_pool["Browser session pool<br/>[pool.ts]"]
  node_remote_browser{{"Browser.cash browsers"}}
  node_config["Environment configuration<br/>[env.ts]"]
  node_perf["Request performance logging<br/>[perf.ts]"]
end

node_client(("API client"))

node_client -->|"sends requests"| node_server
node_server -->|"registers routes"| node_crawl_routes
node_server -->|"registers routes"| node_serp_routes
node_server -->|"serves health"| node_health
node_crawl_routes -->|"handles crawl"| node_crawl_flow
node_crawl_routes -->|"scrapes URL"| node_scraper
node_crawl_flow -->|"queries search"| node_serp_query
node_serp_routes -->|"proxies query"| node_serp_query
node_serp_query -->|"requests results"| node_serp_service
node_crawl_flow -->|"acquires sessions"| node_pool
node_crawl_flow -->|"scrapes result URLs"| node_scraper
node_scraper -->|"converts HTML"| node_markdown
node_scraper -->|"converts PDF"| node_pdf_conversion
node_pdf_conversion -.->|"submits and polls"| node_datalab
node_pool -->|"manages sessions"| node_remote_browser
node_server -->|"initializes and shuts down"| node_pool
node_pool -->|"loads settings"| node_config
node_crawl_routes -->|"loads crawl settings"| node_config
node_scraper -->|"loads scrape settings"| node_config
node_pdf_conversion -->|"loads API key"| node_config
node_crawl_routes -->|"records timings"| node_perf
node_serp_routes -->|"returns search results"| node_client
node_crawl_routes -->|"returns scrape results"| node_client

click node_server "https://github.com/browsercash/teracrawl/blob/main/src/index.ts"
click node_crawl_routes "https://github.com/browsercash/teracrawl/blob/main/src/crawl.ts"
click node_serp_routes "https://github.com/browsercash/teracrawl/blob/main/src/serp.ts"
click node_health "https://github.com/browsercash/teracrawl/blob/main/src/index.ts"
click node_crawl_flow "https://github.com/browsercash/teracrawl/blob/main/src/crawl.ts"
click node_serp_query "https://github.com/browsercash/teracrawl/blob/main/src/serp.ts"
click node_scraper "https://github.com/browsercash/teracrawl/blob/main/src/scraper.ts"
click node_markdown "https://github.com/browsercash/teracrawl/blob/main/src/markdown.ts"
click node_pdf_conversion "https://github.com/browsercash/teracrawl/blob/main/src/pdf.ts"
click node_pool "https://github.com/browsercash/teracrawl/blob/main/src/pool.ts"
click node_config "https://github.com/browsercash/teracrawl/blob/main/src/env.ts"
click node_perf "https://github.com/browsercash/teracrawl/blob/main/src/perf.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_crawl_routes,node_serp_routes,node_health,node_client toneBlue
class node_crawl_flow,node_serp_query,node_serp_service,node_scraper toneAmber
class node_markdown,node_pdf_conversion,node_datalab toneMint
class node_pool,node_remote_browser,node_config,node_perf toneRose
```

## Components

### API entry

- [HTTP server](https://github.com/browsercash/teracrawl/blob/main/src/index.ts): `src/index.ts`
- [Crawl and scrape routes](https://github.com/browsercash/teracrawl/blob/main/src/crawl.ts): `src/crawl.ts`
- [Search proxy route](https://github.com/browsercash/teracrawl/blob/main/src/serp.ts): `src/serp.ts`
- [Health endpoint](https://github.com/browsercash/teracrawl/blob/main/src/index.ts): `src/index.ts`

### Search and crawl

- [Search-and-crawl orchestration](https://github.com/browsercash/teracrawl/blob/main/src/crawl.ts): `src/crawl.ts`
- [SERP query client](https://github.com/browsercash/teracrawl/blob/main/src/serp.ts): `src/serp.ts`
- **SERP service**
- [URL scraper](https://github.com/browsercash/teracrawl/blob/main/src/scraper.ts): `src/scraper.ts`

### Content processing

- [HTML to Markdown](https://github.com/browsercash/teracrawl/blob/main/src/markdown.ts): `src/markdown.ts`
- [PDF conversion](https://github.com/browsercash/teracrawl/blob/main/src/pdf.ts): `src/pdf.ts`
- **Datalab Marker API**

### Browser runtime

- [Browser session pool](https://github.com/browsercash/teracrawl/blob/main/src/pool.ts): `src/pool.ts`
- **Browser.cash browsers**
- [Environment configuration](https://github.com/browsercash/teracrawl/blob/main/src/env.ts): `src/env.ts`
- [Request performance logging](https://github.com/browsercash/teracrawl/blob/main/src/perf.ts): `src/perf.ts`

### Other components

- **API client**

## Connections

- API client → HTTP server: sends requests
- HTTP server → Crawl and scrape routes: registers routes
- HTTP server → Search proxy route: registers routes
- HTTP server → Health endpoint: serves health
- Crawl and scrape routes → Search-and-crawl orchestration: handles crawl
- Crawl and scrape routes → URL scraper: scrapes URL
- Search-and-crawl orchestration → SERP query client: queries search
- Search proxy route → SERP query client: proxies query
- SERP query client → SERP service: requests results
- Search-and-crawl orchestration → Browser session pool: acquires sessions
- Search-and-crawl orchestration → URL scraper: scrapes result URLs
- URL scraper → HTML to Markdown: converts HTML
- URL scraper → PDF conversion: converts PDF
- PDF conversion → Datalab Marker API: submits and polls
- Browser session pool → Browser.cash browsers: manages sessions
- HTTP server → Browser session pool: initializes and shuts down
- Browser session pool → Environment configuration: loads settings
- Crawl and scrape routes → Environment configuration: loads crawl settings
- URL scraper → Environment configuration: loads scrape settings
- PDF conversion → Environment configuration: loads API key
- Crawl and scrape routes → Request performance logging: records timings
- Search proxy route → API client: returns search results
- Crawl and scrape routes → API client: returns scrape results