# sharkpool-sp/reef-proxy architecture

> Architecture diagram of the GitHub repository sharkpool-sp/reef-proxy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sharkpool-sp/reef-proxy
- Repository: https://github.com/sharkpool-sp/reef-proxy
- Video tour (about a minute): https://gitdiagram.com/sharkpool-sp/reef-proxy/video

## Overview

Reef Proxy is an HTTP CORS proxy and scraper. A client calls `/get`, `/post`, or `/scrape` with a target URL; the app routes the request through rate limits, URL and resource checks, then either proxies the target or loads it in a browser. Responses may be transformed, size-checked, cached, and returned, while heat-map controls discourage repeated heavy downloads. The graph covers documented endpoint behavior and sampled implementation paths; some route and integration details are documented rather than independently traced in source.

Main entry points: [`private/proxy-pipeline.js`](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-pipeline.js) (Request pipeline), [`app.js`](https://github.com/sharkpool-sp/reef-proxy/blob/main/app.js) (Express routes), [`private/heat-map.js`](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/heat-map.js) (Heavy-resource controls), [`private/pipeline-utils.js`](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/pipeline-utils.js) (Payload and HEAD checks)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["HTTP entry"]
  node_app["Express routes<br/>[app.js]"]
end

subgraph group_policy["Request controls"]
  node_request["Request pipeline<br/>[proxy-pipeline.js]"]
  node_targeturl["Target URL utilities<br/>[proxy-utils.js]"]
  node_blocklist["Blocked targets<br/>[blocklist.json]"]
  node_rate["Rate limits<br/>[rate-limiter.js]"]
  node_constants["Runtime limits<br/>[constants.js]"]
end

subgraph group_fetch["Fetch and scrape"]
  node_proxy["GET / POST proxy<br/>[app.js]"]
  node_scrape["Rendered-page scrape<br/>[proxy-pipeline.js]"]
  node_browser["Headless browser<br/>[scrape-helper.js]"]
end

subgraph group_processing["Response processing"]
  node_transform["Content transformations<br/>[transformer.js]"]
  node_size["Payload and HEAD checks<br/>[pipeline-utils.js]"]
  node_response["Response handling<br/>[proxy-pipeline.js]"]
end

subgraph group_state["Runtime state"]
  node_heat["Heavy-resource controls<br/>[heat-map.js]"]
  node_cache[("Response cache<br/>[cacher.js]")]
end

node_client(("API client"))
node_target(("Target website"))

node_client -->|"requests"| node_app
node_app -->|"runs checks"| node_request
node_app -->|"routes GET/POST"| node_proxy
node_app -->|"routes scrape"| node_scrape
node_app -->|"initializes limiters"| node_rate
node_app -->|"initializes controls"| node_heat
node_app -->|"initializes cache"| node_cache
node_request -->|"checks limits"| node_rate
node_request -->|"validates target"| node_targeturl
node_request -->|"checks blocked URLs"| node_blocklist
node_request -->|"checks resource flags"| node_heat
node_request -->|"checks cache"| node_cache
node_request -->|"checks target size"| node_size
node_request -->|"dispatches proxy request"| node_proxy
node_proxy -->|"fetches resource"| node_target
node_scrape -->|"opens target page"| node_browser
node_browser -->|"loads page"| node_target
node_scrape -->|"transforms rendered page"| node_transform
node_proxy -->|"transforms response"| node_transform
node_proxy -->|"handles response"| node_response
node_scrape -->|"handles response"| node_response
node_response -->|"records response size"| node_heat
node_response -->|"stores eligible response"| node_cache
node_proxy -->|"watches response size"| node_size
node_scrape -->|"gets target URL"| node_targeturl
node_request -->|"reads debug setting"| node_constants
node_size -->|"imports controls"| node_heat
node_size -->|"reads size limits"| node_constants
node_heat -->|"reads limits"| node_constants
node_targeturl -->|"reads cache expiry"| node_constants

click node_app "https://github.com/sharkpool-sp/reef-proxy/blob/main/app.js"
click node_request "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-pipeline.js"
click node_targeturl "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-utils.js"
click node_blocklist "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/blocklist.json"
click node_rate "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/rate-limiter.js"
click node_heat "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/heat-map.js"
click node_proxy "https://github.com/sharkpool-sp/reef-proxy/blob/main/app.js"
click node_scrape "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-pipeline.js"
click node_browser "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/scrape-helper.js"
click node_transform "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/transformer.js"
click node_size "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/pipeline-utils.js"
click node_response "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-pipeline.js"
click node_cache "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/cacher.js"
click node_constants "https://github.com/sharkpool-sp/reef-proxy/blob/main/private/constants.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_client toneBlue
class node_request,node_targeturl,node_blocklist,node_rate,node_constants toneAmber
class node_proxy,node_scrape,node_browser toneMint
class node_transform,node_size,node_response toneRose
class node_heat,node_cache,node_target toneIndigo
```

## Components

### HTTP entry

- [Express routes](https://github.com/sharkpool-sp/reef-proxy/blob/main/app.js): `app.js`

### Request controls

- [Request pipeline](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-pipeline.js): `private/proxy-pipeline.js`
- [Target URL utilities](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-utils.js): `private/proxy-utils.js`
- [Blocked targets](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/blocklist.json): `private/blocklist.json`
- [Rate limits](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/rate-limiter.js): `private/rate-limiter.js`
- [Runtime limits](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/constants.js): `private/constants.js`

### Fetch and scrape

- [GET / POST proxy](https://github.com/sharkpool-sp/reef-proxy/blob/main/app.js): `app.js`
- [Rendered-page scrape](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-pipeline.js): `private/proxy-pipeline.js`
- [Headless browser](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/scrape-helper.js): `private/scrape-helper.js`

### Response processing

- [Content transformations](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/transformer.js): `private/transformer.js`
- [Payload and HEAD checks](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/pipeline-utils.js): `private/pipeline-utils.js`
- [Response handling](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/proxy-pipeline.js): `private/proxy-pipeline.js`

### Runtime state

- [Heavy-resource controls](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/heat-map.js): `private/heat-map.js`
- [Response cache](https://github.com/sharkpool-sp/reef-proxy/blob/main/private/cacher.js): `private/cacher.js`

### Other components

- **API client**
- **Target website**

## Connections

- API client → Express routes: requests (evidence: `README.md`)
- Express routes → Request pipeline: runs checks (evidence: `app.js`)
- Express routes → GET / POST proxy: routes GET/POST (evidence: `app.js`)
- Express routes → Rendered-page scrape: routes scrape (evidence: `app.js`)
- Express routes → Rate limits: initializes limiters (evidence: `app.js`)
- Express routes → Heavy-resource controls: initializes controls (evidence: `app.js`)
- Express routes → Response cache: initializes cache (evidence: `app.js`)
- Request pipeline → Rate limits: checks limits (evidence: `private/proxy-pipeline.js`)
- Request pipeline → Target URL utilities: validates target (evidence: `private/proxy-pipeline.js`)
- Request pipeline → Blocked targets: checks blocked URLs (evidence: `private/proxy-pipeline.js`)
- Request pipeline → Heavy-resource controls: checks resource flags (evidence: `private/proxy-pipeline.js`)
- Request pipeline → Response cache: checks cache (evidence: `private/proxy-pipeline.js`)
- Request pipeline → Payload and HEAD checks: checks target size (evidence: `private/proxy-pipeline.js`)
- Request pipeline → GET / POST proxy: dispatches proxy request (evidence: `app.js`)
- GET / POST proxy → Target website: fetches resource (evidence: `README.md`)
- Rendered-page scrape → Headless browser: opens target page (evidence: `private/proxy-pipeline.js`)
- Headless browser → Target website: loads page (evidence: `private/scrape-helper.js`)
- Rendered-page scrape → Content transformations: transforms rendered page (evidence: `private/proxy-pipeline.js`)
- GET / POST proxy → Content transformations: transforms response
- GET / POST proxy → Response handling: handles response (evidence: `private/proxy-pipeline.js`)
- Rendered-page scrape → Response handling: handles response (evidence: `private/proxy-pipeline.js`)
- Response handling → Heavy-resource controls: records response size (evidence: `private/proxy-pipeline.js`)
- Response handling → Response cache: stores eligible response (evidence: `private/proxy-pipeline.js`)
- GET / POST proxy → Payload and HEAD checks: watches response size
- Rendered-page scrape → Target URL utilities: gets target URL (evidence: `private/proxy-pipeline.js`)
- Request pipeline → Runtime limits: reads debug setting (evidence: `private/proxy-pipeline.js`)
- Payload and HEAD checks → Heavy-resource controls: imports controls (evidence: `private/pipeline-utils.js`)
- Payload and HEAD checks → Runtime limits: reads size limits (evidence: `private/pipeline-utils.js`)
- Heavy-resource controls → Runtime limits: reads limits (evidence: `private/heat-map.js`)
- Target URL utilities → Runtime limits: reads cache expiry (evidence: `private/proxy-utils.js`)
