How sharkpool-sp/reef-proxy works
Reef Proxy is an HTTP CORS proxy and scraper. A client calls /get, /post, or /scrape with a target URL; the app routes the request through rate limits, URL and resource checks, then either proxies the target or loads it in a browser. Responses may be transformed, size-checked, cached, and returned, while heat-map controls discourage repeated heavy downloads. The graph covers documented endpoint behavior and sampled implementation paths; some route and integration details are documented rather than independently traced in source.
Main entry points: proxy-pipeline.js Request pipeline · app.js Express routes · heat-map.js Heavy-resource controls · pipeline-utils.js Payload and HEAD checks
Components 16
HTTP entry
- Express routes
app.js
Request controls
- Request pipeline
private/proxy-pipeline.js - Target URL utilities
private/proxy-utils.js - Blocked targets
private/blocklist.json - Rate limits
private/rate-limiter.js - Runtime limits
private/constants.js
Fetch and scrape
- GET / POST proxy
app.js - Rendered-page scrape
private/proxy-pipeline.js - Headless browser
private/scrape-helper.js
Response processing
- Content transformations
private/transformer.js - Payload and HEAD checks
private/pipeline-utils.js - Response handling
private/proxy-pipeline.js
Runtime state
- Heavy-resource controls
private/heat-map.js - Response cache
private/cacher.js
Other components
- API client
- Target website
Connections 30
- API client → Express routes: requests
README.md - Express routes → Request pipeline: runs checks
app.js - Express routes → GET / POST proxy: routes GET/POST
app.js - Express routes → Rendered-page scrape: routes scrape
app.js - Express routes → Rate limits: initializes limiters
app.js - Express routes → Heavy-resource controls: initializes controls
app.js - Express routes → Response cache: initializes cache
app.js - Request pipeline → Rate limits: checks limits
private/proxy-pipeline.js - Request pipeline → Target URL utilities: validates target
private/proxy-pipeline.js - Request pipeline → Blocked targets: checks blocked URLs
private/proxy-pipeline.js - Request pipeline → Heavy-resource controls: checks resource flags
private/proxy-pipeline.js - Request pipeline → Response cache: checks cache
private/proxy-pipeline.js - Request pipeline → Payload and HEAD checks: checks target size
private/proxy-pipeline.js - Request pipeline → GET / POST proxy: dispatches proxy request
app.js - GET / POST proxy → Target website: fetches resource
README.md - Rendered-page scrape → Headless browser: opens target page
private/proxy-pipeline.js - Headless browser → Target website: loads page
private/scrape-helper.js - Rendered-page scrape → Content transformations: transforms rendered page
private/proxy-pipeline.js - GET / POST proxy → Content transformations: transforms response
- GET / POST proxy → Response handling: handles response
private/proxy-pipeline.js - Rendered-page scrape → Response handling: handles response
private/proxy-pipeline.js - Response handling → Heavy-resource controls: records response size
private/proxy-pipeline.js - Response handling → Response cache: stores eligible response
private/proxy-pipeline.js - GET / POST proxy → Payload and HEAD checks: watches response size
- Rendered-page scrape → Target URL utilities: gets target URL
private/proxy-pipeline.js - Request pipeline → Runtime limits: reads debug setting
private/proxy-pipeline.js - Payload and HEAD checks → Heavy-resource controls: imports controls
private/pipeline-utils.js - Payload and HEAD checks → Runtime limits: reads size limits
private/pipeline-utils.js - Heavy-resource controls → Runtime limits: reads limits
private/heat-map.js - Target URL utilities → Runtime limits: reads cache expiry
private/proxy-utils.js