How zennomi/pi-web-fetch works
Pi’s fetch_web_md extension turns a public URL into Markdown, preferring markdown.new and using a bounded static fallback after eligible upstream failures. The graph preserves the fallback’s documented discovery stages, conversion, URL safety, and output handling. local-fetch.ts and parts of the tool implementation were only sampled; fallback stage links follow README-documented behavior, while uncertain internal wiring is omitted.
Main entry points: fetch-web-md.ts fetch_web_md tool · local-fetch.ts Fallback dispatcher · url-safety.ts URL validation · local-convert.ts Local conversion
Components 20
Tool workflow
- fetch_web_md tool
fetch-web-md.ts - markdown.new
Local fallback
- Fallback dispatcher
local-fetch.ts - Direct Markdown
local-fetch.ts - Alternate Markdown
local-fetch.ts - llms.txt discovery
local-fetch.ts - GitHub sources
local-fetch.ts - Static HTML
local-fetch.ts - GitHub REST API
Safety and requests
- Bounded request session
local-fetch.ts - URL validation
url-safety.ts - DNS lookup and pinning
local-fetch.ts - IP address checks
url-safety.ts
Conversion and output
- Local conversion
local-convert.ts - Readability extraction
local-convert.ts - Turndown GFM
local-convert.ts - Output truncation
fetch-web-md.ts - Temporary Markdown file
fetch-web-md.ts
Other components
- Pi caller
- Public web hosts
Connections 22
- Pi caller → fetch_web_md tool: invokes
- fetch_web_md tool → URL validation: validates URL
- fetch_web_md tool → markdown.new: POSTs conversion
- fetch_web_md tool → Fallback dispatcher: starts fallback
- Fallback dispatcher → Bounded request session: fetches sources
- Fallback dispatcher → Direct Markdown: tries first
- Fallback dispatcher → Alternate Markdown: then tries
- Fallback dispatcher → llms.txt discovery: then checks
- Fallback dispatcher → GitHub sources: then checks
- Fallback dispatcher → Static HTML: then extracts
- Bounded request session → Public web hosts: requests
- Bounded request session → URL validation: validates redirects
- Bounded request session → DNS lookup and pinning: resolves and pins
- DNS lookup and pinning → URL validation: checks addresses
- URL validation → IP address checks: checks IP ranges
- GitHub sources → GitHub REST API: uses public API
- GitHub sources → Public web hosts: reads public pages
- fetch_web_md tool → Local conversion: converts fallback
- Local conversion → Readability extraction: extracts articles
- Local conversion → Turndown GFM: renders Markdown
- fetch_web_md tool → Output truncation: truncates result
- fetch_web_md tool → Temporary Markdown file: saves full output