# zennomi/pi-web-fetch architecture

> Architecture diagram of the GitHub repository zennomi/pi-web-fetch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zennomi/pi-web-fetch
- Repository: https://github.com/zennomi/pi-web-fetch

## Overview

Pi’s `fetch_web_md` extension turns a public URL into Markdown, preferring markdown.new and using a bounded static fallback after eligible upstream failures. The graph preserves the fallback’s documented discovery stages, conversion, URL safety, and output handling. `local-fetch.ts` and parts of the tool implementation were only sampled; fallback stage links follow README-documented behavior, while uncertain internal wiring is omitted.

Main entry points: [`fetch-web-md.ts`](https://github.com/zennomi/pi-web-fetch/blob/master/fetch-web-md.ts) (fetch_web_md tool), [`local-fetch.ts`](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts) (Fallback dispatcher), [`url-safety.ts`](https://github.com/zennomi/pi-web-fetch/blob/master/url-safety.ts) (URL validation), [`local-convert.ts`](https://github.com/zennomi/pi-web-fetch/blob/master/local-convert.ts) (Local conversion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Tool workflow"]
  node_tool["fetch_web_md tool<br/>[fetch-web-md.ts]"]
  node_upstream["markdown.new"]
end

subgraph group_fallback["Local fallback"]
  node_fallback_dispatch["Fallback dispatcher<br/>[local-fetch.ts]"]
  node_direct_md["Direct Markdown<br/>[local-fetch.ts]"]
  node_alternate_md["Alternate Markdown<br/>[local-fetch.ts]"]
  node_llms_discovery["llms.txt discovery<br/>[local-fetch.ts]"]
  node_github_sources["GitHub sources<br/>[local-fetch.ts]"]
  node_static_html["Static HTML<br/>[local-fetch.ts]"]
  node_github_api["GitHub REST API"]
end

subgraph group_safety["Safety and requests"]
  node_session["Bounded request session<br/>[local-fetch.ts]"]
  node_url_validation["URL validation<br/>[url-safety.ts]"]
  node_dns["DNS lookup and pinning<br/>[local-fetch.ts]"]
  node_ipaddr["IP address checks<br/>[url-safety.ts]"]
end

subgraph group_output["Conversion and output"]
  node_local_conversion["Local conversion<br/>[local-convert.ts]"]
  node_readability["Readability extraction<br/>[local-convert.ts]"]
  node_turndown["Turndown GFM<br/>[local-convert.ts]"]
  node_output_limits["Output truncation<br/>[fetch-web-md.ts]"]
  node_temp_file["Temporary Markdown file<br/>[fetch-web-md.ts]"]
end

node_pi(("Pi caller"))
node_web_hosts(("Public web hosts"))

node_pi -->|"invokes"| node_tool
node_tool -->|"validates URL"| node_url_validation
node_tool -->|"POSTs conversion"| node_upstream
node_tool -->|"starts fallback"| node_fallback_dispatch
node_fallback_dispatch -->|"fetches sources"| node_session
node_fallback_dispatch -->|"tries first"| node_direct_md
node_fallback_dispatch -->|"then tries"| node_alternate_md
node_fallback_dispatch -->|"then checks"| node_llms_discovery
node_fallback_dispatch -->|"then checks"| node_github_sources
node_fallback_dispatch -->|"then extracts"| node_static_html
node_session -->|"requests"| node_web_hosts
node_session -->|"validates redirects"| node_url_validation
node_session -->|"resolves and pins"| node_dns
node_dns -->|"checks addresses"| node_url_validation
node_url_validation -->|"checks IP ranges"| node_ipaddr
node_github_sources -.->|"uses public API"| node_github_api
node_github_sources -->|"reads public pages"| node_web_hosts
node_tool -->|"converts fallback"| node_local_conversion
node_local_conversion -->|"extracts articles"| node_readability
node_local_conversion -->|"renders Markdown"| node_turndown
node_tool -->|"truncates result"| node_output_limits
node_tool -->|"saves full output"| node_temp_file

click node_tool "https://github.com/zennomi/pi-web-fetch/blob/master/fetch-web-md.ts"
click node_fallback_dispatch "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_direct_md "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_alternate_md "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_llms_discovery "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_github_sources "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_static_html "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_session "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_url_validation "https://github.com/zennomi/pi-web-fetch/blob/master/url-safety.ts"
click node_dns "https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts"
click node_ipaddr "https://github.com/zennomi/pi-web-fetch/blob/master/url-safety.ts"
click node_local_conversion "https://github.com/zennomi/pi-web-fetch/blob/master/local-convert.ts"
click node_readability "https://github.com/zennomi/pi-web-fetch/blob/master/local-convert.ts"
click node_turndown "https://github.com/zennomi/pi-web-fetch/blob/master/local-convert.ts"
click node_output_limits "https://github.com/zennomi/pi-web-fetch/blob/master/fetch-web-md.ts"
click node_temp_file "https://github.com/zennomi/pi-web-fetch/blob/master/fetch-web-md.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tool,node_upstream toneBlue
class node_fallback_dispatch,node_direct_md,node_alternate_md,node_llms_discovery,node_github_sources,node_static_html,node_github_api toneAmber
class node_session,node_url_validation,node_dns,node_ipaddr toneMint
class node_local_conversion,node_readability,node_turndown,node_output_limits,node_temp_file toneRose
class node_pi,node_web_hosts toneIndigo
```

## Components

### Tool workflow

- [fetch_web_md tool](https://github.com/zennomi/pi-web-fetch/blob/master/fetch-web-md.ts): `fetch-web-md.ts`
- **markdown.new**

### Local fallback

- [Fallback dispatcher](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- [Direct Markdown](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- [Alternate Markdown](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- [llms.txt discovery](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- [GitHub sources](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- [Static HTML](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- **GitHub REST API**

### Safety and requests

- [Bounded request session](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- [URL validation](https://github.com/zennomi/pi-web-fetch/blob/master/url-safety.ts): `url-safety.ts`
- [DNS lookup and pinning](https://github.com/zennomi/pi-web-fetch/blob/master/local-fetch.ts): `local-fetch.ts`
- [IP address checks](https://github.com/zennomi/pi-web-fetch/blob/master/url-safety.ts): `url-safety.ts`

### Conversion and output

- [Local conversion](https://github.com/zennomi/pi-web-fetch/blob/master/local-convert.ts): `local-convert.ts`
- [Readability extraction](https://github.com/zennomi/pi-web-fetch/blob/master/local-convert.ts): `local-convert.ts`
- [Turndown GFM](https://github.com/zennomi/pi-web-fetch/blob/master/local-convert.ts): `local-convert.ts`
- [Output truncation](https://github.com/zennomi/pi-web-fetch/blob/master/fetch-web-md.ts): `fetch-web-md.ts`
- [Temporary Markdown file](https://github.com/zennomi/pi-web-fetch/blob/master/fetch-web-md.ts): `fetch-web-md.ts`

### Other components

- **Pi caller**
- **Public web hosts**

## Connections

- Pi caller → fetch_web_md tool: invokes
- fetch_web_md tool → URL validation: validates URL
- fetch_web_md tool → markdown.new: POSTs conversion
- fetch_web_md tool → Fallback dispatcher: starts fallback
- Fallback dispatcher → Bounded request session: fetches sources
- Fallback dispatcher → Direct Markdown: tries first
- Fallback dispatcher → Alternate Markdown: then tries
- Fallback dispatcher → llms.txt discovery: then checks
- Fallback dispatcher → GitHub sources: then checks
- Fallback dispatcher → Static HTML: then extracts
- Bounded request session → Public web hosts: requests
- Bounded request session → URL validation: validates redirects
- Bounded request session → DNS lookup and pinning: resolves and pins
- DNS lookup and pinning → URL validation: checks addresses
- URL validation → IP address checks: checks IP ranges
- GitHub sources → GitHub REST API: uses public API
- GitHub sources → Public web hosts: reads public pages
- fetch_web_md tool → Local conversion: converts fallback
- Local conversion → Readability extraction: extracts articles
- Local conversion → Turndown GFM: renders Markdown
- fetch_web_md tool → Output truncation: truncates result
- fetch_web_md tool → Temporary Markdown file: saves full output