# frankoslaw/longscrape architecture

> Architecture diagram of the GitHub repository frankoslaw/longscrape, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/frankoslaw/longscrape
- Repository: https://github.com/frankoslaw/longscrape

## Overview

Longscrape is a small Python scraping library. A caller supplies a URL to the HTTP fetcher, which requests the page and returns a Document. The package also defines Fetcher and Extractor protocols; Extractor describes turning documents into records, but no extractor implementation or end-to-end orchestration is present in the sampled source. The README provides no further product capabilities, so the graph reflects only these supported behaviors.

Main entry points: [`packages/longscrape/src/longscrape/fetchers/httpx_fetcher.py`](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/fetchers/httpx_fetcher.py) (HTTP fetcher), [`packages/longscrape/src/longscrape/protocols.py`](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/protocols.py) (Extractor protocol), [`packages/longscrape/src/longscrape/models.py`](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/models.py) (Fetched document)

## Diagram (Mermaid)

```mermaid
flowchart TD

node_caller(("Library caller"))
node_httpx_fetcher["HTTP fetcher<br/>[httpx_fetcher.py]"]
node_fetcher_protocol["Fetcher protocol<br/>[protocols.py]"]
node_extractor_protocol["Extractor protocol<br/>[protocols.py]"]
node_input_url["URL input<br/>[models.py]"]
node_document["Fetched document<br/>[models.py]"]
node_record["Extracted record<br/>[models.py]"]
node_http_service(("HTTP server"))

node_caller -->|"calls fetch"| node_httpx_fetcher
node_caller -->|"supplies URL"| node_input_url
node_httpx_fetcher -->|"implements"| node_fetcher_protocol
node_httpx_fetcher -->|"requests page"| node_http_service
node_http_service -->|"returns response"| node_httpx_fetcher
node_httpx_fetcher -->|"returns document"| node_document
node_extractor_protocol -->|"accepts document"| node_document
node_extractor_protocol -->|"yields records"| node_record

click node_httpx_fetcher "https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/fetchers/httpx_fetcher.py"
click node_fetcher_protocol "https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/protocols.py"
click node_extractor_protocol "https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/protocols.py"
click node_input_url "https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/models.py"
click node_document "https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/models.py"
click node_record "https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_caller toneIndigo
class node_httpx_fetcher,node_fetcher_protocol,node_extractor_protocol,node_input_url,node_document,node_record toneTeal
class node_http_service toneMint
```

## Components

### Other components

- **Library caller**
- [HTTP fetcher](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/fetchers/httpx_fetcher.py): `packages/longscrape/src/longscrape/fetchers/httpx_fetcher.py`
- [Fetcher protocol](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/protocols.py): `packages/longscrape/src/longscrape/protocols.py`
- [Extractor protocol](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/protocols.py): `packages/longscrape/src/longscrape/protocols.py`
- [URL input](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/models.py): `packages/longscrape/src/longscrape/models.py`
- [Fetched document](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/models.py): `packages/longscrape/src/longscrape/models.py`
- [Extracted record](https://github.com/frankoslaw/longscrape/blob/main/packages/longscrape/src/longscrape/models.py): `packages/longscrape/src/longscrape/models.py`
- **HTTP server**

## Connections

- Library caller → HTTP fetcher: calls fetch
- Library caller → URL input: supplies URL
- HTTP fetcher → Fetcher protocol: implements
- HTTP fetcher → HTTP server: requests page
- HTTP server → HTTP fetcher: returns response
- HTTP fetcher → Fetched document: returns document
- Extractor protocol → Fetched document: accepts document
- Extractor protocol → Extracted record: yields records