# hernehunter/sitemap architecture

> Architecture diagram of the GitHub repository hernehunter/sitemap, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hernehunter/sitemap
- Repository: https://github.com/hernehunter/sitemap

## Overview

This Go library parses sitemap documents from readers, files, or URLs and streams page entries as ParseResult values. Fetching can recursively follow sitemap indexes, with concurrency limits and separate filters for index and page URLs. Parsing uses the standard XML decoder by default or an opt-in custom lexer; selected fields, date cutoffs, Jalali dates, and regex filtering support the workflow. The graph reflects README capabilities and shown calls; unsampled modules are included where documented, but their internal wiring is not inferred.

Main entry points: [`sitemap.go`](https://github.com/hernehunter/sitemap/blob/main/sitemap.go) (Parse and Fetch API), [`parsefast.go`](https://github.com/hernehunter/sitemap/blob/main/parsefast.go) (Custom Lexer Parser), [`parser.go`](https://github.com/hernehunter/sitemap/blob/main/parser.go) (Parser Dispatch), [`parsesafe.go`](https://github.com/hernehunter/sitemap/blob/main/parsesafe.go) (Standard XML Parser)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Streaming API"]
  node_stream_api["Parse and Fetch API<br/>[sitemap.go]"]
  node_file_input["File Input<br/>[sitemap.go]"]
  node_result_stream["Result Channel<br/>[sitemap.go]"]
end

subgraph group_fetch["Fetch and Filter"]
  node_fetch_flow["Recursive Fetch<br/>[sitemap.go]"]
  node_http_fetch["HTTP Fetch<br/>[fetcher.go]"]
  node_filtering["URL Filters<br/>[filter.go]"]
end

subgraph group_parse["XML Parsing"]
  node_parser_dispatch["Parser Dispatch<br/>[parser.go]"]
  node_safe_parser["Standard XML Parser<br/>[parsesafe.go]"]
  node_fast_parser["Custom Lexer Parser<br/>[parsefast.go]"]
  node_xml_lexer["XML Tokenizer<br/>[lexer.go]"]
  node_entity_decode["Entity Decoding<br/>[entity.go]"]
end

subgraph group_data["Fields and Dates"]
  node_field_matching["Field Selection<br/>[fields.go]"]
  node_entry_buffers["Entry Buffers<br/>[buffers.go]"]
  node_time_parse["Date Parsing<br/>[time.go]"]
  node_result_types["Results and Errors<br/>[types.go]"]
  node_error_types["Error Definitions<br/>[errors.go]"]
end

node_caller(("Library Caller"))
node_consumer(("Result Consumer"))
node_http_service(("Sitemap Host"))

node_caller -->|"calls"| node_stream_api
node_stream_api -->|"opens file"| node_file_input
node_stream_api -->|"schedules fetch"| node_fetch_flow
node_stream_api -->|"streams results"| node_result_stream
node_result_stream -->|"yields values"| node_consumer
node_fetch_flow -->|"fetches URL"| node_http_fetch
node_http_service -->|"serves sitemap"| node_http_fetch
node_fetch_flow -->|"parses reader"| node_parser_dispatch
node_fetch_flow -->|"recurses indexes"| node_fetch_flow
node_fetch_flow -->|"checks URLs"| node_filtering
node_parser_dispatch -->|"defaults to"| node_safe_parser
node_parser_dispatch -->|"dispatches option"| node_fast_parser
node_safe_parser -->|"fills buffers"| node_entry_buffers
node_safe_parser -->|"matches fields"| node_field_matching
node_fast_parser -->|"reads tokens"| node_xml_lexer
node_fast_parser -->|"fills buffers"| node_entry_buffers
node_fast_parser -->|"matches fields"| node_field_matching
node_xml_lexer -->|"decodes entities"| node_entity_decode
node_entry_buffers -->|"parses lastmod"| node_time_parse
node_stream_api -->|"returns results"| node_result_types
node_stream_api -->|"reports errors"| node_error_types

click node_stream_api "https://github.com/hernehunter/sitemap/blob/main/sitemap.go"
click node_file_input "https://github.com/hernehunter/sitemap/blob/main/sitemap.go"
click node_result_stream "https://github.com/hernehunter/sitemap/blob/main/sitemap.go"
click node_fetch_flow "https://github.com/hernehunter/sitemap/blob/main/sitemap.go"
click node_http_fetch "https://github.com/hernehunter/sitemap/blob/main/fetcher.go"
click node_filtering "https://github.com/hernehunter/sitemap/blob/main/filter.go"
click node_parser_dispatch "https://github.com/hernehunter/sitemap/blob/main/parser.go"
click node_safe_parser "https://github.com/hernehunter/sitemap/blob/main/parsesafe.go"
click node_fast_parser "https://github.com/hernehunter/sitemap/blob/main/parsefast.go"
click node_xml_lexer "https://github.com/hernehunter/sitemap/blob/main/lexer.go"
click node_entity_decode "https://github.com/hernehunter/sitemap/blob/main/entity.go"
click node_field_matching "https://github.com/hernehunter/sitemap/blob/main/fields.go"
click node_entry_buffers "https://github.com/hernehunter/sitemap/blob/main/buffers.go"
click node_time_parse "https://github.com/hernehunter/sitemap/blob/main/time.go"
click node_result_types "https://github.com/hernehunter/sitemap/blob/main/types.go"
click node_error_types "https://github.com/hernehunter/sitemap/blob/main/errors.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_stream_api,node_file_input,node_result_stream toneBlue
class node_fetch_flow,node_http_fetch,node_filtering toneAmber
class node_parser_dispatch,node_safe_parser,node_fast_parser,node_xml_lexer,node_entity_decode toneMint
class node_field_matching,node_entry_buffers,node_time_parse,node_result_types,node_error_types toneRose
class node_caller,node_consumer,node_http_service toneIndigo
```

## Components

### Streaming API

- [Parse and Fetch API](https://github.com/hernehunter/sitemap/blob/main/sitemap.go): `sitemap.go`
- [File Input](https://github.com/hernehunter/sitemap/blob/main/sitemap.go): `sitemap.go`
- [Result Channel](https://github.com/hernehunter/sitemap/blob/main/sitemap.go): `sitemap.go`

### Fetch and Filter

- [Recursive Fetch](https://github.com/hernehunter/sitemap/blob/main/sitemap.go): `sitemap.go`
- [HTTP Fetch](https://github.com/hernehunter/sitemap/blob/main/fetcher.go): `fetcher.go`
- [URL Filters](https://github.com/hernehunter/sitemap/blob/main/filter.go): `filter.go`

### XML Parsing

- [Parser Dispatch](https://github.com/hernehunter/sitemap/blob/main/parser.go): `parser.go`
- [Standard XML Parser](https://github.com/hernehunter/sitemap/blob/main/parsesafe.go): `parsesafe.go`
- [Custom Lexer Parser](https://github.com/hernehunter/sitemap/blob/main/parsefast.go): `parsefast.go`
- [XML Tokenizer](https://github.com/hernehunter/sitemap/blob/main/lexer.go): `lexer.go`
- [Entity Decoding](https://github.com/hernehunter/sitemap/blob/main/entity.go): `entity.go`

### Fields and Dates

- [Field Selection](https://github.com/hernehunter/sitemap/blob/main/fields.go): `fields.go`
- [Entry Buffers](https://github.com/hernehunter/sitemap/blob/main/buffers.go): `buffers.go`
- [Date Parsing](https://github.com/hernehunter/sitemap/blob/main/time.go): `time.go`
- [Results and Errors](https://github.com/hernehunter/sitemap/blob/main/types.go): `types.go`
- [Error Definitions](https://github.com/hernehunter/sitemap/blob/main/errors.go): `errors.go`

### Other components

- **Library Caller**
- **Result Consumer**
- **Sitemap Host**

## Connections

- Library Caller → Parse and Fetch API: calls (evidence: `README.md`)
- Parse and Fetch API → File Input: opens file (evidence: `sitemap.go`)
- Parse and Fetch API → Recursive Fetch: schedules fetch (evidence: `sitemap.go`)
- Parse and Fetch API → Result Channel: streams results (evidence: `sitemap.go`)
- Result Channel → Result Consumer: yields values (evidence: `README.md`)
- Recursive Fetch → HTTP Fetch: fetches URL (evidence: `sitemap.go`)
- Sitemap Host → HTTP Fetch: serves sitemap (evidence: `README.md`)
- Recursive Fetch → Parser Dispatch: parses reader (evidence: `sitemap.go`)
- Recursive Fetch → Recursive Fetch: recurses indexes (evidence: `sitemap.go`)
- Recursive Fetch → URL Filters: checks URLs (evidence: `sitemap.go`)
- Parser Dispatch → Standard XML Parser: defaults to (evidence: `parser.go`)
- Parser Dispatch → Custom Lexer Parser: dispatches option (evidence: `parser.go`)
- Standard XML Parser → Entry Buffers: fills buffers (evidence: `parsesafe.go`)
- Standard XML Parser → Field Selection: matches fields (evidence: `parsesafe.go`)
- Custom Lexer Parser → XML Tokenizer: reads tokens (evidence: `parsefast.go`)
- Custom Lexer Parser → Entry Buffers: fills buffers (evidence: `parsefast.go`)
- Custom Lexer Parser → Field Selection: matches fields (evidence: `parsefast.go`)
- XML Tokenizer → Entity Decoding: decodes entities (evidence: `entity.go`)
- Entry Buffers → Date Parsing: parses lastmod (evidence: `buffers.go`)
- Parse and Fetch API → Results and Errors: returns results (evidence: `README.md`)
- Parse and Fetch API → Error Definitions: reports errors (evidence: `README.md`)
