# monapdx/bdsmlr-archive-builder architecture

> Architecture diagram of the GitHub repository monapdx/bdsmlr-archive-builder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/monapdx/bdsmlr-archive-builder
- Repository: https://github.com/monapdx/bdsmlr-archive-builder

## Overview

1. **What this repository is**
   - `bdsmlr-archive` is a small Python command-line tool that turns exported BDSMLR JSON into a **static HTML archive**.
   - The architecture is deliberately simple: **input JSON files in, static files out**, with optional network fetches for media mirroring.

2. **Primary runtime and packaging**
   - The project is a **Python 3.9+** package under `src/bdsmlr_archive/`.
   - The key entrypoint is the CLI module, which likely exposes the `bdsmlr-archive build` command described in the README.
   - `pyproject.toml` defines the packaging and installable command-line interface.

3. **Main subsystems**
   - **CLI layer (`cli.py`)**: command parsing, defaults, and user-facing options like input paths, output path, media mode, retries, and timeouts.
   - **Build/orchestration layer (`builder.py`)**: reads JSON exports, normalizes post data, coordinates media handling, and writes the archive.
   - **Presentation layer (`templates.py`)**: HTML template generation for the index page, per-post pages, and likely supporting structures.
   - **Package init (`__init__.py`)**: package marker and possibly version metadata.

4. **End-to-end data flow**
   - Source data begins in `exports/*.json`, or explicit files passed on the CLI.
   - The builder ingests these JSON arrays of post dictionaries, extracts known fields, and ignores unknown keys safely.
   - It then produces a structured output tree in `output/`, including `index.html`, `posts/`, `media/` when enabled, and `data/` JSON/JS assets for client-side browsing.

5. **Archive generation boundary**
   - The output is **fully static**: no server, no database, no runtime backend.
   - The browser consumes the generated HTML plus embedded or referenced data files, so the archive can be opened directly from disk.
   - This is the main architectural boundary: a one-way transformation from exported records to static artifacts.

6. **Media handling as an optional network boundary**
   - Media is the only part that crosses into external network access.
   - The tool supports three modes:
     - **local download/mirroring** for offline use,
     - **remote embedding** for hotlinking,
     - **metadata-only** with no media.
   - When downloading, the builder handles retries, timeouts, delays, Referer/Origin headers, and URL rewriting to local hashed files.

7. **Generated site structure**
   - The archive is not a single HTML file; it is a small static site.
   - The README shows separate layers:
     - a top-level `index.html`,
     - per-post pages under `posts/`,
     - generated data under `data/` such as `posts.json`, `tags.json`, `provenance.json`, `media_map.json`, and a `posts.js` file for client-side use.
   - This indicates a split between **rendered HTML pages** and **machine-readable data assets** for search/filtering and provenance.

8. **Client-side behavior**
   - Search and tag filtering are part of the generated archive, but there is no frontend framework in the repo.
   - That suggests the archive uses lightweight static HTML/JS assets generated by the builder rather than React/Vue/etc.
   - The browser-side interactivity is therefore a downstream consumer of the generated data files, not a separate application.

9. **Provenance and archival integrity**
   - The tool explicitly captures provenance metadata.
   - That makes the archive reproducible and inspectable: the generated data files can record where content came from and how it was transformed.
   - This is important for an architecture diagram because provenance is a distinct concern from rendering.

10. **Input assumptions and parsing model**
   - The tool is built around Easy Scraper-style JSON exports from BDSMLR pages.
   - It expects keys such as post links, dates, image URLs, tags, and comment-like lines, but the parser is tolerant of extra fields.
   - So the input contract is **semi-structured JSON**, not a rigid schema or database dump.

11. **Operational defaults**
   - The default behavior is optimized for non-technical use: no arguments means read from `./exports/` and write to `./output/`.
   - Media mirroring is the default recommendation because the archive is intended to work offline.
   - `run_archive.bat` exists as a Windows convenience wrapper for launching the build flow.

12. **Architectural summary for a diagram**
   - At a high level, the system is:
     - **Input**: exported JSON files
     - **Processing**: Python CLI + builder + templates
     - **Optional external dependency**: remote media hosts/CDNs
     - **Output**: static HTML site + local data files + optional mirrored media
   - The key design idea is an **offline-first static publishing pipeline** with optional network retrieval only for asset completeness.

Main entry points: [`src/bdsmlr_archive/builder.py`](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/builder.py) (Builder), [`output`](https://github.com/monapdx/bdsmlr-archive-builder/tree/main/output) (Static archive), [`src/bdsmlr_archive/templates.py`](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/templates.py) (Templates), [`src/bdsmlr_archive/cli.py`](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/cli.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_input["Input sources"]
  node_node_exports["Exports JSON<br/>input dataset<br/>[data.json]"]
end

subgraph group_grp_pkg["Python package"]
  node_node_cli["CLI<br/>command entrypoint<br/>[cli.py]"]
  node_node_builder["Builder<br/>orchestrator<br/>[builder.py]"]
  node_node_templates["Templates<br/>html rendering<br/>[templates.py]"]
  node_node_init["Package init<br/>package metadata<br/>[__init__.py]"]
  node_node_pyproject["Pyproject<br/>packaging config<br/>[pyproject.toml]"]
end

subgraph group_grp_output["Static archive"]
  node_node_index["Index page<br/>static HTML<br/>[1.txt]"]
  node_node_post_pages["Post pages<br/>static HTML"]
  node_node_data_posts["Posts data<br/>json assets"]
  node_node_data_tags["Tags data<br/>json assets"]
  node_node_data_provenance["Provenance<br/>archive metadata"]
  node_node_data_media["Media map<br/>asset manifest"]
  node_node_posts_js["Posts JS<br/>client data"]
  node_node_media[("Mirrored media<br/>asset cache")]
  node_node_site["Static archive<br/>publishable site"]
end

subgraph group_grp_external["External media hosts"]
  node_node_remote(("Remote hosts<br/>network source"))
end

node_node_pyproject -->|"exposes command"| node_node_cli
node_node_cli -->|"invokes"| node_node_builder
node_node_exports -->|"provides input"| node_node_builder
node_node_builder -->|"renders with"| node_node_templates
node_node_builder -->|"writes"| node_node_data_posts
node_node_builder -->|"writes"| node_node_data_tags
node_node_builder -->|"records"| node_node_data_provenance
node_node_builder -->|"maps"| node_node_data_media
node_node_builder -->|"emits"| node_node_posts_js
node_node_templates -->|"creates"| node_node_index
node_node_templates -->|"creates"| node_node_post_pages
node_node_builder -->|"downloads"| node_node_media
node_node_media -.->|"fetches from"| node_node_remote
node_node_index -->|"part of"| node_node_site
node_node_post_pages -->|"part of"| node_node_site
node_node_data_posts -->|"part of"| node_node_site
node_node_data_tags -->|"part of"| node_node_site
node_node_data_provenance -->|"part of"| node_node_site
node_node_data_media -->|"part of"| node_node_site
node_node_posts_js -->|"part of"| node_node_site
node_node_media -->|"part of"| node_node_site

click node_node_exports "https://github.com/monapdx/bdsmlr-archive-builder/blob/main/exports/data.json"
click node_node_cli "https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/cli.py"
click node_node_builder "https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/builder.py"
click node_node_templates "https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/templates.py"
click node_node_init "https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/__init__.py"
click node_node_pyproject "https://github.com/monapdx/bdsmlr-archive-builder/blob/main/pyproject.toml"
click node_node_index "https://github.com/monapdx/bdsmlr-archive-builder/blob/main/output/1.txt"
click node_node_site "https://github.com/monapdx/bdsmlr-archive-builder/tree/main/output"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_exports toneBlue
class node_node_cli,node_node_builder,node_node_templates,node_node_init,node_node_pyproject toneAmber
class node_node_index,node_node_post_pages,node_node_data_posts,node_node_data_tags,node_node_data_provenance,node_node_data_media,node_node_posts_js,node_node_media,node_node_site toneMint
class node_node_remote toneRose
```

## Components

### Input sources

Semi-structured JSON exports consumed by the builder.

- [Exports JSON](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/exports/data.json): input dataset · BDSMLR export files under exports/ and explicit JSON inputs passed to the CLI. · `exports/data.json`

### Python package

Installable CLI and archive-generation code under src/bdsmlr_archive.

- [CLI](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/cli.py): command entrypoint · Parses build options, defaults, and user-facing arguments. · `src/bdsmlr_archive/cli.py`
- [Builder](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/builder.py): orchestrator · Loads posts, normalizes records, coordinates media handling, and writes the archive. · `src/bdsmlr_archive/builder.py`
- [Templates](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/templates.py): html rendering · Generates the static HTML views and supporting page structures. · `src/bdsmlr_archive/templates.py`
- [Package init](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/src/bdsmlr_archive/__init__.py): package metadata · Package marker and version-facing module entry. · `src/bdsmlr_archive/__init__.py`
- [Pyproject](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/pyproject.toml): packaging config · Defines installation metadata and the console script entrypoint. · `pyproject.toml`

### Static archive

Generated offline site and data assets written to the output tree.

- [Index page](https://github.com/monapdx/bdsmlr-archive-builder/blob/main/output/1.txt): static HTML · Top-level archive landing page for browsing and search. · `output/1.txt`
- **Post pages**: static HTML · Per-post pages generated into the archive tree.
- **Posts data**: json assets · Structured post records for client-side browsing and filtering.
- **Tags data**: json assets · Tag index used by the generated archive UI.
- **Provenance**: archive metadata · Captures source and transformation metadata for the archive.
- **Media map**: asset manifest · Maps remote media references to local mirrored files or remote URLs.
- **Posts JS**: client data · JavaScript data bundle consumed by the static browser UI.
- **Mirrored media**: asset cache · Optional locally downloaded media files for offline viewing.
- [Static archive](https://github.com/monapdx/bdsmlr-archive-builder/tree/main/output): publishable site · Fully static HTML archive that can be opened directly from disk. · `output`

### External media hosts

Optional network boundary for fetching or hotlinking media.

- **Remote hosts**: network source · External media CDNs or origin servers reached only when mirroring or hotlinking.

## Connections

- Pyproject → CLI: exposes command · Packaging defines the installable console entrypoint.
- CLI → Builder: invokes · CLI hands parsed options and defaults to the build pipeline.
- Exports JSON → Builder: provides input · Exported JSON files are read and normalized into post records.
- Builder → Templates: renders with · Builder asks the template layer to create HTML pages.
- Builder → Posts data: writes · Structured post data is emitted for the static client UI.
- Builder → Tags data: writes · Tag index data is emitted alongside the archive.
- Builder → Provenance: records · Source and transformation provenance are captured as generated metadata.
- Builder → Media map: maps · Media URLs are rewritten or recorded for local or remote use.
- Builder → Posts JS: emits · Client-side data bundle is generated for browsing and filtering.
- Templates → Index page: creates · Generates the archive landing page.
- Templates → Post pages: creates · Generates per-post HTML pages.
- Builder → Mirrored media: downloads · Optional mirroring stores media in local archive assets.
- Mirrored media → Remote hosts: fetches from · Media retrieval crosses the network boundary when enabled.
- Index page → Static archive: part of · Rendered HTML is assembled into the static archive.
- Post pages → Static archive: part of · Per-post pages are included in the generated site.
- Posts data → Static archive: part of · Data assets ship with the static archive.
- Tags data → Static archive: part of · Tag data ships with the static archive.
- Provenance → Static archive: part of · Provenance metadata ships with the static archive.
- Media map → Static archive: part of · Media mapping data ships with the static archive.
- Posts JS → Static archive: part of · Client-side data bundle ships with the static archive.
- Mirrored media → Static archive: part of · Mirrored media becomes part of the offline archive.
