How monapdx/bdsmlr-archive-builder works
1. What this repository is
- bdsmlr-archive is a small Python command-line tool that turns exported BDSMLR JSON into a static HTML archive.
- The architecture is deliberately simple: input JSON files in, static files out, with optional network fetches for media mirroring.
2. Primary runtime and packaging
- The project is a Python 3.9+ package under src/bdsmlr_archive/.
- The key entrypoint is the CLI module, which likely exposes the bdsmlr-archive build command described in the README.
- pyproject.toml defines the packaging and installable command-line interface.
3. Main subsystems
- CLI layer (cli.py): command parsing, defaults, and user-facing options like input paths, output path, media mode, retries, and timeouts.
- Build/orchestration layer (builder.py): reads JSON exports, normalizes post data, coordinates media handling, and writes the archive.
- Presentation layer (templates.py): HTML template generation for the index page, per-post pages, and likely supporting structures.
- Package init (__init__.py): package marker and possibly version metadata.
4. End-to-end data flow
- Source data begins in exports/*.json, or explicit files passed on the CLI.
- The builder ingests these JSON arrays of post dictionaries, extracts known fields, and ignores unknown keys safely.
- It then produces a structured output tree in output/, including index.html, posts/, media/ when enabled, and data/ JSON/JS assets for client-side browsing.
5. Archive generation boundary
- The output is fully static: no server, no database, no runtime backend.
- The browser consumes the generated HTML plus embedded or referenced data files, so the archive can be opened directly from disk.
- This is the main architectural boundary: a one-way transformation from exported records to static artifacts.
6. Media handling as an optional network boundary
- Media is the only part that crosses into external network access.
- The tool supports three modes:
- local download/mirroring for offline use,
- remote embedding for hotlinking,
- metadata-only with no media.
- When downloading, the builder handles retries, timeouts, delays, Referer/Origin headers, and URL rewriting to local hashed files.
7. Generated site structure
- The archive is not a single HTML file; it is a small static site.
- The README shows separate layers:
- a top-level index.html,
- per-post pages under posts/,
- generated data under data/ such as posts.json, tags.json, provenance.json, media_map.json, and a posts.js file for client-side use.
- This indicates a split between rendered HTML pages and machine-readable data assets for search/filtering and provenance.
8. Client-side behavior
- Search and tag filtering are part of the generated archive, but there is no frontend framework in the repo.
- That suggests the archive uses lightweight static HTML/JS assets generated by the builder rather than React/Vue/etc.
- The browser-side interactivity is therefore a downstream consumer of the generated data files, not a separate application.
9. Provenance and archival integrity
- The tool explicitly captures provenance metadata.
- That makes the archive reproducible and inspectable: the generated data files can record where content came from and how it was transformed.
- This is important for an architecture diagram because provenance is a distinct concern from rendering.
10. Input assumptions and parsing model
- The tool is built around Easy Scraper-style JSON exports from BDSMLR pages.
- It expects keys such as post links, dates, image URLs, tags, and comment-like lines, but the parser is tolerant of extra fields.
- So the input contract is semi-structured JSON, not a rigid schema or database dump.
11. Operational defaults
- The default behavior is optimized for non-technical use: no arguments means read from ./exports/ and write to ./output/.
- Media mirroring is the default recommendation because the archive is intended to work offline.
- run_archive.bat exists as a Windows convenience wrapper for launching the build flow.
12. Architectural summary for a diagram
- At a high level, the system is:
- Input: exported JSON files
- Processing: Python CLI + builder + templates
- Optional external dependency: remote media hosts/CDNs
- Output: static HTML site + local data files + optional mirrored media
- The key design idea is an offline-first static publishing pipeline with optional network retrieval only for asset completeness.
Main entry points: builder.py Builder · output Static archive · templates.py Templates · cli.py CLI
Components 16
Input sources
Semi-structured JSON exports consumed by the builder.
- Exports JSON · BDSMLR export files under exports/ and explicit JSON inputs passed to the CLI.
exports/data.json
Python package
Installable CLI and archive-generation code under src/bdsmlr_archive.
- CLI · Parses build options, defaults, and user-facing arguments.
src/bdsmlr_archive/cli.py - Builder · Loads posts, normalizes records, coordinates media handling, and writes the archive.
src/bdsmlr_archive/builder.py - Templates · Generates the static HTML views and supporting page structures.
src/bdsmlr_archive/templates.py - Package init · Package marker and version-facing module entry.
src/bdsmlr_archive/__init__.py - Pyproject · Defines installation metadata and the console script entrypoint.
pyproject.toml
Static archive
Generated offline site and data assets written to the output tree.
- Index page · Top-level archive landing page for browsing and search.
output/1.txt - Post pages · Per-post pages generated into the archive tree.
- Posts data · Structured post records for client-side browsing and filtering.
- Tags data · Tag index used by the generated archive UI.
- Provenance · Captures source and transformation metadata for the archive.
- Media map · Maps remote media references to local mirrored files or remote URLs.
- Posts JS · JavaScript data bundle consumed by the static browser UI.
- Mirrored media · Optional locally downloaded media files for offline viewing.
- Static archive · Fully static HTML archive that can be opened directly from disk.
output
External media hosts
Optional network boundary for fetching or hotlinking media.
- Remote hosts · External media CDNs or origin servers reached only when mirroring or hotlinking.
Connections 21
- Pyproject → CLI: exposes command · Packaging defines the installable console entrypoint.
- CLI → Builder: invokes · CLI hands parsed options and defaults to the build pipeline.
- Exports JSON → Builder: provides input · Exported JSON files are read and normalized into post records.
- Builder → Templates: renders with · Builder asks the template layer to create HTML pages.
- Builder → Posts data: writes · Structured post data is emitted for the static client UI.
- Builder → Tags data: writes · Tag index data is emitted alongside the archive.
- Builder → Provenance: records · Source and transformation provenance are captured as generated metadata.
- Builder → Media map: maps · Media URLs are rewritten or recorded for local or remote use.
- Builder → Posts JS: emits · Client-side data bundle is generated for browsing and filtering.
- Templates → Index page: creates · Generates the archive landing page.
- Templates → Post pages: creates · Generates per-post HTML pages.
- Builder → Mirrored media: downloads · Optional mirroring stores media in local archive assets.
- Mirrored media → Remote hosts: fetches from · Media retrieval crosses the network boundary when enabled.
- Index page → Static archive: part of · Rendered HTML is assembled into the static archive.
- Post pages → Static archive: part of · Per-post pages are included in the generated site.
- Posts data → Static archive: part of · Data assets ship with the static archive.
- Tags data → Static archive: part of · Tag data ships with the static archive.
- Provenance → Static archive: part of · Provenance metadata ships with the static archive.
- Media map → Static archive: part of · Media mapping data ships with the static archive.
- Posts JS → Static archive: part of · Client-side data bundle ships with the static archive.
- Mirrored media → Static archive: part of · Mirrored media becomes part of the offline archive.