# jbiaojerry/ebook-treasure-chest architecture

> Architecture diagram of the GitHub repository jbiaojerry/ebook-treasure-chest, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jbiaojerry/ebook-treasure-chest
- Repository: https://github.com/jbiaojerry/ebook-treasure-chest

## Overview

1. **What this repo is**
   - This is a **static ebook catalog and search site**, not a traditional application server.
   - The published product lives in `docs/` and is intended for **GitHub Pages** hosting, with all search and navigation happening client-side.

2. **Primary content layer**
   - The large `md/` directory is the canonical content store.
   - Each Markdown file represents a **category/tag page** such as `文学.md`, `历史.md`, `Python.md`, etc., and contains curated ebook download links or entries for that topic.
   - The README’s “热门分类” section is effectively a human-readable summary of these category pages.

3. **Generated site output**
   - `docs/index.html` is the entry page for the hosted site.
   - `docs/search.js` powers the browser-side search experience.
   - `docs/books.json`, `docs/all-books.json`, and `docs/parse-stats.json` are generated data files consumed by the static site.
   - `docs/.nojekyll` indicates GitHub Pages is serving the folder as plain static content, without Jekyll processing.

4. **Search architecture**
   - Search is fully **client-side**.
   - The browser loads prebuilt JSON indexes from `docs/`, and `search.js` filters/ranks results in real time.
   - The README emphasizes **multi-keyword search**, so the search index is designed around fast local matching rather than server queries.

5. **Data transformation pipeline**
   - `scripts/parse_md_to_json.py` is the main bridge from Markdown content to structured JSON.
   - `scripts/generate_index.py` likely turns the category corpus into the searchable homepage/index structure.
   - `docs/parse-stats.json` suggests the generation step also computes content statistics for display or validation.

6. **Content sync pipeline**
   - `scripts/sync/` contains the ingestion/update machinery:
     - `sync_all_books.py` for full corpus refreshes
     - `incremental_sync.py` for partial updates
     - `find_max_book_id.py` and `max_book_id_found.txt` for checkpointing incremental state
     - `parse_book_detail_enhanced.py` for extracting richer book metadata
     - `backup_md.py` for preserving existing Markdown before overwrites
   - This is the boundary where external ebook source data is normalized into the repo’s markdown format.

7. **Automation and build orchestration**
   - `.github/workflows/full-sync.yml` handles complete refresh runs.
   - `.github/workflows/incremental-sync.yml` handles ongoing deltas.
   - `.github/workflows/generate-site.yml` rebuilds/publishes the static site artifacts.
   - The architecture is therefore **GitHub Actions + generated static assets**, with no always-on backend service.

8. **Quality and validation**
   - `scripts/sync/test_batch_sync.py` indicates batch-sync behavior is tested before or during ingestion.
   - The presence of backup and checkpoint files shows the repo is designed to recover from partial sync failures and avoid losing curated Markdown content.

9. **README generation / topical curation**
   - `scripts/sync/update_readme_hot_categories.py` appears to keep the README’s “热门分类” list aligned with actual category popularity.
   - This means the README is not just documentation; it is partly a generated/maintained presentation layer over the corpus.

10. **End-to-end data flow**
   - External ebook sources → sync scripts → Markdown category files in `md/` → parser/generator scripts → JSON + HTML assets in `docs/` → GitHub Pages static site → browser-side search/navigation.

11. **Important architectural boundary**
   - The repo cleanly separates:
     - **source content and curation** in `md/`
     - **generation logic** in `scripts/`
     - **published runtime** in `docs/`
   - That boundary is the key to drawing the architecture diagram correctly: the site is a generated artifact, not a live application with a server database.

12. **Overall shape**
   - Think of it as a **content pipeline + static site renderer + GitHub Pages deployment**.
   - The core user experience is browsing and searching a large, precomputed ebook index, while maintenance is handled by scheduled or manual GitHub Actions workflows.

Main entry points: [`.github/workflows/generate-site.yml`](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/.github/workflows/generate-site.yml) (Publish WF), [`docs/search.js`](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/search.js) (Search UI), [`scripts/parse_md_to_json.py`](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/parse_md_to_json.py) (MD parser), [`scripts/generate_index.py`](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/generate_index.py) (Index builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_content["Content corpus"]
  node_readme["README<br/>curation overview<br/>[README.md]"]
  node_md_corpus["Category MD<br/>content corpus"]
  node_spine_topics["Top topics<br/>curated categories"]
end

subgraph group_scripts["Build and sync"]
  node_parse_md["MD parser<br/>markdown→json"]
  node_gen_index["Index builder<br/>site generator<br/>[generate_index.py]"]
  node_sync_all["Full sync<br/>ingestion job<br/>[sync_all_books.py]"]
  node_incremental_sync["Delta sync<br/>ingestion job"]
  node_parse_detail["Book parser<br/>metadata extractor"]
  node_backup_md["MD backup<br/>safety step<br/>[backup_md.py]"]
  node_checkpoint["Sync checkpoint<br/>state file"]
  node_validate_batch["Batch test<br/>validation<br/>[test_batch_sync.py]"]
  node_hot_categories["Hot categories<br/>readme updater"]
end

subgraph group_docs["Published site"]
  node_index_html["Home page<br/>static entry<br/>[index.html]"]
  node_search_js["Search UI<br/>client search<br/>[search.js]"]
  node_books_json["Books index<br/>generated data<br/>[books.json]"]
  node_all_books_json["All books<br/>generated data<br/>[all-books.json]"]
  node_parse_stats["Parse stats<br/>generated stats<br/>[parse-stats.json]"]
  node_nojekyll["No Jekyll<br/>pages marker<br/>[.nojekyll]"]
end

subgraph group_ci["GitHub Actions"]
  node_wf_full["Full sync WF<br/>workflow<br/>[full-sync.yml]"]
  node_wf_incremental["Delta WF<br/>workflow"]
  node_wf_site["Publish WF<br/>workflow<br/>[generate-site.yml]"]
end

node_readme -->|"summarizes"| node_spine_topics
node_md_corpus -->|"feeds"| node_parse_md
node_parse_md -->|"emits"| node_books_json
node_parse_md -->|"emits"| node_all_books_json
node_parse_md -->|"emits"| node_parse_stats
node_gen_index -->|"builds"| node_index_html
node_gen_index -->|"consumes"| node_books_json
node_gen_index -->|"consumes"| node_all_books_json
node_index_html -->|"loads"| node_search_js
node_search_js -->|"queries"| node_books_json
node_search_js -->|"queries"| node_all_books_json
node_search_js -.->|"may read"| node_parse_stats
node_sync_all -->|"uses"| node_parse_detail
node_incremental_sync -->|"uses"| node_parse_detail
node_sync_all -->|"protects"| node_backup_md
node_incremental_sync -->|"protects"| node_backup_md
node_incremental_sync -->|"tracks"| node_checkpoint
node_sync_all -->|"checks"| node_validate_batch
node_hot_categories -->|"updates"| node_readme
node_wf_full -->|"runs"| node_sync_all
node_wf_incremental -->|"runs"| node_incremental_sync
node_wf_site -->|"runs"| node_gen_index
node_wf_site -->|"publishes"| node_search_js
node_wf_site -->|"publishes"| node_index_html
node_wf_site -->|"publishes"| node_books_json
node_wf_site -->|"publishes"| node_all_books_json
node_wf_site -->|"publishes"| node_parse_stats
node_wf_site -->|"publishes"| node_nojekyll

click node_readme "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/README.md"
click node_parse_md "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/parse_md_to_json.py"
click node_gen_index "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/generate_index.py"
click node_sync_all "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/sync_all_books.py"
click node_incremental_sync "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/incremental_sync.py"
click node_parse_detail "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/parse_book_detail_enhanced.py"
click node_backup_md "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/backup_md.py"
click node_checkpoint "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/max_book_id_found.txt"
click node_validate_batch "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/test_batch_sync.py"
click node_hot_categories "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/update_readme_hot_categories.py"
click node_index_html "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/index.html"
click node_search_js "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/search.js"
click node_books_json "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/books.json"
click node_all_books_json "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/all-books.json"
click node_parse_stats "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/parse-stats.json"
click node_nojekyll "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/.nojekyll"
click node_wf_full "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/.github/workflows/full-sync.yml"
click node_wf_incremental "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/.github/workflows/incremental-sync.yml"
click node_wf_site "https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/.github/workflows/generate-site.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_readme,node_md_corpus,node_spine_topics toneBlue
class node_parse_md,node_gen_index,node_sync_all,node_incremental_sync,node_parse_detail,node_backup_md,node_checkpoint,node_validate_batch,node_hot_categories toneAmber
class node_index_html,node_search_js,node_books_json,node_all_books_json,node_parse_stats,node_nojekyll toneMint
class node_wf_full,node_wf_incremental,node_wf_site toneRose
```

## Components

### Content corpus

Canonical Markdown category pages and curated reading lists.

- [README](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/README.md): curation overview · Human-facing entry point with hot categories and repo guidance. · `README.md`
- **Category MD**: content corpus · Large set of topic pages that define the ebook catalog and tags.
- **Top topics**: curated categories · Representative high-traffic categories such as 文学, 历史, Python, AI, and related topic pages.

### Build and sync

Scripts that transform, refresh, and validate the corpus.

- [MD parser](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/parse_md_to_json.py): markdown→json · Converts markdown category pages into structured JSON data. · `scripts/parse_md_to_json.py`
- [Index builder](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/generate_index.py): site generator · Builds the searchable site index and homepage-facing artifacts. · `scripts/generate_index.py`
- [Full sync](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/sync_all_books.py): ingestion job · Performs complete corpus refresh from external ebook sources. · `scripts/sync/sync_all_books.py`
- [Delta sync](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/incremental_sync.py): ingestion job · Handles partial updates and incremental corpus changes. · `scripts/sync/incremental_sync.py`
- [Book parser](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/parse_book_detail_enhanced.py): metadata extractor · Extracts richer book metadata during sync. · `scripts/sync/parse_book_detail_enhanced.py`
- [MD backup](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/backup_md.py): safety step · Preserves markdown content before overwrite or regeneration. · `scripts/sync/backup_md.py`
- [Sync checkpoint](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/max_book_id_found.txt): state file · Tracks the last processed book ID for incremental runs. · `max_book_id_found.txt`
- [Batch test](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/test_batch_sync.py): validation · Checks batch sync behavior during ingestion. · `scripts/sync/test_batch_sync.py`
- [Hot categories](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/scripts/sync/update_readme_hot_categories.py): readme updater · Keeps README category highlights aligned with corpus popularity. · `scripts/sync/update_readme_hot_categories.py`

### Published site

Static assets served from GitHub Pages.

- [Home page](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/index.html): static entry · GitHub Pages entry page for browsing the catalog. · `docs/index.html`
- [Search UI](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/search.js): client search · Browser-side search and ranking over prebuilt JSON indexes. · `docs/search.js`
- [Books index](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/books.json): generated data · Primary JSON catalog consumed by the static site. · `docs/books.json`
- [All books](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/all-books.json): generated data · Expanded JSON dataset used by the site and search flow. · `docs/all-books.json`
- [Parse stats](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/parse-stats.json): generated stats · Content statistics emitted during generation for display or validation. · `docs/parse-stats.json`
- [No Jekyll](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/docs/.nojekyll): pages marker · Ensures GitHub Pages serves the docs folder as plain static content. · `docs/.nojekyll`

### GitHub Actions

Automated workflows for refresh and deployment.

- [Full sync WF](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/.github/workflows/full-sync.yml): workflow · GitHub Actions workflow for complete refresh runs. · `.github/workflows/full-sync.yml`
- [Delta WF](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/.github/workflows/incremental-sync.yml): workflow · GitHub Actions workflow for incremental updates. · `.github/workflows/incremental-sync.yml`
- [Publish WF](https://github.com/jbiaojerry/ebook-treasure-chest/blob/main/.github/workflows/generate-site.yml): workflow · GitHub Actions workflow that rebuilds and publishes static site artifacts. · `.github/workflows/generate-site.yml`

## Connections

- README → Top topics: summarizes · README highlights the major topical categories reflected in markdown pages.
- Category MD → MD parser: feeds · Markdown category pages are parsed into structured data.
- MD parser → Books index: emits · Parser produces the main books JSON artifact.
- MD parser → All books: emits · Parser also contributes the broader catalog dataset.
- MD parser → Parse stats: emits · Generation step records parsing statistics.
- Index builder → Home page: builds · Index generator produces the homepage and site shell.
- Index builder → Books index: consumes · Site generator uses the structured catalog data.
- Index builder → All books: consumes · Site generator uses the expanded book dataset.
- Home page → Search UI: loads · The static home page loads the client search script.
- Search UI → Books index: queries · Browser search filters the prebuilt catalog index.
- Search UI → All books: queries · Browser search can also inspect the larger catalog dataset.
- Search UI → Parse stats: may read · Search or display logic can use generation statistics as supporting data.
- Full sync → Book parser: uses · Full sync enriches book records during ingestion.
- Delta sync → Book parser: uses · Delta sync also extracts detailed metadata.
- Full sync → MD backup: protects · Full refresh backs up markdown before overwriting content.
- Delta sync → MD backup: protects · Incremental updates preserve existing markdown first.
- Delta sync → Sync checkpoint: tracks · Incremental sync relies on the last processed book ID.
- Full sync → Batch test: checks · Batch behavior is validated during sync runs.
- Hot categories → README: updates · Hot category generation keeps the README presentation current.
- Full sync WF → Full sync: runs · Full-sync workflow executes the corpus refresh job.
- Delta WF → Delta sync: runs · Incremental workflow executes delta ingestion.
- Publish WF → Index builder: runs · Site workflow rebuilds the generated index and assets.
- Publish WF → Search UI: publishes · Workflow publishes the client-side search runtime with the site.
- Publish WF → Home page: publishes · Workflow publishes the static entry page.
- Publish WF → Books index: publishes · Workflow deploys the generated catalog data.
- Publish WF → All books: publishes · Workflow deploys the expanded book dataset.
- Publish WF → Parse stats: publishes · Workflow deploys generated parsing statistics.
- Publish WF → No Jekyll: publishes · Workflow keeps GitHub Pages serving the docs folder as static files.