# crocodilestick/calibre-web-automated architecture

> Architecture diagram of the GitHub repository crocodilestick/calibre-web-automated, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crocodilestick/calibre-web-automated
- Repository: https://github.com/crocodilestick/calibre-web-automated

## Overview

1. **What this repository is**
   Calibre-Web Automated (CWA) is a self-hosted ebook library system built on top of Calibre-Web, but extended with automation. The core product boundary is a web application that manages a Calibre library, plus a set of background services that ingest, transform, enrich, back up, and distribute books.

2. **Top-level runtime model**
   The app is a Python service, deployed primarily in Docker. The repository includes container build files, Docker Compose examples, Kubernetes manifests, and an `s6-overlay` init layout under `root/etc/s6-overlay`, which means the container runs multiple coordinated processes rather than a single monolith. The web app itself is served through the Python entrypoints in `cps.py`, `cps/main.py`, `cps/server.py`, `cps/web.py`, with WSGI adapters for different server modes.

3. **Persistent storage boundaries**
   The most important architectural boundary is filesystem-based:
   - `/config` holds app configuration, the app database, logs, and retained processing artifacts.
   - `/calibre-library` is the Calibre library root and contains the Calibre `metadata.db` plus book files.
   - `/cwa-book-ingest` is the drop folder where new files arrive for processing.
   - `/config/.config/calibre/plugins` is an optional mounted plugin area for Calibre plugins.
   CWA also keeps processed originals in `/config/processed_books` and can zip them later. The code treats the Calibre library as an external system of record, while its own app settings live separately.

4. **Data stores and library model**
   There are two distinct SQLite-backed worlds in play: the Calibre library database (`metadata.db`) and CWA’s own application settings/state database (`app.db`). The `cps/db.py`, `cps/config_sql.py`, and `scripts/cwa_db.py`/`scripts/cwa_schema.sql` files indicate a dedicated app schema for automation state, tasks, and settings, while the Calibre database continues to own book catalog metadata. That separation is important for diagrams: library content and app control-plane state are not the same store.

5. **Web UI and page rendering**
   The user-facing application is server-rendered with templates in `cps/templates` and shared layout in `cps/layout.html` plus Jinja helpers in `cps/jinjia.py` and `cps/render_template.py`. Static assets live under `cps/static` and include the reader, admin tools, charts, modals, and upload/duplicate-management UI. The UI covers library browsing, OPDS, book detail/edit pages, admin configuration, stats, tasks, search, shelves, reading views, and specialized CWA pages such as KOSync and stats dashboards.

6. **Authentication and access control**
   Authentication is a major subsystem, not just a login form. The repository includes local login management, OAuth/OIDC support, and remote-login helpers in `cps/cw_login`, `cps/oauth.py`, `cps/oauth_bb.py`, `cps/remotelogin.py`, and `cps/MyLoginManager.py`. The app also accounts for reverse-proxy deployment through `cps/reverseproxy.py` and warns about trusted proxy depth in the README. Access control is therefore split between native accounts, external identity providers, and proxy-aware session handling.

7. **Ingest and automation pipeline**
   The main automation flow starts when files appear in the ingest directory. Watcher logic in the service layer and scripts detects new books, then triggers processing through modules such as `cps/tasks/upload.py`, `cps/tasks/convert.py`, `cps/tasks/clean.py`, `cps/tasks/metadata_backup.py`, `cps/tasks/thumbnail.py`, and `cps/tasks/auto_send.py`. The pipeline can:
   - import books into the Calibre library
   - auto-convert formats
   - repair EPUBs
   - fetch metadata
   - enforce covers/metadata into the actual ebook files
   - create backups of originals
   - send finished books by email
   The README’s `NETWORK_SHARE_MODE` matters here: it switches watchers from event-based detection to polling for NAS/NFS/SMB reliability.

8. **Background services and task orchestration**
   CWA is built around asynchronous service processes and task tracking. The `cps/services` package and `cps/tasks_status.py` coordinate workers, background schedulers, sync tokens, and service integrations. The `root/etc/s6-overlay/s6-rc.d` tree shows separate long-running services for initialization, ingest service, process recovery, auto-library detection, auto-zip, checksum backfill, and metadata change detection. In other words, the system is a small service mesh inside one container: the web app handles requests, while supervisors run file watchers and maintenance jobs.

9. **Metadata, search, and enrichment**
   Metadata enrichment is modular. The `cps/metadata_provider` directory contains provider adapters for Amazon, Google, Kobo, Hardcover, LitRes, Douban, ComicVine, and others. Additional supporting modules such as `cps/metadata_helper.py`, `cps/search_metadata.py`, `cps/auto_metadata.py`, and `cps/search.py` provide lookup, normalization, and search behavior. This subsystem is what allows ingest-time metadata fetching and manual metadata editing to feed both the UI and the on-disk book files.

10. **Reading, device sync, and library publishing**
   CWA exposes multiple reading and integration surfaces: in-browser readers for EPUB, PDF, DJVU, TXT, comics, and audio-related flows; OPDS feeds for external apps; Kobo sync support through `cps/kobo*` modules; and KOReader sync through `cps/progress_syncing` plus the `koreader/plugins/cwasync.koplugin` client plugin. The KOReader path uses book checksums and read-state models to synchronize progress, while Kobo integration focuses on device auth, cover cache, and status updates.

11. **Library operations beyond ingest**
   The application also contains normal library-management features that are implemented as first-class modules: shelves and magic shelves, duplicate detection, batch editing/deletion, cover handling, file helpers, converters, and uploader logic. These are not separate systems, but they sit at the intersection of the web UI, the Calibre library database, and the ingest pipeline. For a diagram, they should be shown as application services that mutate library state rather than as standalone subsystems.

12. **Operational tooling and delivery**
   The repo includes CI/CD workflows under `.github/workflows`, a `Dockerfile`, Compose files, Kubernetes manifests, and helper scripts under `scripts/`. There are also translation workflows and a large locale bundle under `cps/translations` and `cps/static/locale`, which indicates the UI is internationally localized. The `tests/` tree is broad, covering unit, integration, smoke, and container-startup behavior, so the architecture is strongly validated around ingest, sync, and startup flows.

Main entry points: [`cps/server.py`](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/server.py) (Server core), [`cps/tasks`](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/tasks) (Task workers), [`cps/tasks/upload.py`](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/tasks/upload.py) (Ingest flow), [`empty_library/app.db`](https://github.com/crocodilestick/calibre-web-automated/blob/main/empty_library/app.db) (App DB)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_cps_entry["cps.py<br/>app entrypoint<br/>[cps.py]"]
  node_server_stack["Server core<br/>web runtime<br/>[server.py]"]
  node_s6_services["s6 services<br/>process supervisor<br/>[s6-rc.d]"]
end

subgraph group_web["Web app"]
  node_templates_ui["Templates<br/>server-rendered UI"]
  node_static_ui["Static assets<br/>frontend assets"]
  node_library_ui["Library UI<br/>library features<br/>[admin.py]"]
  node_search_ui["Search & browse<br/>discovery<br/>[search.py]"]
end

subgraph group_automation["Automation"]
  node_ingest["Ingest flow<br/>automation pipeline<br/>[upload.py]"]
  node_task_workers["Task workers<br/>background jobs"]
  node_services["Services<br/>orchestration"]
  node_metadata["Metadata<br/>enrichment"]
end

subgraph group_storage["Storage"]
  node_library_db[("Calibre DB<br/>library sqlite<br/>[metadata.db]")]
  node_app_db[("App DB<br/>control sqlite<br/>[app.db]")]
  node_library_fs[("Library files<br/>book store")]
  node_config_fs[("Config volume<br/>app state")]
  node_ingest_fs["Ingest drop<br/>input folder"]
end

subgraph group_sync["Sync"]
  node_sync_ui["Sync surfaces<br/>reader sync"]
  node_ebooks["Readers<br/>viewer<br/>[readingservices.py]"]
end

subgraph group_access["Access"]
  node_auth["Auth<br/>identity"]
  node_proxy["Proxy handling<br/>session edge<br/>[reverseproxy.py]"]
end

subgraph group_ops["Ops"]
  node_ops_delivery["Delivery<br/>deploy"]
end

node_cps_entry -->|"starts"| node_server_stack
node_server_stack -->|"renders"| node_templates_ui
node_server_stack -->|"serves"| node_static_ui
node_server_stack -->|"authenticates"| node_auth
node_server_stack -->|"adapts"| node_proxy
node_server_stack -->|"hosts"| node_library_ui
node_server_stack -->|"exposes"| node_search_ui
node_library_ui -->|"reads/writes"| node_library_db
node_library_ui -->|"reads/writes"| node_app_db
node_search_ui -->|"queries"| node_library_db
node_auth -->|"stores"| node_app_db
node_ingest -->|"watches"| node_ingest_fs
node_ingest -->|"imports"| node_library_fs
node_ingest -->|"queues"| node_task_workers
node_task_workers -->|"enriches"| node_metadata
node_task_workers -->|"archives"| node_config_fs
node_task_workers -->|"tracks"| node_app_db
node_services -->|"coordinates"| node_task_workers
node_services -->|"syncs"| node_app_db
node_services -->|"drives"| node_ingest
node_sync_ui -->|"reads"| node_library_db
node_sync_ui -->|"writes"| node_app_db
node_ebooks -->|"opens"| node_library_fs
node_ebooks -->|"reports progress"| node_sync_ui
node_ops_delivery -->|"launches"| node_s6_services
node_ops_delivery -->|"packages"| node_server_stack
node_s6_services -->|"runs"| node_server_stack
node_s6_services -->|"runs"| node_ingest
node_s6_services -->|"runs"| node_services

click node_cps_entry "https://github.com/crocodilestick/calibre-web-automated/blob/main/cps.py"
click node_server_stack "https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/server.py"
click node_s6_services "https://github.com/crocodilestick/calibre-web-automated/blob/main/root/etc/s6-overlay/s6-rc.d"
click node_templates_ui "https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/templates"
click node_static_ui "https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/static"
click node_library_ui "https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/admin.py"
click node_search_ui "https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/search.py"
click node_auth "https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/cw_login"
click node_proxy "https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/reverseproxy.py"
click node_ingest "https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/tasks/upload.py"
click node_task_workers "https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/tasks"
click node_services "https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/services"
click node_metadata "https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/metadata_provider"
click node_library_db "https://github.com/crocodilestick/calibre-web-automated/blob/main/empty_library/metadata.db"
click node_app_db "https://github.com/crocodilestick/calibre-web-automated/blob/main/empty_library/app.db"
click node_library_fs "https://github.com/crocodilestick/calibre-web-automated/tree/main/empty_library"
click node_config_fs "https://github.com/crocodilestick/calibre-web-automated/tree/main/root"
click node_ingest_fs "https://github.com/crocodilestick/calibre-web-automated/blob/main/scripts/ingest_processor.py"
click node_sync_ui "https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/progress_syncing"
click node_ebooks "https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/readingservices.py"
click node_ops_delivery "https://github.com/crocodilestick/calibre-web-automated/tree/main/Dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cps_entry,node_server_stack,node_s6_services,node_ops_delivery toneBlue
class node_templates_ui,node_static_ui,node_library_ui,node_search_ui toneAmber
class node_ingest,node_task_workers,node_services,node_metadata toneMint
class node_library_db,node_app_db,node_library_fs,node_config_fs,node_ingest_fs toneRose
class node_sync_ui,node_ebooks toneIndigo
class node_auth,node_proxy toneTeal
```

## Components

### Runtime

Container entrypoints, WSGI adapters, and process supervisor layout

- [cps.py](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps.py): app entrypoint · Primary application launcher and top-level startup path · `cps.py`
- [Server core](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/server.py): web runtime · Main server, app bootstrap, WSGI adapters, and reverse-proxy handling · `cps/server.py`
- [s6 services](https://github.com/crocodilestick/calibre-web-automated/blob/main/root/etc/s6-overlay/s6-rc.d): process supervisor · Container service layout for init, ingest, recovery, zipping, and maintenance daemons · `root/etc/s6-overlay/s6-rc.d`

### Web app

Request handling, templates, and browser-facing features

- [Templates](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/templates): server-rendered UI · Jinja templates, layout, and rendering helpers for the web interface · `cps/templates`
- [Static assets](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/static): frontend assets · CSS, JS, locale bundles, reader widgets, and client-side UI logic · `cps/static`
- [Library UI](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/admin.py): library features · Browse, edit, shelves, duplicates, stats, uploads, and admin pages · `cps/admin.py`
- [Search & browse](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/search.py): discovery · Search, pagination, feeds, OPDS, and listing views · `cps/search.py`

### Automation

Ingest, background services, and scheduled maintenance

- [Ingest flow](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/tasks/upload.py): automation pipeline · File-drop ingest, upload handling, import, and conversion orchestration · `cps/tasks/upload.py`
- [Task workers](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/tasks): background jobs · Conversion, cleaning, metadata backup, thumbnails, mail, auto-send, and duplicate scan jobs · `cps/tasks`
- [Services](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/services): orchestration · Background scheduler, worker coordination, sync tokens, and service integrations · `cps/services`
- [Metadata](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/metadata_provider): enrichment · Provider adapters, metadata lookup, normalization, and search-assisted enrichment · `cps/metadata_provider`

### Storage

SQLite-backed state and filesystem boundaries represented in-repo

- [Calibre DB](https://github.com/crocodilestick/calibre-web-automated/blob/main/empty_library/metadata.db): library sqlite · Calibre library catalog metadata and book index owned by the external library system · `empty_library/metadata.db`
- [App DB](https://github.com/crocodilestick/calibre-web-automated/blob/main/empty_library/app.db): control sqlite · CWA settings, automation state, tasks, and operational records · `empty_library/app.db`
- [Library files](https://github.com/crocodilestick/calibre-web-automated/tree/main/empty_library): book store · Calibre library root with book files and metadata database · `empty_library`
- [Config volume](https://github.com/crocodilestick/calibre-web-automated/tree/main/root): app state · Configuration, logs, processed originals, and retained artifacts · `root`
- [Ingest drop](https://github.com/crocodilestick/calibre-web-automated/blob/main/scripts/ingest_processor.py): input folder · Watched landing zone for new files to be processed · `scripts/ingest_processor.py`

### Sync

Reader, device, and progress synchronization surfaces

- [Sync surfaces](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/progress_syncing): reader sync · OPDS, Kobo sync, KOReader progress sync, and in-browser readers · `cps/progress_syncing`
- [Readers](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/readingservices.py): viewer · EPUB, PDF, DJVU, text, comic, audio, and embedded reading views · `cps/readingservices.py`

### Access

Authentication, remote login, and proxy-aware session handling

- [Auth](https://github.com/crocodilestick/calibre-web-automated/tree/main/cps/cw_login): identity · Local login, OAuth/OIDC, remote login, and login manager integration · `cps/cw_login`
- [Proxy handling](https://github.com/crocodilestick/calibre-web-automated/blob/main/cps/reverseproxy.py): session edge · Reverse-proxy awareness and trusted-forwarded-header handling · `cps/reverseproxy.py`

### Ops

Containerization, deployment, and build tooling

- [Delivery](https://github.com/crocodilestick/calibre-web-automated/tree/main/Dockerfile): deploy · Docker, Compose, Kubernetes, CI, translations, and helper scripts · `Dockerfile`

## Connections

- cps.py → Server core: starts · Top-level entrypoint boots the server/runtime
- Server core → Templates: renders · Server-side pages are assembled from templates
- Server core → Static assets: serves · Browser assets are delivered by the web app
- Server core → Auth: authenticates · Login and session management are enforced in the request path
- Server core → Proxy handling: adapts · Proxy headers and deployment modes affect request handling
- Server core → Library UI: hosts · Admin and library-management actions are exposed through the app
- Server core → Search & browse: exposes · Search, feeds, and browse pages are part of the web surface
- Library UI → Calibre DB: reads/writes · Library operations mutate Calibre catalog state
- Library UI → App DB: reads/writes · UI settings and automation state live in the app database
- Search & browse → Calibre DB: queries · Browse and search depend on the Calibre catalog
- Auth → App DB: stores · Accounts, sessions, and identity configuration are persisted in app state
- Ingest flow → Ingest drop: watches · New files arrive through the ingest drop folder
- Ingest flow → Library files: imports · Processed books are moved into the Calibre library
- Ingest flow → Task workers: queues · Ingest triggers downstream processing tasks
- Task workers → Metadata: enriches · Automation fetches and normalizes external metadata
- Task workers → Config volume: archives · Backups, logs, and retained originals are kept under config storage
- Task workers → App DB: tracks · Task status and automation records are persisted
- Services → Task workers: coordinates · Schedulers and workers orchestrate background jobs
- Services → App DB: syncs · Service state and sync tokens are stored in the app database
- Services → Ingest flow: drives · Background services activate ingest processing and recovery
- Sync surfaces → Calibre DB: reads · Sync and publishing features depend on library metadata
- Sync surfaces → App DB: writes · Progress, device, and sync state are tracked in app storage
- Readers → Library files: opens · Readers stream files from the library store
- Readers → Sync surfaces: reports progress · Reader views feed progress synchronization features
- Delivery → s6 services: launches · The container supervisor starts the internal service mesh
- Delivery → Server core: packages · Container and deployment files define how the runtime runs
- s6 services → Server core: runs · The main web process is one supervised service
- s6 services → Ingest flow: runs · Dedicated ingest watcher/service runs alongside the web app
- s6 services → Services: runs · Maintenance and coordination jobs are supervised processes
