How crocodilestick/calibre-web-automated works
1. What this repository is
Calibre-Web Automated (CWA) is a self-hosted ebook library system built on top of Calibre-Web, but extended with automation. The core product boundary is a web application that manages a Calibre library, plus a set of background services that ingest, transform, enrich, back up, and distribute books.
2. Top-level runtime model
The app is a Python service, deployed primarily in Docker. The repository includes container build files, Docker Compose examples, Kubernetes manifests, and an s6-overlay init layout under root/etc/s6-overlay, which means the container runs multiple coordinated processes rather than a single monolith. The web app itself is served through the Python entrypoints in cps.py, cps/main.py, cps/server.py, cps/web.py, with WSGI adapters for different server modes.
3. Persistent storage boundaries
The most important architectural boundary is filesystem-based:
- /config holds app configuration, the app database, logs, and retained processing artifacts.
- /calibre-library is the Calibre library root and contains the Calibre metadata.db plus book files.
- /cwa-book-ingest is the drop folder where new files arrive for processing.
- /config/.config/calibre/plugins is an optional mounted plugin area for Calibre plugins.
CWA also keeps processed originals in /config/processed_books and can zip them later. The code treats the Calibre library as an external system of record, while its own app settings live separately.
4. Data stores and library model
There are two distinct SQLite-backed worlds in play: the Calibre library database (metadata.db) and CWA’s own application settings/state database (app.db). The cps/db.py, cps/config_sql.py, and scripts/cwa_db.py/scripts/cwa_schema.sql files indicate a dedicated app schema for automation state, tasks, and settings, while the Calibre database continues to own book catalog metadata. That separation is important for diagrams: library content and app control-plane state are not the same store.
5. Web UI and page rendering
The user-facing application is server-rendered with templates in cps/templates and shared layout in cps/layout.html plus Jinja helpers in cps/jinjia.py and cps/render_template.py. Static assets live under cps/static and include the reader, admin tools, charts, modals, and upload/duplicate-management UI. The UI covers library browsing, OPDS, book detail/edit pages, admin configuration, stats, tasks, search, shelves, reading views, and specialized CWA pages such as KOSync and stats dashboards.
6. Authentication and access control
Authentication is a major subsystem, not just a login form. The repository includes local login management, OAuth/OIDC support, and remote-login helpers in cps/cw_login, cps/oauth.py, cps/oauth_bb.py, cps/remotelogin.py, and cps/MyLoginManager.py. The app also accounts for reverse-proxy deployment through cps/reverseproxy.py and warns about trusted proxy depth in the README. Access control is therefore split between native accounts, external identity providers, and proxy-aware session handling.
7. Ingest and automation pipeline
The main automation flow starts when files appear in the ingest directory. Watcher logic in the service layer and scripts detects new books, then triggers processing through modules such as cps/tasks/upload.py, cps/tasks/convert.py, cps/tasks/clean.py, cps/tasks/metadata_backup.py, cps/tasks/thumbnail.py, and cps/tasks/auto_send.py. The pipeline can:
- import books into the Calibre library
- auto-convert formats
- repair EPUBs
- fetch metadata
- enforce covers/metadata into the actual ebook files
- create backups of originals
- send finished books by email
The README’s NETWORK_SHARE_MODE matters here: it switches watchers from event-based detection to polling for NAS/NFS/SMB reliability.
8. Background services and task orchestration
CWA is built around asynchronous service processes and task tracking. The cps/services package and cps/tasks_status.py coordinate workers, background schedulers, sync tokens, and service integrations. The root/etc/s6-overlay/s6-rc.d tree shows separate long-running services for initialization, ingest service, process recovery, auto-library detection, auto-zip, checksum backfill, and metadata change detection. In other words, the system is a small service mesh inside one container: the web app handles requests, while supervisors run file watchers and maintenance jobs.
9. Metadata, search, and enrichment
Metadata enrichment is modular. The cps/metadata_provider directory contains provider adapters for Amazon, Google, Kobo, Hardcover, LitRes, Douban, ComicVine, and others. Additional supporting modules such as cps/metadata_helper.py, cps/search_metadata.py, cps/auto_metadata.py, and cps/search.py provide lookup, normalization, and search behavior. This subsystem is what allows ingest-time metadata fetching and manual metadata editing to feed both the UI and the on-disk book files.
10. Reading, device sync, and library publishing
CWA exposes multiple reading and integration surfaces: in-browser readers for EPUB, PDF, DJVU, TXT, comics, and audio-related flows; OPDS feeds for external apps; Kobo sync support through cps/kobo* modules; and KOReader sync through cps/progress_syncing plus the koreader/plugins/cwasync.koplugin client plugin. The KOReader path uses book checksums and read-state models to synchronize progress, while Kobo integration focuses on device auth, cover cache, and status updates.
11. Library operations beyond ingest
The application also contains normal library-management features that are implemented as first-class modules: shelves and magic shelves, duplicate detection, batch editing/deletion, cover handling, file helpers, converters, and uploader logic. These are not separate systems, but they sit at the intersection of the web UI, the Calibre library database, and the ingest pipeline. For a diagram, they should be shown as application services that mutate library state rather than as standalone subsystems.
12. Operational tooling and delivery
The repo includes CI/CD workflows under .github/workflows, a Dockerfile, Compose files, Kubernetes manifests, and helper scripts under scripts/. There are also translation workflows and a large locale bundle under cps/translations and cps/static/locale, which indicates the UI is internationally localized. The tests/ tree is broad, covering unit, integration, smoke, and container-startup behavior, so the architecture is strongly validated around ingest, sync, and startup flows.
Main entry points: server.py Server core · tasks Task workers · upload.py Ingest flow · app.db App DB
Components 21
Runtime
Container entrypoints, WSGI adapters, and process supervisor layout
- cps.py · Primary application launcher and top-level startup path
cps.py - Server core · Main server, app bootstrap, WSGI adapters, and reverse-proxy handling
cps/server.py - s6 services · Container service layout for init, ingest, recovery, zipping, and maintenance daemons
root/etc/s6-overlay/s6-rc.d
Web app
Request handling, templates, and browser-facing features
- Templates · Jinja templates, layout, and rendering helpers for the web interface
cps/templates - Static assets · CSS, JS, locale bundles, reader widgets, and client-side UI logic
cps/static - Library UI · Browse, edit, shelves, duplicates, stats, uploads, and admin pages
cps/admin.py - Search & browse · Search, pagination, feeds, OPDS, and listing views
cps/search.py
Automation
Ingest, background services, and scheduled maintenance
- Ingest flow · File-drop ingest, upload handling, import, and conversion orchestration
cps/tasks/upload.py - Task workers · Conversion, cleaning, metadata backup, thumbnails, mail, auto-send, and duplicate scan jobs
cps/tasks - Services · Background scheduler, worker coordination, sync tokens, and service integrations
cps/services - Metadata · Provider adapters, metadata lookup, normalization, and search-assisted enrichment
cps/metadata_provider
Storage
SQLite-backed state and filesystem boundaries represented in-repo
- Calibre DB · Calibre library catalog metadata and book index owned by the external library system
empty_library/metadata.db - App DB · CWA settings, automation state, tasks, and operational records
empty_library/app.db - Library files · Calibre library root with book files and metadata database
empty_library - Config volume · Configuration, logs, processed originals, and retained artifacts
root - Ingest drop · Watched landing zone for new files to be processed
scripts/ingest_processor.py
Sync
Reader, device, and progress synchronization surfaces
- Sync surfaces · OPDS, Kobo sync, KOReader progress sync, and in-browser readers
cps/progress_syncing - Readers · EPUB, PDF, DJVU, text, comic, audio, and embedded reading views
cps/readingservices.py
Access
Authentication, remote login, and proxy-aware session handling
- Auth · Local login, OAuth/OIDC, remote login, and login manager integration
cps/cw_login - Proxy handling · Reverse-proxy awareness and trusted-forwarded-header handling
cps/reverseproxy.py
Ops
Containerization, deployment, and build tooling
- Delivery · Docker, Compose, Kubernetes, CI, translations, and helper scripts
Dockerfile
Connections 29
- cps.py → Server core: starts · Top-level entrypoint boots the server/runtime
- Server core → Templates: renders · Server-side pages are assembled from templates
- Server core → Static assets: serves · Browser assets are delivered by the web app
- Server core → Auth: authenticates · Login and session management are enforced in the request path
- Server core → Proxy handling: adapts · Proxy headers and deployment modes affect request handling
- Server core → Library UI: hosts · Admin and library-management actions are exposed through the app
- Server core → Search & browse: exposes · Search, feeds, and browse pages are part of the web surface
- Library UI → Calibre DB: reads/writes · Library operations mutate Calibre catalog state
- Library UI → App DB: reads/writes · UI settings and automation state live in the app database
- Search & browse → Calibre DB: queries · Browse and search depend on the Calibre catalog
- Auth → App DB: stores · Accounts, sessions, and identity configuration are persisted in app state
- Ingest flow → Ingest drop: watches · New files arrive through the ingest drop folder
- Ingest flow → Library files: imports · Processed books are moved into the Calibre library
- Ingest flow → Task workers: queues · Ingest triggers downstream processing tasks
- Task workers → Metadata: enriches · Automation fetches and normalizes external metadata
- Task workers → Config volume: archives · Backups, logs, and retained originals are kept under config storage
- Task workers → App DB: tracks · Task status and automation records are persisted
- Services → Task workers: coordinates · Schedulers and workers orchestrate background jobs
- Services → App DB: syncs · Service state and sync tokens are stored in the app database
- Services → Ingest flow: drives · Background services activate ingest processing and recovery
- Sync surfaces → Calibre DB: reads · Sync and publishing features depend on library metadata
- Sync surfaces → App DB: writes · Progress, device, and sync state are tracked in app storage
- Readers → Library files: opens · Readers stream files from the library store
- Readers → Sync surfaces: reports progress · Reader views feed progress synchronization features
- Delivery → s6 services: launches · The container supervisor starts the internal service mesh
- Delivery → Server core: packages · Container and deployment files define how the runtime runs
- s6 services → Server core: runs · The main web process is one supervised service
- s6 services → Ingest flow: runs · Dedicated ingest watcher/service runs alongside the web app
- s6 services → Services: runs · Maintenance and coordination jobs are supervised processes