# ahmedallam222/book-bot architecture

> Architecture diagram of the GitHub repository ahmedallam222/book-bot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmedallam222/book-bot
- Repository: https://github.com/ahmedallam222/book-bot

## Overview

1. **Overall shape**
   - This repo is a **Telegram-first book search and delivery system** with a small HTTP control plane.
   - The runtime is a single Node.js/TypeScript service that hosts:
     - the user-facing Telegram bot,
     - an Express-based admin/API server,
     - background queue workers,
     - and, optionally, a second admin Telegram bot.

2. **Primary entry points**
   - `server/index.ts` is the main bootstrap: it starts the bot, workers, and HTTP server, and handles graceful shutdown.
   - `server/bot/index.ts` wires Telegram listeners for commands, callbacks, and free-text handling.
   - `server/routes.ts` exposes the admin dashboard and REST API.
   - `server/dashboard.html` is the single-file admin UI served by the backend.

3. **User request flow**
   - A user sends an Arabic book query in Telegram.
   - `bookNameParser.ts` strips intent words and normalizes Arabic text.
   - Guard checks run early: ban, maintenance, rate limit, and daily quota.
   - If allowed, the request is enqueued into Redis with priority based on user tier.
   - Workers consume the queue and execute the search/download pipeline.

4. **Search subsystem**
   - `server/bot/engine.ts` orchestrates discovery across the configured Arabic libraries.
   - `server/bot/sources.ts` defines the 14 source backends and their priorities.
   - Search uses Firecrawl for extraction, then fuzzy matching for typos or weak title matches.
   - `noorBookResolver.ts` is a special boundary for Cloudflare-protected Noor Book content, using Playwright where needed.

5. **Validation and delivery pipeline**
   - Candidate URLs are filtered through `verify.ts` and then downloaded in `download.ts`.
   - `pdfValidator.ts` applies multi-stage checks before a file is delivered:
     HTTP probe, PDF magic bytes, text density, page count, and AI judgment when needed.
   - If validated, the bot sends the file to Telegram and stores the Telegram `file_id` for reuse.
   - This creates a strong boundary between “found on the web” and “safe to deliver in-chat.”

6. **Data and state boundaries**
   - **Redis** is the operational state layer:
     queues, cache, rate limits, streaks, source health, DLQ, and other short-lived counters.
   - **Postgres** is the durable state layer:
     users, premium status, request history, audit data, and cached file metadata.
   - `shared/schema.ts` is the shared schema definition used across server-side data access.
   - `server/storage.ts` is the persistence boundary around Postgres/Drizzle.

7. **Engagement and retention systems**
   - `streak.ts`, `badges.ts`, `referral.ts`, and `analytics.ts` implement the retention loop.
   - These subsystems are intentionally decoupled from delivery: they run after a successful download and are designed to fail open.
   - Leaderboards and request stats are derived from canonicalized book titles to avoid duplicate entries.

8. **AI-dependent features**
   - The bot uses AI in two distinct places:
     - `pdfValidator.ts` for judging ambiguous PDFs,
     - `summary.ts` / `summaryHandler.ts` for generating structured book summaries.
   - `server/bot/aiProviders/` contains the summary/validation provider adapters and failover registry.
   - The AI layer is provider-agnostic by design, so no single vendor is a hard dependency.

9. **Admin control plane**
   - Operators use either the dashboard or the REST API for source toggles, user management, premium grants, broadcasts, and maintenance mode.
   - The dashboard is protected by a bearer token and serves as the main observability and operations surface.
   - `admin.ts`, `alertWatcher.ts`, `dailyDigest.ts`, and `weekly.ts` connect operational state back to administrators.

10. **Admin AI agent as a separate control path**
   - `server/bot/adminAgent/` is a second Telegram bot for admins only.
   - It exposes a tool-driven conversational interface for the same operational actions as the dashboard.
   - This is a distinct subsystem with its own LLM chain, conversation memory, loop guards, telemetry, and write-confirmation flow.

11. **Resilience and queue semantics**
   - Work is intentionally asynchronous: the bot acknowledges queue position, then workers process jobs independently.
   - Redis-backed retries and a dead-letter queue isolate failures.
   - `progressWatchdog.ts`, `failureRetry.ts`, and `alertWatcher.ts` handle stuck jobs, retries, and operator alerts.
   - This lets the system survive slow sources, AI outages, and transient Telegram issues without losing user requests.

12. **Integration boundaries**
   - External dependencies materially shape the architecture:
     - Telegram is the user interface and delivery channel.
     - Firecrawl is the search/extraction layer.
     - Redis and Postgres are the main state stores.
     - AI providers are used behind failover adapters, not directly by business logic.
     - Playwright is only used for the one protected source that needs browser mediation.
   - `cloudflare/welib-proxy/` is a separate worker-based proxy integration for another source path.

13. **Supporting tooling and delivery**
   - Docker Compose runs the bot, Postgres, and Redis together for local/dev parity.
   - `script/` and `scripts/` contain build, migration, backfill, and maintenance jobs.
   - CI runs typecheck, build, and smoke tests from `tests/`, which are an important architectural safety net because the repo relies on many cross-cutting behaviors.

Main entry points: [`server/bot/telegramFallback.ts`](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/telegramFallback.ts) (Telegram send), [`server/bot/engine.ts`](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/engine.ts) (Search engine), [`server/index.ts`](https://github.com/ahmedallam222/book-bot/blob/main/server/index.ts) (Server boot), [`server/bot/index.ts`](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/index.ts) (Bot entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_server_index["Server boot<br/>bootstrap<br/>[index.ts]"]
  node_bot_entry["Bot entry<br/>telegram entry<br/>[index.ts]"]
end

subgraph group_search["Search & Delivery"]
  node_parser["Query parser<br/>arabic normalize<br/>[bookNameParser.ts]"]
  node_guards["Guard checks<br/>access control<br/>[guards.ts]"]
  node_engine["Search engine<br/>orchestrator<br/>[engine.ts]"]
  node_sources["Sources<br/>source registry<br/>[sources.ts]"]
  node_firecrawl["Firecrawl parse<br/>extraction<br/>[firecrawlParse.ts]"]
  node_noor_resolver["Noor resolver<br/>browser boundary"]
  node_verify["Verify<br/>url filter<br/>[verify.ts]"]
  node_download["Download<br/>file fetch<br/>[download.ts]"]
  node_pdf_validator["PDF validator<br/>quality gate<br/>[pdfValidator.ts]"]
  node_telegram_fallback["Telegram send<br/>delivery"]
end

subgraph group_state["State"]
  node_queue["Queue<br/>redis jobs<br/>[queue.ts]"]
  node_redis_state[("Redis<br/>ephemeral state<br/>[redis.ts]")]
  node_storage[("Postgres store<br/>durable storage<br/>[storage.ts]")]
  node_schema["Schema<br/>shared model<br/>[schema.ts]"]
end

subgraph group_retention["Retention"]
  node_streaks(("Streaks<br/>retention<br/>[streak.ts]"))
  node_badges(("Badges<br/>retention<br/>[badges.ts]"))
  node_referrals(("Referrals<br/>retention<br/>[referral.ts]"))
  node_analytics(("Analytics<br/>retention<br/>[analytics.ts]"))
end

subgraph group_admin["Admin"]
  node_http_routes["Admin API<br/>express routes<br/>[routes.ts]"]
  node_dashboard_ui["Dashboard<br/>single-page UI<br/>[dashboard.html]"]
  node_admin_bot["Admin bot<br/>telegram ops<br/>[index.ts]"]
end

subgraph group_ai["AI Layer"]
  node_summary["Summary<br/>ai feature<br/>[summary.ts]"]
  node_ai_registry["AI registry<br/>provider adapters<br/>[registry.ts]"]
end

node_server_index -->|"starts"| node_bot_entry
node_server_index -->|"starts"| node_http_routes
node_server_index -->|"connects"| node_redis_state
node_server_index -->|"connects"| node_storage
node_bot_entry -->|"parses"| node_parser
node_bot_entry -->|"checks"| node_guards
node_bot_entry -->|"enqueues"| node_queue
node_queue -->|"dispatches"| node_engine
node_engine -->|"consults"| node_sources
node_engine -->|"extracts"| node_firecrawl
node_engine -->|"special-cases"| node_noor_resolver
node_engine -->|"filters"| node_verify
node_verify -->|"permits"| node_download
node_download -->|"checks"| node_pdf_validator
node_pdf_validator -->|"consults"| node_ai_registry
node_pdf_validator -->|"approves"| node_telegram_fallback
node_telegram_fallback -->|"stores"| node_storage
node_storage -->|"uses"| node_schema
node_telegram_fallback -->|"updates"| node_streaks
node_telegram_fallback -->|"updates"| node_badges
node_telegram_fallback -->|"updates"| node_referrals
node_telegram_fallback -->|"updates"| node_analytics
node_http_routes -->|"serves"| node_dashboard_ui
node_http_routes -->|"reads/writes"| node_storage
node_admin_bot -->|"mirrors"| node_http_routes
node_admin_bot -->|"uses"| node_ai_registry
node_summary -->|"uses"| node_ai_registry
node_redis_state -->|"backs"| node_queue

click node_server_index "https://github.com/ahmedallam222/book-bot/blob/main/server/index.ts"
click node_bot_entry "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/index.ts"
click node_http_routes "https://github.com/ahmedallam222/book-bot/blob/main/server/routes.ts"
click node_dashboard_ui "https://github.com/ahmedallam222/book-bot/blob/main/server/dashboard.html"
click node_admin_bot "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/adminAgent/index.ts"
click node_parser "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/bookNameParser.ts"
click node_guards "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/guards.ts"
click node_queue "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/queue.ts"
click node_engine "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/engine.ts"
click node_sources "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/sources.ts"
click node_firecrawl "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/firecrawlParse.ts"
click node_noor_resolver "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/noorBookResolver.ts"
click node_verify "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/verify.ts"
click node_download "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/download.ts"
click node_pdf_validator "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/pdfValidator.ts"
click node_telegram_fallback "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/telegramFallback.ts"
click node_redis_state "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/redis.ts"
click node_storage "https://github.com/ahmedallam222/book-bot/blob/main/server/storage.ts"
click node_schema "https://github.com/ahmedallam222/book-bot/blob/main/shared/schema.ts"
click node_streaks "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/streak.ts"
click node_badges "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/badges.ts"
click node_referrals "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/referral.ts"
click node_analytics "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/analytics.ts"
click node_summary "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/summary.ts"
click node_ai_registry "https://github.com/ahmedallam222/book-bot/blob/main/server/bot/aiProviders/registry.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server_index,node_bot_entry toneBlue
class node_parser,node_guards,node_engine,node_sources,node_firecrawl,node_noor_resolver,node_verify,node_download,node_pdf_validator,node_telegram_fallback toneAmber
class node_queue,node_redis_state,node_storage,node_schema toneMint
class node_streaks,node_badges,node_referrals,node_analytics toneRose
class node_http_routes,node_dashboard_ui,node_admin_bot toneIndigo
class node_summary,node_ai_registry toneTeal
```

## Components

### Runtime

Node.js service entrypoints and request handling

- [Server boot](https://github.com/ahmedallam222/book-bot/blob/main/server/index.ts): bootstrap · Starts bot, workers, HTTP server, and shutdown handling · `server/index.ts`
- [Bot entry](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/index.ts): telegram entry · Wires commands, callbacks, and free-text listeners · `server/bot/index.ts`

### Search & Delivery

Book discovery, validation, and Telegram delivery pipeline

- [Query parser](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/bookNameParser.ts): arabic normalize · Cleans intent words and normalizes Arabic book queries · `server/bot/bookNameParser.ts`
- [Guard checks](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/guards.ts): access control · Ban, maintenance, rate limit, and quota enforcement · `server/bot/guards.ts`
- [Search engine](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/engine.ts): orchestrator · Coordinates discovery across configured source backends · `server/bot/engine.ts`
- [Sources](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/sources.ts): source registry · Defines library backends and their priorities · `server/bot/sources.ts`
- [Firecrawl parse](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/firecrawlParse.ts): extraction · Fetches and extracts candidate content from web sources · `server/bot/firecrawlParse.ts`
- [Noor resolver](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/noorBookResolver.ts): browser boundary · Playwright-assisted resolver for protected Noor Book content · `server/bot/noorBookResolver.ts`
- [Verify](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/verify.ts): url filter · Filters candidate URLs before download · `server/bot/verify.ts`
- [Download](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/download.ts): file fetch · Retrieves candidate files for validation · `server/bot/download.ts`
- [PDF validator](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/pdfValidator.ts): quality gate · Checks file integrity and ambiguous PDF quality · `server/bot/pdfValidator.ts`
- [Telegram send](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/telegramFallback.ts): delivery · Sends validated files and stores reusable Telegram file IDs · `server/bot/telegramFallback.ts`

### State

Redis and Postgres boundaries

- [Queue](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/queue.ts): redis jobs · Async request queue and priority handling · `server/bot/queue.ts`
- [Redis](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/redis.ts): ephemeral state · Queues, cache, rate limits, retries, and source health · `server/bot/redis.ts`
- [Postgres store](https://github.com/ahmedallam222/book-bot/blob/main/server/storage.ts): durable storage · Users, premium status, history, audit data, and file metadata · `server/storage.ts`
- [Schema](https://github.com/ahmedallam222/book-bot/blob/main/shared/schema.ts): shared model · Shared database schema and model definitions · `shared/schema.ts`

### Retention

Post-download engagement loops

- [Streaks](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/streak.ts): retention · Engagement streak tracking · `server/bot/streak.ts`
- [Badges](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/badges.ts): retention · Achievement and reward logic · `server/bot/badges.ts`
- [Referrals](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/referral.ts): retention · Invite and referral attribution · `server/bot/referral.ts`
- [Analytics](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/analytics.ts): retention · Request stats and leaderboard derivations · `server/bot/analytics.ts`

### Admin

Dashboard, API, and admin agent

- [Admin API](https://github.com/ahmedallam222/book-bot/blob/main/server/routes.ts): express routes · Serves dashboard and REST operations · `server/routes.ts`
- [Dashboard](https://github.com/ahmedallam222/book-bot/blob/main/server/dashboard.html): single-page UI · Admin control surface served by the backend · `server/dashboard.html`
- [Admin bot](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/adminAgent/index.ts): telegram ops · Tool-driven admin Telegram control path · `server/bot/adminAgent/index.ts`

### AI Layer

Provider-agnostic AI adapters and consumers

- [Summary](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/summary.ts): ai feature · Generates structured book summaries · `server/bot/summary.ts`
- [AI registry](https://github.com/ahmedallam222/book-bot/blob/main/server/bot/aiProviders/registry.ts): provider adapters · Provider-agnostic failover and adapter registry · `server/bot/aiProviders/registry.ts`

## Connections

- Server boot → Bot entry: starts · Bootstraps the Telegram runtime
- Server boot → Admin API: starts · Mounts the admin HTTP control plane
- Server boot → Redis: connects · Initializes operational Redis state
- Server boot → Postgres store: connects · Initializes durable storage
- Bot entry → Query parser: parses · Normalizes incoming book queries
- Bot entry → Guard checks: checks · Applies access and abuse controls
- Bot entry → Queue: enqueues · Hands accepted requests to background processing
- Queue → Search engine: dispatches · Workers consume queued jobs
- Search engine → Sources: consults · Selects and ranks source backends
- Search engine → Firecrawl parse: extracts · Uses web extraction for discovery
- Search engine → Noor resolver: special-cases · Uses browser mediation for protected Noor content
- Search engine → Verify: filters · Validates candidate URLs before fetch
- Verify → Download: permits · Approved candidates move to download
- Download → PDF validator: checks · Downloaded files are quality-gated
- PDF validator → AI registry: consults · Falls back to AI judgment when needed
- PDF validator → Telegram send: approves · Validated files are delivered to Telegram
- Telegram send → Postgres store: stores · Persists Telegram file IDs and file metadata
- Postgres store → Schema: uses · Schema defines the persistence contract
- Telegram send → Streaks: updates · Triggers retention after successful delivery
- Telegram send → Badges: updates · Awards engagement badges
- Telegram send → Referrals: updates · Records referral effects
- Telegram send → Analytics: updates · Feeds usage and leaderboard stats
- Admin API → Dashboard: serves · Backend hosts the admin dashboard
- Admin API → Postgres store: reads/writes · Admin actions mutate persistent state
- Admin bot → Admin API: mirrors · Admin bot exposes the same operational actions
- Admin bot → AI registry: uses · Conversational admin agent relies on LLM adapters
- Summary → AI registry: uses · Summary generation runs through provider adapters
- Redis → Queue: backs · Queue semantics are Redis-backed