How ahmedallam222/book-bot works
1. Overall shape
- This repo is a Telegram-first book search and delivery system with a small HTTP control plane.
- The runtime is a single Node.js/TypeScript service that hosts:
- the user-facing Telegram bot,
- an Express-based admin/API server,
- background queue workers,
- and, optionally, a second admin Telegram bot.
2. Primary entry points
- server/index.ts is the main bootstrap: it starts the bot, workers, and HTTP server, and handles graceful shutdown.
- server/bot/index.ts wires Telegram listeners for commands, callbacks, and free-text handling.
- server/routes.ts exposes the admin dashboard and REST API.
- server/dashboard.html is the single-file admin UI served by the backend.
3. User request flow
- A user sends an Arabic book query in Telegram.
- bookNameParser.ts strips intent words and normalizes Arabic text.
- Guard checks run early: ban, maintenance, rate limit, and daily quota.
- If allowed, the request is enqueued into Redis with priority based on user tier.
- Workers consume the queue and execute the search/download pipeline.
4. Search subsystem
- server/bot/engine.ts orchestrates discovery across the configured Arabic libraries.
- server/bot/sources.ts defines the 14 source backends and their priorities.
- Search uses Firecrawl for extraction, then fuzzy matching for typos or weak title matches.
- noorBookResolver.ts is a special boundary for Cloudflare-protected Noor Book content, using Playwright where needed.
5. Validation and delivery pipeline
- Candidate URLs are filtered through verify.ts and then downloaded in download.ts.
- pdfValidator.ts applies multi-stage checks before a file is delivered:
HTTP probe, PDF magic bytes, text density, page count, and AI judgment when needed.
- If validated, the bot sends the file to Telegram and stores the Telegram file_id for reuse.
- This creates a strong boundary between “found on the web” and “safe to deliver in-chat.”
6. Data and state boundaries
- Redis is the operational state layer:
queues, cache, rate limits, streaks, source health, DLQ, and other short-lived counters.
- Postgres is the durable state layer:
users, premium status, request history, audit data, and cached file metadata.
- shared/schema.ts is the shared schema definition used across server-side data access.
- server/storage.ts is the persistence boundary around Postgres/Drizzle.
7. Engagement and retention systems
- streak.ts, badges.ts, referral.ts, and analytics.ts implement the retention loop.
- These subsystems are intentionally decoupled from delivery: they run after a successful download and are designed to fail open.
- Leaderboards and request stats are derived from canonicalized book titles to avoid duplicate entries.
8. AI-dependent features
- The bot uses AI in two distinct places:
- pdfValidator.ts for judging ambiguous PDFs,
- summary.ts / summaryHandler.ts for generating structured book summaries.
- server/bot/aiProviders/ contains the summary/validation provider adapters and failover registry.
- The AI layer is provider-agnostic by design, so no single vendor is a hard dependency.
9. Admin control plane
- Operators use either the dashboard or the REST API for source toggles, user management, premium grants, broadcasts, and maintenance mode.
- The dashboard is protected by a bearer token and serves as the main observability and operations surface.
- admin.ts, alertWatcher.ts, dailyDigest.ts, and weekly.ts connect operational state back to administrators.
10. Admin AI agent as a separate control path
- server/bot/adminAgent/ is a second Telegram bot for admins only.
- It exposes a tool-driven conversational interface for the same operational actions as the dashboard.
- This is a distinct subsystem with its own LLM chain, conversation memory, loop guards, telemetry, and write-confirmation flow.
11. Resilience and queue semantics
- Work is intentionally asynchronous: the bot acknowledges queue position, then workers process jobs independently.
- Redis-backed retries and a dead-letter queue isolate failures.
- progressWatchdog.ts, failureRetry.ts, and alertWatcher.ts handle stuck jobs, retries, and operator alerts.
- This lets the system survive slow sources, AI outages, and transient Telegram issues without losing user requests.
12. Integration boundaries
- External dependencies materially shape the architecture:
- Telegram is the user interface and delivery channel.
- Firecrawl is the search/extraction layer.
- Redis and Postgres are the main state stores.
- AI providers are used behind failover adapters, not directly by business logic.
- Playwright is only used for the one protected source that needs browser mediation.
- cloudflare/welib-proxy/ is a separate worker-based proxy integration for another source path.
13. Supporting tooling and delivery
- Docker Compose runs the bot, Postgres, and Redis together for local/dev parity.
- script/ and scripts/ contain build, migration, backfill, and maintenance jobs.
- CI runs typecheck, build, and smoke tests from tests/, which are an important architectural safety net because the repo relies on many cross-cutting behaviors.
Main entry points: telegramFallback.ts Telegram send · engine.ts Search engine · server/index.ts Server boot · bot/index.ts Bot entry
Components 25
Runtime
Node.js service entrypoints and request handling
- Server boot · Starts bot, workers, HTTP server, and shutdown handling
server/index.ts - Bot entry · Wires commands, callbacks, and free-text listeners
server/bot/index.ts
Search & Delivery
Book discovery, validation, and Telegram delivery pipeline
- Query parser · Cleans intent words and normalizes Arabic book queries
server/bot/bookNameParser.ts - Guard checks · Ban, maintenance, rate limit, and quota enforcement
server/bot/guards.ts - Search engine · Coordinates discovery across configured source backends
server/bot/engine.ts - Sources · Defines library backends and their priorities
server/bot/sources.ts - Firecrawl parse · Fetches and extracts candidate content from web sources
server/bot/firecrawlParse.ts - Noor resolver · Playwright-assisted resolver for protected Noor Book content
server/bot/noorBookResolver.ts - Verify · Filters candidate URLs before download
server/bot/verify.ts - Download · Retrieves candidate files for validation
server/bot/download.ts - PDF validator · Checks file integrity and ambiguous PDF quality
server/bot/pdfValidator.ts - Telegram send · Sends validated files and stores reusable Telegram file IDs
server/bot/telegramFallback.ts
State
Redis and Postgres boundaries
- Queue · Async request queue and priority handling
server/bot/queue.ts - Redis · Queues, cache, rate limits, retries, and source health
server/bot/redis.ts - Postgres store · Users, premium status, history, audit data, and file metadata
server/storage.ts - Schema · Shared database schema and model definitions
shared/schema.ts
Retention
Post-download engagement loops
Admin
Dashboard, API, and admin agent
AI Layer
Provider-agnostic AI adapters and consumers
- Summary · Generates structured book summaries
server/bot/summary.ts - AI registry · Provider-agnostic failover and adapter registry
server/bot/aiProviders/registry.ts
Connections 28
- Server boot → Bot entry: starts · Bootstraps the Telegram runtime
- Server boot → Admin API: starts · Mounts the admin HTTP control plane
- Server boot → Redis: connects · Initializes operational Redis state
- Server boot → Postgres store: connects · Initializes durable storage
- Bot entry → Query parser: parses · Normalizes incoming book queries
- Bot entry → Guard checks: checks · Applies access and abuse controls
- Bot entry → Queue: enqueues · Hands accepted requests to background processing
- Queue → Search engine: dispatches · Workers consume queued jobs
- Search engine → Sources: consults · Selects and ranks source backends
- Search engine → Firecrawl parse: extracts · Uses web extraction for discovery
- Search engine → Noor resolver: special-cases · Uses browser mediation for protected Noor content
- Search engine → Verify: filters · Validates candidate URLs before fetch
- Verify → Download: permits · Approved candidates move to download
- Download → PDF validator: checks · Downloaded files are quality-gated
- PDF validator → AI registry: consults · Falls back to AI judgment when needed
- PDF validator → Telegram send: approves · Validated files are delivered to Telegram
- Telegram send → Postgres store: stores · Persists Telegram file IDs and file metadata
- Postgres store → Schema: uses · Schema defines the persistence contract
- Telegram send → Streaks: updates · Triggers retention after successful delivery
- Telegram send → Badges: updates · Awards engagement badges
- Telegram send → Referrals: updates · Records referral effects
- Telegram send → Analytics: updates · Feeds usage and leaderboard stats
- Admin API → Dashboard: serves · Backend hosts the admin dashboard
- Admin API → Postgres store: reads/writes · Admin actions mutate persistent state
- Admin bot → Admin API: mirrors · Admin bot exposes the same operational actions
- Admin bot → AI registry: uses · Conversational admin agent relies on LLM adapters
- Summary → AI registry: uses · Summary generation runs through provider adapters
- Redis → Queue: backs · Queue semantics are Redis-backed