How codad5/pdfz works
Project Type and Purpose
• PDFz is a microservices‐based document‐processing platform. It comprises:
• An API Service (Node.js/Express + TypeScript) for file uploads, job orchestration, progress tracking, and model management.
• An Extractor Service (Rust) that consumes jobs, runs OCR or LLM‐based extraction engines (Tesseract, Ollama), and writes results.
• Supporting infra: RabbitMQ for job queues, Redis for state/progress caching, shared file storage for PDFs and outputs, and an optional Ollama container for local LLM hosting.
File Structure
• Top-level:
• docker-compose.yml orchestrates five containers: api, extractor, redis, rabbitmq, (and optionally ollama).
• api/: Express server in TypeScript; routes, helpers, Redis and RabbitMQ clients, static web UI under public.
• extractor/: Rust workspace; main.rs and worker.rs launch controllers, engine handlers (Tesseract, Ollama), Redis job services.
• Config files: .env.sample, nodemon.json, tsconfig.json, Cargo.toml, Dockerfiles in each service.
Extract Key Insights from the README
• Microservices flow:
• Client → API (upload, process, progress, content, model).
• API → RabbitMQ extractor queue.
• Extractor → Redis updates (job and model download progress) and writes extracted text JSON.
• API polls Redis to return statuses.
• Optional Ollama container for model‐based extraction.
• Tech stack: Express+TS, Rust, Redis, RabbitMQ, Tesseract, Ollama LLM.
Main Components
a. Frontend / Client
• Browser or CLI client – requests the API endpoints.
• Static Web UI served by API under /web.
b. API Service (Node.js/Express + TS)
• Upload endpoint writes PDF to shared volume.
• /process enqueues job to RabbitMQ.
• Polls Redis for progress and content.
• Model endpoints (/model/pull, /model/progress, /models) enqueue or read from Redis.
c. RabbitMQ
• Exchange/queue for file‐processing jobs and model‐download jobs.
d. Extractor Service (Rust)
• Worker subscribes to RabbitMQ.
• Dispatches to extraction engines: Tesseract OCR or Ollama LLM.
• Updates job progress and final content in Redis (via BaseRedisService, FileProcessingService, ModelDownloadService).
• Writes JSON output to shared storage.
e. Redis
• Caching job state, progress, model‐download state.
f. Shared Storage Volume
• /shared_storage/upload/pdf/ – holds uploaded PDFs.
• /shared_storage/output/json/ – stores extracted JSON.
g. (Optional) Ollama Container
• Serves locally hosted LLMs via HTTP API.
Relationships and Data Flows
1. Client → API: upload, process request, polling endpoints.
2. API → RabbitMQ: publish “file‐processing” or “model‐download” messages.
3. RabbitMQ → Extractor: subscribe and consume messages.
4. Extractor → Redis: update progress / status / model‐download progress.
5. Extractor → Shared Storage: write extracted JSON.
6. API → Redis: read job and model state for polling endpoints.
7. Extractor → (optional) Ollama: HTTP calls for model‐based extraction.
Architectural Patterns and Design Principles
• Microservices with asynchronous messaging (RabbitMQ) for decoupling and scalability.
• Shared storage pattern for large file handoff instead of embedding binary data in messages.
• Redis as a fast key/value store for ephemeral state and progress tracking.
• Clear separation of concerns: API handles ingress, Extractor handles CPU‐intensive tasks.
• Plugin‐style “engine” handlers in Rust to support multiple OCR/LLM engines.