# codad5/pdfz architecture

> Architecture diagram of the GitHub repository codad5/pdfz, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/codad5/pdfz
- Repository: https://github.com/codad5/pdfz

## Overview

##### Project Type and Purpose
- PDFz is a microservices‐based document‐processing platform. It comprises:
- An API Service (Node.js/Express + TypeScript) for file uploads, job orchestration, progress tracking, and model management.
- An Extractor Service (Rust) that consumes jobs, runs OCR or LLM‐based extraction engines (Tesseract, Ollama), and writes results.
- Supporting infra: RabbitMQ for job queues, Redis for state/progress caching, shared file storage for PDFs and outputs, and an optional Ollama container for local LLM hosting.

##### File Structure
- Top-level:
- docker-compose.yml orchestrates five containers: api, extractor, redis, rabbitmq, (and optionally ollama).
- `api/`: Express server in TypeScript; routes, helpers, Redis and RabbitMQ clients, static web UI under `public`.
- `extractor/`: Rust workspace; `main.rs` and `worker.rs` launch controllers, engine handlers (Tesseract, Ollama), Redis job services.
- Config files: .env.sample, nodemon.json, tsconfig.json, Cargo.toml, Dockerfiles in each service.

##### Extract Key Insights from the README
- Microservices flow:
- Client → API (upload, process, progress, content, model).
- API → RabbitMQ extractor queue.
- Extractor → Redis updates (job and model download progress) and writes extracted text JSON.
- API polls Redis to return statuses.
- Optional Ollama container for model‐based extraction.
- Tech stack: Express+TS, Rust, Redis, RabbitMQ, Tesseract, Ollama LLM.

##### Main Components
a. Frontend / Client
- Browser or CLI client – requests the API endpoints.
- Static Web UI served by API under `/web`.
b. API Service (Node.js/Express + TS)
- Upload endpoint writes PDF to shared volume.
- `/process` enqueues job to RabbitMQ.
- Polls Redis for progress and content.
- Model endpoints (`/model/pull`, `/model/progress`, `/models`) enqueue or read from Redis.
c. RabbitMQ
- Exchange/queue for file‐processing jobs and model‐download jobs.
d. Extractor Service (Rust)
- Worker subscribes to RabbitMQ.
- Dispatches to extraction engines: Tesseract OCR or Ollama LLM.
- Updates job progress and final content in Redis (via BaseRedisService, FileProcessingService, ModelDownloadService).
- Writes JSON output to shared storage.
e. Redis
- Caching job state, progress, model‐download state.
f. Shared Storage Volume
- `/shared_storage/upload/pdf/` – holds uploaded PDFs.
- `/shared_storage/output/json/` – stores extracted JSON.
g. (Optional) Ollama Container
- Serves locally hosted LLMs via HTTP API.

##### Relationships and Data Flows
1. Client → API: upload, process request, polling endpoints.
2. API → RabbitMQ: publish “file‐processing” or “model‐download” messages.
3. RabbitMQ → Extractor: subscribe and consume messages.
4. Extractor → Redis: update progress / status / model‐download progress.
5. Extractor → Shared Storage: write extracted JSON.
6. API → Redis: read job and model state for polling endpoints.
7. Extractor → (optional) Ollama: HTTP calls for model‐based extraction.

##### Architectural Patterns and Design Principles
- Microservices with asynchronous messaging (RabbitMQ) for decoupling and scalability.
- Shared storage pattern for large file handoff instead of embedding binary data in messages.
- Redis as a fast key/value store for ephemeral state and progress tracking.
- Clear separation of concerns: API handles ingress, Extractor handles CPU‐intensive tasks.
- Plugin‐style “engine” handlers in Rust to support multiple OCR/LLM engines.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients
    subgraph "Client"
        direction TB
        Browser["Browser / CLI Client"]:::client
    end

    %% API Service
    subgraph "API Service"    
        direction TB
        Server["api/src/server.ts"]:::service
        UploadHelper["uploadhelper.ts"]:::service
        ResponseHelper["response.ts"]:::service
        RabbitClient["rabbitmq.ts"]:::service
        BaseRedis["BaseRedisService.ts"]:::service
        FileRedis["FileProcessingService.ts"]:::service
        ModelRedis["ModelDownloadService.ts"]:::service
        TypesQueue["queue.ts"]:::service
        TypesReq["request.ts"]:::service
        TypesRes["response.ts"]:::service
        StaticUI["index.html"]:::service
    end

    %% RabbitMQ
    subgraph "RabbitMQ"
        direction TB
        Queue["Exchange / Queue"]:::queue
    end

    %% Extractor Service
    subgraph "Extractor Service"
        direction TB
        Main["main.rs"]:::service
        Worker["worker.rs"]:::service
        ControllersMod["mod.rs"]:::service
        QueueHandler["queue_handler.rs"]:::service
        Tesseract["tesseract.rs"]:::engine
        OllamaEngine["ollama.rs"]:::engine
        FileHelper["file_helper.rs"]:::service
        RedisLib["redis.rs"]:::service
        ExtractorLib["extractor.rs"]:::service
        TypesEngine["engine_handler.rs"]:::service
    end

    %% Redis
    subgraph "Redis"
        direction TB
        RedisDB["Redis (Progress & State)"]:::db
    end

    %% Shared Storage
    subgraph "Shared Storage"
        direction TB
        StoragePDF["/upload/pdf"]:::storage
        StorageJSON["/output/json"]:::storage
    end

    %% Ollama (Optional)
    subgraph "Ollama Container"
        direction TB
        OllamaAPI["Ollama LLM API"]:::engine
    end

    %% Connections
    Browser -->|upload PDF, process request| Server
    Server -->|uses| UploadHelper
    Server -->|uses| ResponseHelper
    Server -->|publishes job| RabbitClient
    Server -->|reads/writes| BaseRedis
    Server -->|reads/writes| FileRedis
    Server -->|reads/writes| ModelRedis
    Server -->|serves UI| StaticUI

    RabbitClient -->|publish file-processing/model-download| Queue

    Queue -->|consume messages| Worker
    Worker -->|initializes| Main
    Worker -->|uses| QueueHandler
    QueueHandler -->|dispatch to| Tesseract
    QueueHandler -->|dispatch to| OllamaEngine
    Tesseract -->|reads/writes| FileHelper
    Tesseract -->|updates| RedisLib
    Tesseract -->|writes output| StorageJSON
    OllamaEngine -->|calls HTTP| OllamaAPI
    OllamaEngine -->|updates| RedisLib
    OllamaEngine -->|writes output| StorageJSON

    UploadHelper -->|store PDF| StoragePDF

    BaseRedis -->|poll status| RedisDB
    FileRedis -->|update/read progress| RedisDB
    ModelRedis -->|update/read model state| RedisDB
    RedisLib -->|update progress| RedisDB

    %% Click Events
    click Server "https://github.com/codad5/pdfz/blob/master/api/src/server.ts"
    click UploadHelper "https://github.com/codad5/pdfz/blob/master/api/src/helpers/uploadhelper.ts"
    click ResponseHelper "https://github.com/codad5/pdfz/blob/master/api/src/helpers/response.ts"
    click RabbitClient "https://github.com/codad5/pdfz/blob/master/api/src/lib/rabbitmq.ts"
    click BaseRedis "https://github.com/codad5/pdfz/blob/master/api/src/lib/redis/BaseRedisService.ts"
    click FileRedis "https://github.com/codad5/pdfz/blob/master/api/src/lib/redis/FileProcessingService.ts"
    click ModelRedis "https://github.com/codad5/pdfz/blob/master/api/src/lib/redis/ModelDownloadService.ts"
    click TypesQueue "https://github.com/codad5/pdfz/blob/master/api/src/types/queue.ts"
    click TypesReq "https://github.com/codad5/pdfz/blob/master/api/src/types/request.ts"
    click TypesRes "https://github.com/codad5/pdfz/blob/master/api/src/types/response.ts"
    click StaticUI "https://github.com/codad5/pdfz/blob/master/api/public/index.html"

    click Main "https://github.com/codad5/pdfz/blob/master/extractor/src/main.rs"
    click Worker "https://github.com/codad5/pdfz/blob/master/extractor/src/worker.rs"
    click ControllersMod "https://github.com/codad5/pdfz/blob/master/extractor/src/controllers/mod.rs"
    click QueueHandler "https://github.com/codad5/pdfz/blob/master/extractor/src/controllers/queue_handler.rs"
    click Tesseract "https://github.com/codad5/pdfz/blob/master/extractor/src/engine/tesseract.rs"
    click OllamaEngine "https://github.com/codad5/pdfz/blob/master/extractor/src/engine/ollama.rs"
    click FileHelper "https://github.com/codad5/pdfz/blob/master/extractor/src/helper/file_helper.rs"
    click RedisLib "https://github.com/codad5/pdfz/blob/master/extractor/src/libs/redis.rs"
    click ExtractorLib "https://github.com/codad5/pdfz/blob/master/extractor/src/libs/extractor.rs"
    click TypesEngine "https://github.com/codad5/pdfz/blob/master/extractor/src/types/engine_handler.rs"

    %% Styles
    classDef client fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef service fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef queue fill:#ffe6cc,stroke:#333,stroke-width:1px
    classDef db fill:#e6ccff,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef storage fill:#ffffcc,stroke:#333,stroke-width:1px
    classDef engine fill:#ffcccc,stroke:#333,stroke-width:1px
```
