# saurabhhbansal/polyglot-pdf architecture

> Architecture diagram of the GitHub repository saurabhhbansal/polyglot-pdf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saurabhhbansal/polyglot-pdf
- Repository: https://github.com/saurabhhbansal/polyglot-pdf

## Overview

##### Project Type and Purpose
- PolyglotPDF is a full-stack PDF translation tool with a professional web interface (Streamlit) on the front end and a Python-based pipeline on the back end.
- Its purpose is to ingest complex PDFs, extract and analyze their content (text, images, tables), translate that content via multiple AI models or APIs, then reconstruct and deliver a translated PDF with original layout preserved.

##### File Structure
Top-level directories and their roles:
- app/
– main.py: Streamlit UI, session management, file upload/download, progress tracking
- pdf_processing/
– pdf_translate.py: Core pipeline for extraction, analysis, layout clustering, reconstruction
- translation_models/
– translators.py: Wrappers for Google Gemini, Qwen, Helsinki-NLP, Google Translate, each with rate limiting and batch logic
- test-manuals/: Static sample PDFs for manual testing
- .devcontainer/: Dev environment config
- requirements.txt: Python dependencies
- .gitignore: excludes outputs, intermediate files

##### Architectural Patterns
- Layered pipeline architecture: Input → Extraction → Analysis → Translation → Reconstruction → Output
- Modular separation: UI layer, processing layer, model layer, external services
- Session and resource management: temporary files, garbage collection, caching

##### Main Components
a. Frontend (Streamlit)
– File upload/download
– Session & UUID tracking
– Real-time progress updates
– API key management
b. Core Processing Engine
– Text & image extraction (PyMuPDF, Spire.PDF)
– Table detection (Camelot)
– Layout analysis & clustering
– Reconstruction (ReportLab)
c. Translation Layer
– Google Gemini 2.0 Flash (Generative AI API)
– Qwen 3 (OpenRouter API)
– Helsinki-NLP (local MarianMT + PyTorch)
– Google Translate fallback
d. External Services & Storage
– External translation APIs (with rate limiters)
– Temporary storage (intermediate PDF objects, image buffers)
– Caching layer (session-based translation results)
e. Infrastructure
– Dev container (.devcontainer)
– Dependencies (requirements.txt)

##### Relationships and Data Flows
1. User action in Streamlit → upload PDF
2. Streamlit calls pdf_translate.py →
a. Extract text spans, images, tables
b. Perform layout analysis (cluster spans)
3. pdf_translate.py invokes translators.py → batch translation via chosen model(s)
4. Translated clusters returned → layout reconstruction creates new PDF
5. Reconstructed PDF returned to Streamlit → user download
6. Throughout: session store, temporary files, error handling, progress events

##### Important Architectural Patterns & Principles
- Pipeline pattern with clearly defined stages
- Single Responsibility Principle: each module has a distinct role
- Strategy pattern: interchangeable translation back-ends
- Resource optimization: streaming processing, garbage collection, caching

##### Technologies & Frameworks
- Streamlit, PyMuPDF (fitz), Spire.PDF, Camelot-py, ReportLab
- Hugging Face Transformers, PyTorch, SentencePiece
- Google Generative AI client, OpenRouter API, Google Translate (gtx client)
- NumPy, Pillow, Matplotlib

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Frontend (Streamlit)" 
        direction TB
        UI["Streamlit UI\n(file upload/download,\nsession & UUID,\nprogress updates,\nAPI key management)"]:::frontend
    end

    %% Backend Pipeline
    subgraph "Backend Pipeline"
        direction TB
        CoreEngine["PDF Processing Engine\n(core extraction, analysis,\ntranslation orchestration,\nreconstruction)"]:::core
    end

    %% Translation Layer
    subgraph "Translation Layer"
        direction TB
        Translators["Strategy Wrappers\n(Google Gemini, Qwen,\nHelsinki-NLP, Google Translate)"]:::translation
    end

    %% External Services & Storage
    subgraph "External Services & Storage"
        direction TB
        ExternalAPIs("External Translation APIs\n(rate limited)"):::external
        TempStorage("Temporary Storage\n(intermediate PDFs, images)"):::external
        Cache("Session Cache\n(translation results)"):::external
        ManualTests("Manual Testing Samples"):::external
    end

    %% Infrastructure
    subgraph "Infrastructure"
        direction TB
        DevContainer(".devcontainer\n(dev environment config)"):::infra
        Dependencies("requirements.txt\n(dependencies)"):::infra
    end

    %% Data Flows
    UI -->|"upload PDF"| CoreEngine
    CoreEngine -->|"extract text spans,\nimages, tables"| CoreEngine
    CoreEngine -->|"layout analysis & clustering"| CoreEngine
    CoreEngine -->|"batch translation request"| Translators
    Translators -->|"API calls"| ExternalAPIs
    ExternalAPIs -->|"translations"| Translators
    Translators -->|"translated clusters"| CoreEngine
    CoreEngine -->|"reconstructed PDF"| UI

    %% Side Interactions
    UI --> Cache
    CoreEngine --> TempStorage
    CoreEngine --> Cache
    UI --> ManualTests

    %% Click Events
    click UI "https://github.com/saurabhhbansal/polyglot-pdf/blob/main/app/main.py"
    click CoreEngine "https://github.com/saurabhhbansal/polyglot-pdf/blob/main/pdf_processing/pdf_translate.py"
    click Translators "https://github.com/saurabhhbansal/polyglot-pdf/blob/main/translation_models/translators.py"
    click DevContainer "https://github.com/saurabhhbansal/polyglot-pdf/blob/main/.devcontainer/devcontainer.json"
    click Dependencies "https://github.com/saurabhhbansal/polyglot-pdf/blob/main/requirements.txt"
    click ManualTests "https://github.com/saurabhhbansal/polyglot-pdf/tree/main/test-manuals/"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085;
    classDef core fill:#d4edda,stroke:#155724,color:#155724;
    classDef translation fill:#ffeeba,stroke:#856404,color:#856404;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef infra fill:#f8d7da,stroke:#721c24,color:#721c24;
```
