How saurabhhbansal/polyglot-pdf works
Project Type and Purpose
• PolyglotPDF is a full-stack PDF translation tool with a professional web interface (Streamlit) on the front end and a Python-based pipeline on the back end.
• Its purpose is to ingest complex PDFs, extract and analyze their content (text, images, tables), translate that content via multiple AI models or APIs, then reconstruct and deliver a translated PDF with original layout preserved.
File Structure
Top-level directories and their roles:
• app/
– main.py: Streamlit UI, session management, file upload/download, progress tracking
• pdf_processing/
– pdf_translate.py: Core pipeline for extraction, analysis, layout clustering, reconstruction
• translation_models/
– translators.py: Wrappers for Google Gemini, Qwen, Helsinki-NLP, Google Translate, each with rate limiting and batch logic
• test-manuals/: Static sample PDFs for manual testing
• .devcontainer/: Dev environment config
• requirements.txt: Python dependencies
• .gitignore: excludes outputs, intermediate files
Architectural Patterns
• Layered pipeline architecture: Input → Extraction → Analysis → Translation → Reconstruction → Output
• Modular separation: UI layer, processing layer, model layer, external services
• Session and resource management: temporary files, garbage collection, caching
Main Components
a. Frontend (Streamlit)
– File upload/download
– Session & UUID tracking
– Real-time progress updates
– API key management
b. Core Processing Engine
– Text & image extraction (PyMuPDF, Spire.PDF)
– Table detection (Camelot)
– Layout analysis & clustering
– Reconstruction (ReportLab)
c. Translation Layer
– Google Gemini 2.0 Flash (Generative AI API)
– Qwen 3 (OpenRouter API)
– Helsinki-NLP (local MarianMT + PyTorch)
– Google Translate fallback
d. External Services & Storage
– External translation APIs (with rate limiters)
– Temporary storage (intermediate PDF objects, image buffers)
– Caching layer (session-based translation results)
e. Infrastructure
– Dev container (.devcontainer)
– Dependencies (requirements.txt)
Relationships and Data Flows
1. User action in Streamlit → upload PDF
2. Streamlit calls pdf_translate.py →
a. Extract text spans, images, tables
b. Perform layout analysis (cluster spans)
3. pdf_translate.py invokes translators.py → batch translation via chosen model(s)
4. Translated clusters returned → layout reconstruction creates new PDF
5. Reconstructed PDF returned to Streamlit → user download
6. Throughout: session store, temporary files, error handling, progress events
Important Architectural Patterns & Principles
• Pipeline pattern with clearly defined stages
• Single Responsibility Principle: each module has a distinct role
• Strategy pattern: interchangeable translation back-ends
• Resource optimization: streaming processing, garbage collection, caching
Technologies & Frameworks
• Streamlit, PyMuPDF (fitz), Spire.PDF, Camelot-py, ReportLab
• Hugging Face Transformers, PyTorch, SentencePiece
• Google Generative AI client, OpenRouter API, Google Translate (gtx client)
• NumPy, Pillow, Matplotlib