How swiftlatex/htmlizer works
Project type and purpose
• This is a standalone RESTful web service (“htmlizer”) whose sole purpose is to convert uploaded PDFs into HTML.
• Key indicators: the README calls it “A Restful Service Converting PDF to HTML,” there is an app.py/WSGI.py pair, and a worker.py that implies asynchronous task processing.
File structure
• Top-level files:
• app.py / WSGI.py – the web entrypoints serving the REST API (probably via Flask, Django or similar)
• worker.py – background process for off-line or delay-tolerant conversions
• htmlizer.py / pdf2htmlex/ – core conversion logic and any shims around the pdf2htmlex binary or library
• cleaner.py, string_utils.py – post-processing utilities for sanitizing or formatting the converted HTML
• requests_utils.py – helper code for making HTTP requests (possibly for callbacks or external API integration)
• config.py – central configuration (broker URLs, timeouts, paths to converters)
• Dockerfile / docker-compose.yml – containerization and orchestration of web + worker + supporting services
• test.py – unit or integration tests
• Notice there are no explicit database folders, so persistent storage is likely the file system, object store, or ephemeral only.
Main components
a. Client(s)
– Web or CLI clients that upload PDFs and request HTML.
b. API Gateway / Web Service
– app.py served via WSGI.py (Gunicorn/uwsgi). Accepts HTTP requests, handles authentication/validation, and routes to the conversion pipeline.
c. Task Queue / Message Broker
– e.g., Redis or RabbitMQ (referenced in docker-compose.yml) used by app.py to enqueue PDF conversion jobs.
d. Worker Process
– worker.py that pulls jobs from the broker, invokes htmlizer.py (which wraps pdf2htmlex), and then post-processes output via cleaner.py/string_utils.py.
e. Conversion Engine
– pdf2htmlex binary or library, plus compatibility.js/manifest, where the bulk of PDF → HTML transformation happens.
f. Storage / Result Delivery
– local file system or cloud storage for holding generated HTML until the client fetches it, or piping back over HTTP.
g. Utilities and Shared Libraries
– config.py for settings, requests_utils.py for any downstream callbacks, string_utils.py for text handling.
Relationships and interactions
• Client → HTTP POST to API Gateway →
• If synchronous: Gateway invokes conversion directly (htmlizer.py) and streams HTML back.
• If asynchronous: Gateway pushes job to Message Broker and returns a Job ID.
• Worker → consumes job → calls htmlizer.py → calls pdf2htmlex → passes output to cleaner/string_utils → stores HTML → (optional) notifies client via HTTP callback (requests_utils) or client polls status endpoint.
• All services run in containers orchestrated by docker-compose.
Architectural patterns and design principles
• Separation of concerns: web tier vs. worker tier
• Asynchronous processing with a message broker for scalability and reliability
• Utility modules abstracting configuration and HTTP calls
• Containerization for consistent deployments (Dockerfile + docker-compose)
Technologies and frameworks
• Python web framework (Flask/Django) behind WSGI
• Celery or similar for task queue (implied)
• Redis or RabbitMQ as broker
• pdf2htmlex converter
• Docker + Docker-Compose for deployment
• Standard Python libraries (requests, etc.)