How freelawproject/doctor works
Project type and purpose
• This is a standalone, stateless HTTP microservice (“Doctor”) implemented in Django, containerized via Docker, and intended for high-performance document and audio file processing (text extraction, format conversion, OCR, thumbnails, metadata, etc.).
• It exposes a range of REST endpoints (under /extract/…, /utils/…, /convert/…) backed by command-line tools (ffmpeg, pdftotext, tesseract, ghostscript, etc.).
File structure
• Top-level:
• docker/ → Dockerfile and helper Makefile for building the container image
• compose.yaml → Docker Compose setup for local development/testing
• manage.py → Django management entrypoint
• doctor/ → Django project package
– settings.py, wsgi.py, urls.py, views.py, forms.py
– tasks.py → potentially scheduled/background work
– lib/ → Python wrappers around converters (text_extraction.py, mojibake.py, utils.py)
– test_assets/ → sample files for integration tests
– tests.py → end-to-end test suite
• Project metadata: pyproject.toml, .pre-commit-config.yaml, GitHub Actions workflows (CI, lint, tests, deploy).
README for additional insights
• Confirms use of gunicorn to serve the Django WSGI app.
• Describes each endpoint’s purpose and what underlying binary or library it invokes.
• Mentions horizontal scalability (Kubernetes or any orchestrator) via multiple gunicorn workers.
Main components and interactions
a. Clients
– HTTP clients (curl, downstream services) calling REST endpoints.
b. Ingress / Load Balancer
– Route requests across multiple Doctor instances (Kubernetes Ingress or cloud load balancer).
c. Doctor service
– Docker container running:
• Gunicorn master + N worker processes
• Django application (urls.py → views.py → forms.py → lib/ + tasks.py)
• OS‐level converter binaries (ffmpeg, pdftotext, tesseract, ghostscript, antiword, docx2txt, wpd2html, etc.)
– Local temp file storage (/tmp or mounted volume) to hold uploaded files and intermediate outputs.
d. Optional background queue (if tasks.py is used asynchronously)
– A message broker (Redis or RabbitMQ – not present in file tree but may be introduced if async processing is desired).
e. External services
– Sentry (via SENTRY_DSN) for error logging.
– GitHub Actions → Docker registry → Kubernetes (CI/CD pipeline).
Architectural patterns / design principles
• Layered architecture
1) HTTP Layer (Gunicorn + Django WSGI)
2) Application Layer (Django routing, controllers in views.py, request validation in forms.py)
3) Processing Layer (doctor.lib wrappers calling subprocesses to convert/extract)
4) Storage Layer (ephemeral file I/O)
• Stateless microservice: no persistent database, horizontal scaling via identical containers.
• Modularization: each major function (extract, utils, convert) implemented in a separate Django view → Python module → binary wrapper.