# freelawproject/doctor architecture

> Architecture diagram of the GitHub repository freelawproject/doctor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/freelawproject/doctor
- Repository: https://github.com/freelawproject/doctor

## Overview

##### Project type and purpose
- This is a standalone, stateless HTTP microservice (“Doctor”) implemented in Django, containerized via Docker, and intended for high-performance document and audio file processing (text extraction, format conversion, OCR, thumbnails, metadata, etc.).
- It exposes a range of REST endpoints (under `/extract/…`, `/utils/…`, `/convert/…`) backed by command-line tools (ffmpeg, pdftotext, tesseract, ghostscript, etc.).

##### File structure
- Top-level:
- docker/            → Dockerfile and helper Makefile for building the container image
- compose.yaml       → Docker Compose setup for local development/testing
- manage.py          → Django management entrypoint
- doctor/            → Django project package
– settings.py, wsgi.py, urls.py, views.py, forms.py
– tasks.py         → potentially scheduled/background work
– lib/             → Python wrappers around converters (text_extraction.py, mojibake.py, utils.py)
– test_assets/     → sample files for integration tests
– tests.py         → end-to-end test suite
- Project metadata: pyproject.toml, .pre-commit-config.yaml, GitHub Actions workflows (CI, lint, tests, deploy).

##### README for additional insights
- Confirms use of gunicorn to serve the Django WSGI app.
- Describes each endpoint’s purpose and what underlying binary or library it invokes.
- Mentions horizontal scalability (Kubernetes or any orchestrator) via multiple gunicorn workers.

##### Main components and interactions
a. Clients
– HTTP clients (curl, downstream services) calling REST endpoints.
b. Ingress / Load Balancer
– Route requests across multiple Doctor instances (Kubernetes Ingress or cloud load balancer).
c. Doctor service
– Docker container running:
- Gunicorn master + N worker processes
- Django application (urls.py → views.py → forms.py → lib/ + tasks.py)
- OS‐level converter binaries (ffmpeg, pdftotext, tesseract, ghostscript, antiword, docx2txt, wpd2html, etc.)
– Local temp file storage (/tmp or mounted volume) to hold uploaded files and intermediate outputs.
d. Optional background queue (if tasks.py is used asynchronously)
– A message broker (Redis or RabbitMQ – not present in file tree but may be introduced if async processing is desired).
e. External services
– Sentry (via SENTRY_DSN) for error logging.
– GitHub Actions → Docker registry → Kubernetes (CI/CD pipeline).

##### Architectural patterns / design principles
- Layered architecture
1) HTTP Layer (Gunicorn + Django WSGI)
2) Application Layer (Django routing, controllers in views.py, request validation in forms.py)
3) Processing Layer (doctor.lib wrappers calling subprocesses to convert/extract)
4) Storage Layer (ephemeral file I/O)
- Stateless microservice: no persistent database, horizontal scaling via identical containers.
- Modularization: each major function (extract, utils, convert) implemented in a separate Django view → Python module → binary wrapper.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client to Ingress
    Client["HTTP Client"]:::network
    Ingress["Load Balancer / Ingress"]:::network
    Client -->|"HTTP POST + multipart file"| Ingress

    %% Doctor Cluster
    subgraph "Doctor Cluster"
        direction TB
        subgraph "Doctor Pod"
            direction TB
            GMaster["Gunicorn Master"]:::runtime
            GWorker["Gunicorn Worker"]:::runtime
            GMaster --> GWorker

            Manage["manage.py"]:::runtime
            Settings["settings.py"]:::runtime
            WSGI["wsgi.py"]:::runtime
            Router["urls.py"]:::runtime
            Views["views.py"]:::runtime
            Forms["forms.py"]:::runtime
            Tasks["tasks.py"]:::runtime

            GWorker --> WSGI
            WSGI --> Router
            Router --> Views
            Views --> Forms
            Views --> Tasks

            subgraph "Lib Wrappers" 
                direction TB
                TextExt["text_extraction.py"]:::conversion
                Mojibake["mojibake.py"]:::conversion
                UtilsLib["utils.py"]:::conversion
            end
            Forms --> TextExt
            Forms --> Mojibake
            Forms --> UtilsLib

            Binaries["OS-level Converter Binaries"]:::conversion
            TextExt --> Binaries
            Mojibake --> Binaries
            UtilsLib --> Binaries

            TempVol["Temp File Storage (/tmp)"]:::storage
            Binaries --> TempVol
            TempVol --> GWorker
        end

        Ingress -->|"HTTP request"| GMaster
        GWorker -->|"HTTP JSON or file response"| Ingress
        GWorker -.->|"Error logs"| Sentry
    end

    %% External services
    Sentry["Sentry"]:::external

    %% CI/CD Pipeline
    subgraph "CI/CD Pipeline"
        direction TB
        Lint["lint.yml"]:::external
        TestsWF["tests.yml"]:::external
        Deploy["deploy.yml"]:::external
        Lint --> TestsWF --> Deploy
    end

    %% Docker Config & Kubernetes
    subgraph "Docker Configuration"
        direction TB
        Dockerfile["Dockerfile"]:::external
        Makefile["Makefile"]:::external
        Compose["compose.yaml"]:::external
    end
    Deploy --> Dockerfile & Makefile & Compose
    Dockerfile & Makefile & Compose --> Kubernetes
    Kubernetes["Kubernetes Cluster"]:::runtime
    Kubernetes --> Ingress

    %% Testing & Tooling
    subgraph "Testing & Project Tooling"
        direction TB
        TestAssets["test_assets/"]:::test
        Tests["tests.py"]:::test
        PyProj["pyproject.toml"]:::tooling
        PreCommit[".pre-commit-config.yaml"]:::tooling
        EnvExample[".env.example"]:::tooling
        DockerIgnore[".dockerignore"]:::tooling
        GitIgnore[".gitignore"]:::tooling
    end

    %% Click Events
    click Dockerfile "https://github.com/freelawproject/doctor/tree/main/docker/Dockerfile"
    click Makefile "https://github.com/freelawproject/doctor/tree/main/docker/Makefile"
    click Compose "https://github.com/freelawproject/doctor/blob/main/compose.yaml"
    click Manage "https://github.com/freelawproject/doctor/blob/main/manage.py"
    click Settings "https://github.com/freelawproject/doctor/blob/main/doctor/settings.py"
    click WSGI "https://github.com/freelawproject/doctor/blob/main/doctor/wsgi.py"
    click Router "https://github.com/freelawproject/doctor/blob/main/doctor/urls.py"
    click Views "https://github.com/freelawproject/doctor/blob/main/doctor/views.py"
    click Forms "https://github.com/freelawproject/doctor/blob/main/doctor/forms.py"
    click Tasks "https://github.com/freelawproject/doctor/blob/main/doctor/tasks.py"
    click TextExt "https://github.com/freelawproject/doctor/blob/main/doctor/lib/text_extraction.py"
    click Mojibake "https://github.com/freelawproject/doctor/blob/main/doctor/lib/mojibake.py"
    click UtilsLib "https://github.com/freelawproject/doctor/blob/main/doctor/lib/utils.py"
    click TestAssets "https://github.com/freelawproject/doctor/tree/main/doctor/test_assets/"
    click Tests "https://github.com/freelawproject/doctor/blob/main/doctor/tests.py"
    click Lint "https://github.com/freelawproject/doctor/blob/main/.github/workflows/lint.yml"
    click TestsWF "https://github.com/freelawproject/doctor/blob/main/.github/workflows/tests.yml"
    click Deploy "https://github.com/freelawproject/doctor/blob/main/.github/workflows/deploy.yml"
    click PyProj "https://github.com/freelawproject/doctor/blob/main/pyproject.toml"
    click PreCommit "https://github.com/freelawproject/doctor/blob/main/.pre-commit-config.yaml"
    click EnvExample "https://github.com/freelawproject/doctor/blob/main/.env.example"
    click DockerIgnore "https://github.com/freelawproject/doctor/blob/main/.dockerignore"
    click GitIgnore "https://github.com/freelawproject/doctor/blob/main/.gitignore"

    %% Styles
    classDef network fill:#BFDBFE,stroke:#2563EB,color:#1E3A8A
    classDef runtime fill:#D1FAE5,stroke:#10B981,color:#065F46
    classDef conversion fill:#FEF3C7,stroke:#F59E0B,color:#92400E
    classDef storage fill:#FDE68A,stroke:#D97706,color:#92400E
    classDef external fill:#E5E7EB,stroke:#6B7280,color:#374151
    classDef test fill:#E0E7FF,stroke:#6366F1,color:#3730A3
    classDef tooling fill:#ECFDF5,stroke:#34D399,color:#065F46
```
