# shafqaatmalik/llm-based-invoice-ocr architecture

> Architecture diagram of the GitHub repository shafqaatmalik/llm-based-invoice-ocr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shafqaatmalik/llm-based-invoice-ocr
- Repository: https://github.com/shafqaatmalik/llm-based-invoice-ocr

## Overview

This is a web-based invoice OCR pipeline: users upload a PDF or image, choose paid vision-model extraction or open-source OCR, and receive aggregated, schema-checked JSON. The graph follows the documented multi-page conversion and extraction stages and the calls visible in the sampled source. The README documents both extraction modes; no persistent application store is described.

Main entry points: [`src/backend/main.py`](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py) (FastAPI endpoint), [`src/backend/together_api.py`](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/together_api.py) (Together extractor), [`src/backend/open_source_mode.py`](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/open_source_mode.py) (OCR heuristics), [`src/frontend/gradio_app.py`](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/frontend/gradio_app.py) (Gradio interface)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["User interface"]
  node_gradio["Gradio interface<br/>[gradio_app.py]"]
end

subgraph group_pipeline["Invoice pipeline"]
  node_api["FastAPI endpoint<br/>[main.py]"]
  node_pdf["PDF conversion<br/>[main.py]"]
  node_page_images["Page images<br/>[main.py]"]
  node_router["Mode dispatch<br/>[main.py]"]
  node_aggregate["Page aggregation<br/>[parser.py]"]
  node_validate["Schema validation<br/>[parser.py]"]
  node_result["Structured JSON<br/>[main.py]"]
end

subgraph group_extractors["Extraction modes"]
  node_paid["Together extractor<br/>[together_api.py]"]
  node_opensource["OCR heuristics"]
end

subgraph group_integrations["External services"]
  node_together{{"Together AI"}}
  node_tesseract{{"Tesseract OCR"}}
  node_poppler{{"Poppler renderer"}}
end

node_user(("Invoice user"))

node_user -->|"uploads and selects"| node_gradio
node_gradio -->|"posts file and mode"| node_api
node_api -->|"converts PDFs"| node_pdf
node_pdf -->|"renders pages"| node_poppler
node_pdf -->|"writes page images"| node_page_images
node_api -->|"copies image uploads"| node_page_images
node_api -->|"dispatches pages"| node_router
node_router -->|"paid mode"| node_paid
node_router -->|"open-source mode"| node_opensource
node_paid -->|"reads image"| node_page_images
node_paid -->|"sends image request"| node_together
node_together -->|"returns model response"| node_paid
node_opensource -->|"reads image"| node_page_images
node_opensource -->|"runs OCR"| node_tesseract
node_router -->|"aggregates page results"| node_aggregate
node_aggregate -->|"passes merged fields"| node_validate
node_api -->|"returns validated JSON"| node_result
node_result -->|"displays response"| node_gradio

click node_gradio "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/frontend/gradio_app.py"
click node_api "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py"
click node_pdf "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py"
click node_page_images "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py"
click node_router "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py"
click node_aggregate "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/parser.py"
click node_validate "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/parser.py"
click node_result "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py"
click node_paid "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/together_api.py"
click node_opensource "https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/open_source_mode.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gradio,node_user toneBlue
class node_api,node_pdf,node_page_images,node_router,node_aggregate,node_validate,node_result toneAmber
class node_paid,node_opensource toneMint
class node_together,node_tesseract,node_poppler toneRose
```

## Components

### User interface

- [Gradio interface](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/frontend/gradio_app.py): `src/frontend/gradio_app.py`

### Invoice pipeline

- [FastAPI endpoint](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py): `src/backend/main.py`
- [PDF conversion](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py): `src/backend/main.py`
- [Page images](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py): `src/backend/main.py`
- [Mode dispatch](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py): `src/backend/main.py`
- [Page aggregation](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/parser.py): `src/backend/parser.py`
- [Schema validation](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/parser.py): `src/backend/parser.py`
- [Structured JSON](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/main.py): `src/backend/main.py`

### Extraction modes

- [Together extractor](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/together_api.py): `src/backend/together_api.py`
- [OCR heuristics](https://github.com/shafqaatmalik/llm-based-invoice-ocr/blob/main/src/backend/open_source_mode.py): `src/backend/open_source_mode.py`

### External services

- **Together AI**
- **Tesseract OCR**
- **Poppler renderer**

### Other components

- **Invoice user**

## Connections

- Invoice user → Gradio interface: uploads and selects
- Gradio interface → FastAPI endpoint: posts file and mode
- FastAPI endpoint → PDF conversion: converts PDFs
- PDF conversion → Poppler renderer: renders pages
- PDF conversion → Page images: writes page images
- FastAPI endpoint → Page images: copies image uploads
- FastAPI endpoint → Mode dispatch: dispatches pages
- Mode dispatch → Together extractor: paid mode
- Mode dispatch → OCR heuristics: open-source mode
- Together extractor → Page images: reads image
- Together extractor → Together AI: sends image request
- Together AI → Together extractor: returns model response
- OCR heuristics → Page images: reads image
- OCR heuristics → Tesseract OCR: runs OCR
- Mode dispatch → Page aggregation: aggregates page results
- Page aggregation → Schema validation: passes merged fields
- FastAPI endpoint → Structured JSON: returns validated JSON
- Structured JSON → Gradio interface: displays response