# abinash123hg/document-understanding-agent architecture

> Architecture diagram of the GitHub repository abinash123hg/document-understanding-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abinash123hg/document-understanding-agent
- Repository: https://github.com/abinash123hg/document-understanding-agent

## Overview

This local document-QA application turns uploaded files into document-scoped, cited answers, with a per-page choice between embedded text and handwriting recognition. The API coordinates ingestion and chat; processing covers extraction through chunking, then hybrid retrieval and verified local-model answering. Storage combines JSON chunks with a vector index. README documents API, isolation, refusal and verification behavior; detailed internal calls are shown only where the excerpts support them, so unsampled retrieval/indexing and UI links are omitted.

Main entry points: [`backend/main.py`](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/main.py) (HTTP API), [`backend/processor.py`](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/processor.py) (Extract and chunk), [`backend/retriever.py`](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/retriever.py) (Hybrid retrieval), [`backend/storage.py`](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/storage.py) (Chunk store)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and interface"]
  node_ui["Browser UI"]
  node_apiapp["HTTP API<br/>[main.py]"]
  node_settings["Settings<br/>[config.py]"]
end

subgraph group_ingest["Document ingestion"]
  node_extract["Extract and chunk<br/>[processor.py]"]
  node_image["Image preparation<br/>[imaging.py]"]
  node_htr["Handwriting recognition<br/>[handwriting.py]"]
end

subgraph group_qa["Grounded Q&amp;A"]
  node_retrieval["Hybrid retrieval<br/>[retriever.py]"]
  node_answer["Answer and verification<br/>[llm.py]"]
end

subgraph group_state["Index and state"]
  node_jsonstore[("Chunk store<br/>[storage.py]")]
  node_vectors[("Vector index<br/>[retriever.py]")]
end

node_user(("User"))
node_ollama["Local Ollama"]

node_user -->|"uses"| node_ui
node_ui -->|"HTTP requests"| node_apiapp
node_apiapp -->|"processes upload"| node_extract
node_extract -->|"prepares scanned pages"| node_image
node_extract -->|"recognizes handwriting"| node_htr
node_image -.->|"supplies line crops"| node_htr
node_extract -.->|"stores chunks"| node_jsonstore
node_apiapp -->|"retrieves for chat"| node_retrieval
node_retrieval -->|"loads document chunks"| node_jsonstore
node_retrieval -->|"dense search"| node_vectors
node_apiapp -.->|"generates and verifies"| node_answer
node_answer -->|"requests model output"| node_ollama
node_apiapp -->|"lists documents"| node_jsonstore
node_apiapp -->|"uses configuration"| node_settings

click node_ui "https://github.com/abinash123hg/document-understanding-agent/tree/main/frontend"
click node_apiapp "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/main.py"
click node_settings "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/config.py"
click node_extract "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/processor.py"
click node_image "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/imaging.py"
click node_htr "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/handwriting.py"
click node_jsonstore "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/storage.py"
click node_vectors "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/retriever.py"
click node_retrieval "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/retriever.py"
click node_answer "https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/llm.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_apiapp,node_settings,node_user toneBlue
class node_extract,node_image,node_htr toneAmber
class node_retrieval,node_answer toneMint
class node_jsonstore,node_vectors toneRose
class node_ollama toneIndigo
```

## Components

### API and interface

- [Browser UI](https://github.com/abinash123hg/document-understanding-agent/tree/main/frontend): `frontend`
- [HTTP API](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/main.py): `backend/main.py`
- [Settings](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/config.py): `backend/config.py`

### Document ingestion

- [Extract and chunk](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/processor.py): `backend/processor.py`
- [Image preparation](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/imaging.py): `backend/imaging.py`
- [Handwriting recognition](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/handwriting.py): `backend/handwriting.py`

### Grounded Q&A

- [Hybrid retrieval](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/retriever.py): `backend/retriever.py`
- [Answer and verification](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/llm.py): `backend/llm.py`

### Index and state

- [Chunk store](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/storage.py): `backend/storage.py`
- [Vector index](https://github.com/abinash123hg/document-understanding-agent/blob/main/backend/retriever.py): `backend/retriever.py`

### Other components

- **User**
- **Local Ollama**

## Connections

- User → Browser UI: uses (evidence: `README.md`)
- Browser UI → HTTP API: HTTP requests (evidence: `README.md`)
- HTTP API → Extract and chunk: processes upload (evidence: `backend/main.py`)
- Extract and chunk → Image preparation: prepares scanned pages (evidence: `backend/processor.py`)
- Extract and chunk → Handwriting recognition: recognizes handwriting (evidence: `backend/processor.py`)
- Image preparation → Handwriting recognition: supplies line crops (evidence: `backend/handwriting.py`)
- Extract and chunk → Chunk store: stores chunks (evidence: `backend/processor.py`)
- HTTP API → Hybrid retrieval: retrieves for chat (evidence: `backend/main.py`)
- Hybrid retrieval → Chunk store: loads document chunks (evidence: `backend/retriever.py`)
- Hybrid retrieval → Vector index: dense search (evidence: `README.md`)
- HTTP API → Answer and verification: generates and verifies (evidence: `backend/main.py`)
- Answer and verification → Local Ollama: requests model output (evidence: `backend/llm.py`)
- HTTP API → Chunk store: lists documents (evidence: `backend/main.py`)
- HTTP API → Settings: uses configuration (evidence: `backend/main.py`)
