# devprad-ml/real_project architecture

> Architecture diagram of the GitHub repository devprad-ml/real_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devprad-ml/real_project
- Repository: https://github.com/devprad-ml/real_project

## Overview

This repository implements a document-processing backend with a PostgreSQL-backed job queue and asynchronous worker. The visible processing workflow normalizes received documents, OCRs pages, then chunks and embeds their text into tenant-specific vector collections; handlers record status changes in an append-only audit log. The sampled API exposes health checking, but does not show document submission or retrieval endpoints. The README is empty, so broader capabilities are limited to what the source index and excerpts support.

Main entry points: [`src/app/jobs/handlers.py`](https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/handlers.py) (Job handlers), [`src/app/jobs/queue.py`](https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/queue.py) (Job queue), [`src/app/db.py`](https://github.com/devprad-ml/real_project/blob/main/src/app/db.py) (DB sessions), [`src/app/audit/log.py`](https://github.com/devprad-ml/real_project/blob/main/src/app/audit/log.py) (Audit writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_api["API entrypoint<br/>[main.py]"]
  node_health["Health check<br/>[health.py]"]
  node_config["Settings<br/>[config.py]"]
end

subgraph group_processing["Document processing"]
  node_normalizer["PDF normalization<br/>[normalize.py]"]
  node_ocr["Page OCR<br/>[ocr.py]"]
  node_chunker["Text chunking<br/>[chunk.py]"]
  node_embedder["Embedding<br/>[embed.py]"]
end

subgraph group_persistence["Persistence"]
  node_database[("PostgreSQL")]
  node_dbsession["DB sessions<br/>[db.py]"]
  node_documents["Documents<br/>[document.py]"]
  node_pages["OCR pages<br/>[document_page.py]"]
  node_jobmodel["Job records<br/>[job.py]"]
  node_auditmodel["Audit records<br/>[audit_log.py]"]
  node_clients["Tenant clients<br/>[client.py]"]
  node_blob["Blob storage adapter<br/>[blob.py]"]
  node_vectors["Vector storage adapter<br/>[vectors.py]"]
  node_vectorstore[("Qdrant")]
end

subgraph group_operations["Jobs and audit"]
  node_worker["Worker loop<br/>[worker.py]"]
  node_queue["Job queue<br/>[queue.py]"]
  node_handlers["Job handlers<br/>[handlers.py]"]
  node_audit["Audit writer<br/>[log.py]"]
end

node_api -->|"mounts"| node_health
node_api -->|"uses"| node_dbsession
node_health -->|"checks"| node_dbsession
node_dbsession -->|"loads settings"| node_config
node_dbsession -.->|"opens sessions"| node_database
node_worker -->|"opens sessions"| node_dbsession
node_worker -->|"claims and retries"| node_queue
node_worker -->|"dispatches jobs"| node_handlers
node_queue -->|"creates and claims"| node_jobmodel
node_queue -->|"updates status"| node_documents
node_queue -->|"records failures"| node_audit
node_queue -->|"reads retry settings"| node_config
node_queue -.->|"stores queue rows"| node_database
node_handlers -->|"transitions status"| node_documents
node_handlers -->|"writes and reads pages"| node_pages
node_handlers -->|"normalizes PDFs"| node_normalizer
node_handlers -->|"extracts page text"| node_ocr
node_handlers -->|"chunks page text"| node_chunker
node_handlers -->|"encodes chunks"| node_embedder
node_handlers -->|"reads and writes blobs"| node_blob
node_handlers -->|"upserts vectors"| node_vectors
node_handlers -->|"enqueues next jobs"| node_queue
node_handlers -->|"records transitions"| node_audit
node_audit -->|"stages audit rows"| node_auditmodel
node_audit -.->|"writes with caller transaction"| node_database
node_vectors -.->|"upserts collections"| node_vectorstore

click node_api "https://github.com/devprad-ml/real_project/blob/main/src/app/main.py"
click node_health "https://github.com/devprad-ml/real_project/blob/main/src/app/api/health.py"
click node_config "https://github.com/devprad-ml/real_project/blob/main/src/app/config.py"
click node_dbsession "https://github.com/devprad-ml/real_project/blob/main/src/app/db.py"
click node_documents "https://github.com/devprad-ml/real_project/blob/main/src/app/models/document.py"
click node_pages "https://github.com/devprad-ml/real_project/blob/main/src/app/models/document_page.py"
click node_jobmodel "https://github.com/devprad-ml/real_project/blob/main/src/app/models/job.py"
click node_auditmodel "https://github.com/devprad-ml/real_project/blob/main/src/app/models/audit_log.py"
click node_clients "https://github.com/devprad-ml/real_project/blob/main/src/app/models/client.py"
click node_worker "https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/worker.py"
click node_queue "https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/queue.py"
click node_handlers "https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/handlers.py"
click node_audit "https://github.com/devprad-ml/real_project/blob/main/src/app/audit/log.py"
click node_normalizer "https://github.com/devprad-ml/real_project/blob/main/src/app/processing/normalize.py"
click node_ocr "https://github.com/devprad-ml/real_project/blob/main/src/app/processing/ocr.py"
click node_chunker "https://github.com/devprad-ml/real_project/blob/main/src/app/processing/chunk.py"
click node_embedder "https://github.com/devprad-ml/real_project/blob/main/src/app/processing/embed.py"
click node_blob "https://github.com/devprad-ml/real_project/blob/main/src/app/storage/blob.py"
click node_vectors "https://github.com/devprad-ml/real_project/blob/main/src/app/storage/vectors.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_health,node_config toneBlue
class node_normalizer,node_ocr,node_chunker,node_embedder toneAmber
class node_database,node_dbsession,node_documents,node_pages,node_jobmodel,node_auditmodel,node_clients,node_blob,node_vectors,node_vectorstore toneMint
class node_worker,node_queue,node_handlers,node_audit toneRose
```

## Components

### Runtime

- [API entrypoint](https://github.com/devprad-ml/real_project/blob/main/src/app/main.py): `src/app/main.py`
- [Health check](https://github.com/devprad-ml/real_project/blob/main/src/app/api/health.py): `src/app/api/health.py`
- [Settings](https://github.com/devprad-ml/real_project/blob/main/src/app/config.py): `src/app/config.py`

### Document processing

- [PDF normalization](https://github.com/devprad-ml/real_project/blob/main/src/app/processing/normalize.py): `src/app/processing/normalize.py`
- [Page OCR](https://github.com/devprad-ml/real_project/blob/main/src/app/processing/ocr.py): `src/app/processing/ocr.py`
- [Text chunking](https://github.com/devprad-ml/real_project/blob/main/src/app/processing/chunk.py): `src/app/processing/chunk.py`
- [Embedding](https://github.com/devprad-ml/real_project/blob/main/src/app/processing/embed.py): `src/app/processing/embed.py`

### Persistence

- **PostgreSQL**
- [DB sessions](https://github.com/devprad-ml/real_project/blob/main/src/app/db.py): `src/app/db.py`
- [Documents](https://github.com/devprad-ml/real_project/blob/main/src/app/models/document.py): `src/app/models/document.py`
- [OCR pages](https://github.com/devprad-ml/real_project/blob/main/src/app/models/document_page.py): `src/app/models/document_page.py`
- [Job records](https://github.com/devprad-ml/real_project/blob/main/src/app/models/job.py): `src/app/models/job.py`
- [Audit records](https://github.com/devprad-ml/real_project/blob/main/src/app/models/audit_log.py): `src/app/models/audit_log.py`
- [Tenant clients](https://github.com/devprad-ml/real_project/blob/main/src/app/models/client.py): `src/app/models/client.py`
- [Blob storage adapter](https://github.com/devprad-ml/real_project/blob/main/src/app/storage/blob.py): `src/app/storage/blob.py`
- [Vector storage adapter](https://github.com/devprad-ml/real_project/blob/main/src/app/storage/vectors.py): `src/app/storage/vectors.py`
- **Qdrant**

### Jobs and audit

- [Worker loop](https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/worker.py): `src/app/jobs/worker.py`
- [Job queue](https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/queue.py): `src/app/jobs/queue.py`
- [Job handlers](https://github.com/devprad-ml/real_project/blob/main/src/app/jobs/handlers.py): `src/app/jobs/handlers.py`
- [Audit writer](https://github.com/devprad-ml/real_project/blob/main/src/app/audit/log.py): `src/app/audit/log.py`

## Connections

- API entrypoint → Health check: mounts (evidence: `src/app/main.py`)
- API entrypoint → DB sessions: uses (evidence: `src/app/main.py`)
- Health check → DB sessions: checks (evidence: `src/app/api/health.py`)
- DB sessions → Settings: loads settings (evidence: `src/app/db.py`)
- DB sessions → PostgreSQL: opens sessions
- Worker loop → DB sessions: opens sessions (evidence: `src/app/jobs/worker.py`)
- Worker loop → Job queue: claims and retries (evidence: `src/app/jobs/worker.py`)
- Worker loop → Job handlers: dispatches jobs (evidence: `src/app/jobs/worker.py`)
- Job queue → Job records: creates and claims (evidence: `src/app/jobs/queue.py`)
- Job queue → Documents: updates status (evidence: `src/app/jobs/queue.py`)
- Job queue → Audit writer: records failures (evidence: `src/app/jobs/queue.py`)
- Job queue → Settings: reads retry settings (evidence: `src/app/jobs/queue.py`)
- Job queue → PostgreSQL: stores queue rows
- Job handlers → Documents: transitions status (evidence: `src/app/jobs/handlers.py`)
- Job handlers → OCR pages: writes and reads pages (evidence: `src/app/jobs/handlers.py`)
- Job handlers → PDF normalization: normalizes PDFs (evidence: `src/app/jobs/handlers.py`)
- Job handlers → Page OCR: extracts page text (evidence: `src/app/jobs/handlers.py`)
- Job handlers → Text chunking: chunks page text (evidence: `src/app/jobs/handlers.py`)
- Job handlers → Embedding: encodes chunks (evidence: `src/app/jobs/handlers.py`)
- Job handlers → Blob storage adapter: reads and writes blobs (evidence: `src/app/jobs/handlers.py`)
- Job handlers → Vector storage adapter: upserts vectors (evidence: `src/app/jobs/handlers.py`)
- Job handlers → Job queue: enqueues next jobs (evidence: `src/app/jobs/handlers.py`)
- Job handlers → Audit writer: records transitions (evidence: `src/app/jobs/handlers.py`)
- Audit writer → Audit records: stages audit rows (evidence: `src/app/audit/log.py`)
- Audit writer → PostgreSQL: writes with caller transaction
- Vector storage adapter → Qdrant: upserts collections
