How devprad-ml/real_project works
This repository implements a document-processing backend with a PostgreSQL-backed job queue and asynchronous worker. The visible processing workflow normalizes received documents, OCRs pages, then chunks and embeds their text into tenant-specific vector collections; handlers record status changes in an append-only audit log. The sampled API exposes health checking, but does not show document submission or retrieval endpoints. The README is empty, so broader capabilities are limited to what the source index and excerpts support.
Main entry points: handlers.py Job handlers · queue.py Job queue · db.py DB sessions · log.py Audit writer
Components 21
Runtime
- API entrypoint
src/app/main.py - Health check
src/app/api/health.py - Settings
src/app/config.py
Document processing
- PDF normalization
src/app/processing/normalize.py - Page OCR
src/app/processing/ocr.py - Text chunking
src/app/processing/chunk.py - Embedding
src/app/processing/embed.py
Persistence
- PostgreSQL
- DB sessions
src/app/db.py - Documents
src/app/models/document.py - OCR pages
src/app/models/document_page.py - Job records
src/app/models/job.py - Audit records
src/app/models/audit_log.py - Tenant clients
src/app/models/client.py - Blob storage adapter
src/app/storage/blob.py - Vector storage adapter
src/app/storage/vectors.py - Qdrant
Jobs and audit
- Worker loop
src/app/jobs/worker.py - Job queue
src/app/jobs/queue.py - Job handlers
src/app/jobs/handlers.py - Audit writer
src/app/audit/log.py
Connections 26
- API entrypoint → Health check: mounts
src/app/main.py - API entrypoint → DB sessions: uses
src/app/main.py - Health check → DB sessions: checks
src/app/api/health.py - DB sessions → Settings: loads settings
src/app/db.py - DB sessions → PostgreSQL: opens sessions
- Worker loop → DB sessions: opens sessions
src/app/jobs/worker.py - Worker loop → Job queue: claims and retries
src/app/jobs/worker.py - Worker loop → Job handlers: dispatches jobs
src/app/jobs/worker.py - Job queue → Job records: creates and claims
src/app/jobs/queue.py - Job queue → Documents: updates status
src/app/jobs/queue.py - Job queue → Audit writer: records failures
src/app/jobs/queue.py - Job queue → Settings: reads retry settings
src/app/jobs/queue.py - Job queue → PostgreSQL: stores queue rows
- Job handlers → Documents: transitions status
src/app/jobs/handlers.py - Job handlers → OCR pages: writes and reads pages
src/app/jobs/handlers.py - Job handlers → PDF normalization: normalizes PDFs
src/app/jobs/handlers.py - Job handlers → Page OCR: extracts page text
src/app/jobs/handlers.py - Job handlers → Text chunking: chunks page text
src/app/jobs/handlers.py - Job handlers → Embedding: encodes chunks
src/app/jobs/handlers.py - Job handlers → Blob storage adapter: reads and writes blobs
src/app/jobs/handlers.py - Job handlers → Vector storage adapter: upserts vectors
src/app/jobs/handlers.py - Job handlers → Job queue: enqueues next jobs
src/app/jobs/handlers.py - Job handlers → Audit writer: records transitions
src/app/jobs/handlers.py - Audit writer → Audit records: stages audit rows
src/app/audit/log.py - Audit writer → PostgreSQL: writes with caller transaction
- Vector storage adapter → Qdrant: upserts collections