How thakshil/rag-intelligence works
RAG Document Intelligence turns uploaded PDFs into document-scoped, searchable chunks and answers questions with grounded citations. Users enter through the Streamlit interface or REST API; ingestion extracts text with conditional OCR, chunks and embeds it, then indexes it. Queries are embedded, retrieved from the active document, and passed to grounded generation with provider fallback. The graph preserves the documented telemetry and status capabilities. Some UI and ingestion-stage links rely on the README; unsampled implementation details are not asserted.
Main entry points: pipeline.py RAG pipeline · main.py REST API · retrieve.py Dense retrieval · embed.py Embedding model
Components 18
User experience
- Streamlit UI
ui/app.py - REST API
api/main.py - Document status
api/status_store.py
Document ingestion
- PDF extraction
src/rag/extract.py - Tesseract OCR
- Text chunking
src/rag/chunk.py - Token counting
src/rag/tokenizer.py - Embedding model
src/rag/embed.py
Retrieval and answers
- RAG pipeline
src/rag/pipeline.py - Dense retrieval
src/rag/retrieve.py - Grounded generation
src/rag/generate.py - LLM providers
src/rag/llm.py
State and configuration
- Embedding cache
src/rag/embed.py - Vector store
src/rag/store.py - Qdrant
- Ingestion manifest
src/rag/pipeline.py - Settings
src/rag/config.py
Other components
- User
Connections 21
- User → Streamlit UI: uses
README.md - User → REST API: uploads or queries
README.md - Streamlit UI → RAG pipeline: uploads documents
README.md - REST API → RAG pipeline: ingests and queries
api/main.py - REST API → Document status: updates status
api/main.py - RAG pipeline → PDF extraction: extracts text
README.md - PDF extraction → Tesseract OCR: uses when needed
README.md - RAG pipeline → Text chunking: chunks text
README.md - Text chunking → Token counting: counts tokens
src/rag/chunk.py - RAG pipeline → Embedding model: encodes chunks
README.md - Embedding model → Embedding cache: reads and writes
src/rag/embed.py - RAG pipeline → Vector store: indexes chunks
README.md - Vector store → Qdrant: reads and writes
src/rag/store.py - RAG pipeline → Dense retrieval: retrieves passages
src/rag/pipeline.py - Dense retrieval → Embedding model: embeds query
src/rag/retrieve.py - Dense retrieval → Vector store: searches vectors
src/rag/retrieve.py - RAG pipeline → Grounded generation: generates answer
src/rag/pipeline.py - Grounded generation → LLM providers: invokes provider
src/rag/generate.py - RAG pipeline → Ingestion manifest: reads document stats
src/rag/pipeline.py - REST API → Settings: loads settings
api/main.py - RAG pipeline → Settings: uses settings
src/rag/pipeline.py