How kingmaker1234/tn-agri-rag works
Uzhavan Assist is a Tamil Nadu farm-scheme question-answering app: users ask in Tamil, English, or Tanglish and receive answers grounded in government documents with citations and retrieval traces. Its main workflow runs from the browser’s Ask page through the API, query planning, hybrid retrieval and reranking, then answer generation. A separate ingestion workflow accepts files and URLs, extracts and chunks content, enriches it with document context, embeds it, and indexes it. The README also documents corpus discovery, comparison, library, source, and system views. Unsampled API wiring is shown conservatively.
Main entry points: main.py FastAPI routes · pipeline.py Ingest pipeline · hybrid.py Hybrid retrieval · App.tsx Navigation shell
Components 27
Web experience
- Navigation shell
frontend/src/App.tsx - Ask and citations
frontend/src/pages/Ask.tsx - Retrieval comparison
frontend/src/pages/Explore.tsx - Ingest controls
frontend/src/pages/Ingest.tsx - Document library
frontend/src/pages/Library.tsx - Source catalog
frontend/src/pages/Sources.tsx - System status
frontend/src/pages/System.tsx - Typed API client
frontend/src/api.ts
API orchestration
- FastAPI routes
backend/app/main.py - Ingestion jobs
backend/app/jobs.py
Corpus ingestion
- Source crawler
backend/app/ingest/crawler.py - File loaders
backend/app/ingest/loaders.py - Chunk splitter
backend/app/ingest/splitter.py - Document context
backend/app/ingest/doc_context.py - Ingest pipeline
backend/app/ingest/pipeline.py
Retrieval and answers
- Query planner
backend/app/retrieval/query_planner.py - Hybrid retrieval
backend/app/retrieval/hybrid.py - Passage reranker
backend/app/retrieval/reranker.py - Grounded answer
backend/app/llm/generator.py
Index and operations
- Embedding backends
backend/app/store/embeddings.py - FAISS and SQLite index
backend/app/store/vector_store.py - BM25 index
backend/app/store/bm25.py - Usage and telemetry
backend/app/telemetry.py - Token cost accounting
backend/app/llm/usage.py - Tamil text utilities
backend/app/text_utils.py
Other components
- Farmer or admin
- OpenAI-compatible model API
Connections 41
- Farmer or admin → Navigation shell: uses
- Navigation shell → Ask and citations: routes to
frontend/src/App.tsx - Navigation shell → Retrieval comparison: routes to
frontend/src/App.tsx - Navigation shell → Ingest controls: routes to
frontend/src/App.tsx - Navigation shell → Document library: routes to
frontend/src/App.tsx - Navigation shell → Source catalog: routes to
frontend/src/App.tsx - Navigation shell → System status: routes to
frontend/src/App.tsx - Ask and citations → Typed API client: requests
frontend/src/pages/Ask.tsx - Retrieval comparison → Typed API client: requests
frontend/src/pages/Explore.tsx - Ingest controls → Typed API client: requests
frontend/src/pages/Ingest.tsx - Document library → Typed API client: requests
frontend/src/pages/Library.tsx - Source catalog → Typed API client: requests
frontend/src/pages/Sources.tsx - System status → Typed API client: requests
frontend/src/pages/System.tsx - Typed API client → FastAPI routes: HTTP requests
frontend/src/api.ts - FastAPI routes → Hybrid retrieval: searches
backend/app/main.py - FastAPI routes → Ingest pipeline: starts ingestion
backend/app/main.py - FastAPI routes → Ingestion jobs: manages jobs
backend/app/main.py - FastAPI routes → Source crawler: discovers sources
backend/app/main.py - FastAPI routes → FAISS and SQLite index: serves documents
backend/app/main.py - FastAPI routes → Usage and telemetry: reports status
backend/app/main.py - Ingest pipeline → Source crawler: ingests discovered
backend/app/ingest/pipeline.py - Ingest pipeline → File loaders: loads files
backend/app/ingest/pipeline.py - Ingest pipeline → Chunk splitter: splits sections
backend/app/ingest/pipeline.py - Ingest pipeline → Document context: adds context
backend/app/ingest/pipeline.py - Ingest pipeline → Embedding backends: embeds chunks
backend/app/ingest/pipeline.py - Ingest pipeline → FAISS and SQLite index: indexes chunks
backend/app/ingest/pipeline.py - File loaders → Tamil text utilities: cleans Tamil text
backend/app/ingest/loaders.py - Document context → OpenAI-compatible model API: summarizes documents
backend/app/ingest/doc_context.py - Hybrid retrieval → Query planner: plans variants
backend/app/retrieval/hybrid.py - Hybrid retrieval → Embedding backends: embeds queries
backend/app/retrieval/hybrid.py - Hybrid retrieval → FAISS and SQLite index: dense search
backend/app/retrieval/hybrid.py - Hybrid retrieval → BM25 index: keyword search
- Hybrid retrieval → Passage reranker: reranks candidates
backend/app/retrieval/hybrid.py - Hybrid retrieval → Grounded answer: supplies retrieved context
- Query planner → OpenAI-compatible model API: rewrites queries
backend/app/retrieval/query_planner.py - Passage reranker → OpenAI-compatible model API: scores passages
- Grounded answer → OpenAI-compatible model API: generates answers
backend/app/llm/generator.py - Embedding backends → OpenAI-compatible model API: creates vectors
- Ingest pipeline → Token cost accounting: records cost
backend/app/ingest/pipeline.py - Hybrid retrieval → Token cost accounting: records cost
backend/app/retrieval/hybrid.py - FastAPI routes → Token cost accounting: reports usage
backend/app/main.py