How piyush271005/ai-document-search-engine works
1. What this repo is
- This is a two-part system: a Python backend that crawls and indexes technical docs for semantic search, and a React/Vite frontend that lets users query the index and view answers.
- The architecture is centered on a RAG-style search pipeline: crawl → parse → chunk → embed → index → retrieve → rerank → LLM answer.
2. Backend as the orchestration layer
- backend/main.py is the application entrypoint; backend/api/router.py exposes the HTTP endpoints.
- backend/config.py likely holds environment-driven settings for crawl limits, model selection, and storage paths.
- backend/models/schemas.py defines request/response payloads, which acts as the boundary between the UI/API and the retrieval pipeline.
3. Ingestion pipeline
- backend/crawler/crawler.py performs the BFS crawl over a documentation site, discovering pages by depth/page limits.
- backend/parser/parser.py cleans HTML into structured text, stripping navigation and boilerplate.
- backend/chunking/chunker.py turns parsed content into paragraph-aware chunks that preserve source context.
- backend/embeddings/embedding.py converts chunks into dense vectors using a local sentence-transformers model.
4. Indexing and persistence
- backend/retrieval/db.py connects the pipeline to ChromaDB, which is the persistent vector store for embeddings, chunk text, and metadata.
- The checked-in chroma_db/ directory shows this is a local embedded index, not an external managed database.
- In parallel, the chunk corpus is also kept for BM25 keyword search in memory, so the system has two retrieval indexes over the same content.
5. Query-time retrieval flow
- User questions enter through the backend API and are searched along two paths:
- Vector retrieval against ChromaDB for semantic matches.
- BM25 retrieval for exact keyword matches.
- backend/retrieval/reranker.py combines both result sets using Reciprocal Rank Fusion (RRF) to produce one hybrid ranking.
6. Answer generation layer
- backend/llm/llm.py is an abstraction over multiple LLM providers: Gemini, OpenAI, Ollama, and a local Mock provider.
- The top retrieved chunks are formatted into a RAG prompt, and the chosen provider synthesizes the final answer with citations or source grounding.
7. System boundaries inside the backend
- The code is split into clear stages: crawling, parsing, chunking, embedding, retrieval, reranking, and generation.
- That separation matters architecturally because each stage can be swapped or scaled independently: for example, changing the LLM provider does not affect crawling or indexing, and BM25 remains separate from vector search.
8. Frontend role
- frontend/src/App.jsx and frontend/src/main.jsx show a standard React client that consumes the backend API.
- The frontend is not doing retrieval itself; it is a presentation layer for query input, result display, and likely pipeline/status visualization.
- frontend/Dockerfile and Vite config indicate the UI is built and served separately from the API.
9. Deployment topology
- docker-compose.yml and the two Dockerfiles indicate the project is designed to run as containerized services.
- The README says compose orchestrates backend, frontend, and Redis; Redis is likely used as a supporting queue/cache component even though its code is not visible in the tree.
- The backend and frontend are independently runnable in local development, but also packaged for multi-container deployment.
10. Operational/runtime characteristics
- The backend is Python-based and depends on AI/search tooling such as BeautifulSoup, ChromaDB, sentence-transformers, and rank-bm25.
- This is a mostly local-first architecture: embeddings are computed locally, ChromaDB persists locally, and only the LLM layer may reach external providers depending on configuration.
- backend/test_backend.py suggests there is at least some automated validation around the API or pipeline behavior.
11. How to draw the architecture
- A good diagram should show two major lanes:
- Ingestion lane: documentation website → crawler → parser → chunker → embeddings → ChromaDB + BM25 corpus.
- Query lane: user → frontend → backend API → vector search + BM25 search → RRF reranker → LLM provider → answer back to frontend.
- The central backend is the control plane connecting all stages, while ChromaDB and the LLM provider are the main external dependencies in the runtime path.
Main entry points: router.py API router · reranker.py RRF reranker · main.py Main · docker-compose.yml Compose
Components 18
Backend
Python control plane for ingestion, retrieval, and answer generation.
- Main · Backend application startup and wiring.
backend/main.py - API router · HTTP endpoints that expose search and ingestion operations.
backend/api/router.py - Config · Environment-driven limits, model selection, and storage paths.
backend/config.py - Schemas · Request and response shapes for the backend boundary.
backend/models/schemas.py - Crawler · BFS crawl over documentation pages with depth and page limits.
backend/crawler/crawler.py - Parser · Cleans HTML into structured text and removes boilerplate.
backend/parser/parser.py - Chunker · Breaks parsed content into paragraph-aware source chunks.
backend/chunking/chunker.py - Embedding · Converts chunks into dense vectors with local sentence-transformer models.
backend/embeddings/embedding.py - Vector DB · Persists embeddings, chunk text, and metadata in ChromaDB.
backend/retrieval/db.py - BM25 · In-memory keyword corpus for exact-match retrieval.
backend/retrieval/db.py - RRF reranker · Merges vector and keyword results using reciprocal rank fusion.
backend/retrieval/reranker.py - LLM layer · Chooses among Gemini, OpenAI, Ollama, or mock generation providers.
backend/llm/llm.py
Frontend
React client for query entry and result display.
Storage
Local persistent vector store and corpus state.
- Chroma store · On-disk embedded ChromaDB persistence for the indexed corpus.
chroma_db
Deployment
Container composition and service wiring.
- Compose · Container topology for backend, frontend, and Redis.
docker-compose.yml - Redis · Auxiliary runtime service used for queueing or caching.
Other components
- Docs site · Technical documentation site being crawled and indexed.
Connections 21
- Docs site → Crawler: crawl · Fetch documentation pages for indexing.
- Crawler → Parser: html · Pass crawled pages for cleanup and structure extraction.
- Parser → Chunker: text · Feed cleaned content into chunking.
- Chunker → Embedding: chunks · Generate vectors for each chunk.
- Embedding → Vector DB: store · Persist embeddings and metadata in ChromaDB.
- Chunker → BM25: corpus · Populate the keyword search corpus.
- API router → RRF reranker: query · Route search requests into hybrid retrieval.
- RRF reranker → Vector DB: vector search · Retrieve semantically similar chunks.
- RRF reranker → BM25: keyword search · Retrieve exact-match candidates.
- RRF reranker → LLM layer: context · Send top-ranked chunks into answer generation.
- LLM layer → API router: answer · Return grounded response payloads to the API.
- API router → Schemas: payloads · Validate and serialize API messages.
- Main → API router: mount · Start and wire HTTP routes.
- Main → Config: load · Read runtime configuration at startup.
- Main → Crawler: ingest · Launch ingestion pipeline orchestration.
- UI entry → App: render · Bootstrap the React client.
- App → API router: call api · Send user queries and receive answers.
- Compose → Main: service · Run the backend container.
- Compose → UI entry: service · Run the frontend container.
- Compose → Redis: service · Provision supporting Redis service.
- Vector DB → Chroma store: persist · Store index state on disk.