# piyush271005/ai-document-search-engine architecture

> Architecture diagram of the GitHub repository piyush271005/ai-document-search-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/piyush271005/ai-document-search-engine
- Repository: https://github.com/piyush271005/ai-document-search-engine

## Overview

1. **What this repo is**
   - This is a two-part system: a **Python backend** that crawls and indexes technical docs for semantic search, and a **React/Vite frontend** that lets users query the index and view answers.
   - The architecture is centered on a **RAG-style search pipeline**: crawl → parse → chunk → embed → index → retrieve → rerank → LLM answer.

2. **Backend as the orchestration layer**
   - `backend/main.py` is the application entrypoint; `backend/api/router.py` exposes the HTTP endpoints.
   - `backend/config.py` likely holds environment-driven settings for crawl limits, model selection, and storage paths.
   - `backend/models/schemas.py` defines request/response payloads, which acts as the boundary between the UI/API and the retrieval pipeline.

3. **Ingestion pipeline**
   - `backend/crawler/crawler.py` performs the **BFS crawl** over a documentation site, discovering pages by depth/page limits.
   - `backend/parser/parser.py` cleans HTML into structured text, stripping navigation and boilerplate.
   - `backend/chunking/chunker.py` turns parsed content into paragraph-aware chunks that preserve source context.
   - `backend/embeddings/embedding.py` converts chunks into dense vectors using a local `sentence-transformers` model.

4. **Indexing and persistence**
   - `backend/retrieval/db.py` connects the pipeline to **ChromaDB**, which is the persistent vector store for embeddings, chunk text, and metadata.
   - The checked-in `chroma_db/` directory shows this is a **local embedded index**, not an external managed database.
   - In parallel, the chunk corpus is also kept for **BM25 keyword search** in memory, so the system has two retrieval indexes over the same content.

5. **Query-time retrieval flow**
   - User questions enter through the backend API and are searched along two paths:
     - **Vector retrieval** against ChromaDB for semantic matches.
     - **BM25 retrieval** for exact keyword matches.
   - `backend/retrieval/reranker.py` combines both result sets using **Reciprocal Rank Fusion (RRF)** to produce one hybrid ranking.

6. **Answer generation layer**
   - `backend/llm/llm.py` is an abstraction over multiple LLM providers: **Gemini, OpenAI, Ollama, and a local Mock provider**.
   - The top retrieved chunks are formatted into a RAG prompt, and the chosen provider synthesizes the final answer with citations or source grounding.

7. **System boundaries inside the backend**
   - The code is split into clear stages: crawling, parsing, chunking, embedding, retrieval, reranking, and generation.
   - That separation matters architecturally because each stage can be swapped or scaled independently: for example, changing the LLM provider does not affect crawling or indexing, and BM25 remains separate from vector search.

8. **Frontend role**
   - `frontend/src/App.jsx` and `frontend/src/main.jsx` show a standard React client that consumes the backend API.
   - The frontend is not doing retrieval itself; it is a presentation layer for query input, result display, and likely pipeline/status visualization.
   - `frontend/Dockerfile` and Vite config indicate the UI is built and served separately from the API.

9. **Deployment topology**
   - `docker-compose.yml` and the two Dockerfiles indicate the project is designed to run as **containerized services**.
   - The README says compose orchestrates **backend, frontend, and Redis**; Redis is likely used as a supporting queue/cache component even though its code is not visible in the tree.
   - The backend and frontend are independently runnable in local development, but also packaged for multi-container deployment.

10. **Operational/runtime characteristics**
   - The backend is Python-based and depends on AI/search tooling such as **BeautifulSoup, ChromaDB, sentence-transformers, and rank-bm25**.
   - This is a mostly local-first architecture: embeddings are computed locally, ChromaDB persists locally, and only the LLM layer may reach external providers depending on configuration.
   - `backend/test_backend.py` suggests there is at least some automated validation around the API or pipeline behavior.

11. **How to draw the architecture**
   - A good diagram should show two major lanes:
     - **Ingestion lane:** documentation website → crawler → parser → chunker → embeddings → ChromaDB + BM25 corpus.
     - **Query lane:** user → frontend → backend API → vector search + BM25 search → RRF reranker → LLM provider → answer back to frontend.
   - The central backend is the control plane connecting all stages, while ChromaDB and the LLM provider are the main external dependencies in the runtime path.

Main entry points: [`backend/api/router.py`](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/api/router.py) (API router), [`backend/retrieval/reranker.py`](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/retrieval/reranker.py) (RRF reranker), [`backend/main.py`](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/main.py) (Main), [`docker-compose.yml`](https://github.com/piyush271005/ai-document-search-engine/blob/main/docker-compose.yml) (Compose)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_backend["Backend"]
  node_backend_main["Main<br/>app entry<br/>[main.py]"]
  node_api_router["API router<br/>http api<br/>[router.py]"]
  node_config["Config<br/>runtime settings<br/>[config.py]"]
  node_schemas["Schemas<br/>api models<br/>[schemas.py]"]
  node_crawler["Crawler<br/>ingestion<br/>[crawler.py]"]
  node_parser["Parser<br/>html cleaning<br/>[parser.py]"]
  node_chunker["Chunker<br/>text splitting<br/>[chunker.py]"]
  node_embedding["Embedding<br/>vectorization<br/>[embedding.py]"]
  node_vector_db[("Vector DB<br/>chroma adapter<br/>[db.py]")]
  node_bm25["BM25<br/>keyword index<br/>[db.py]"]
  node_reranker["RRF reranker<br/>fusion ranking<br/>[reranker.py]"]
  node_llm["LLM layer<br/>provider abstraction<br/>[llm.py]"]
end

subgraph group_frontend["Frontend"]
  node_frontend_app["App<br/>react ui<br/>[App.jsx]"]
  node_frontend_main["UI entry<br/>react bootstrap<br/>[main.jsx]"]
end

subgraph group_storage["Storage"]
  node_chroma_store[("Chroma store<br/>local index")]
end

subgraph group_deploy["Deployment"]
  node_compose["Compose<br/>service orchestration<br/>[docker-compose.yml]"]
  node_redis[("Redis<br/>support service")]
end

node_docs_source(("Docs site<br/>external source"))

node_docs_source -->|"crawl"| node_crawler
node_crawler -->|"html"| node_parser
node_parser -->|"text"| node_chunker
node_chunker -->|"chunks"| node_embedding
node_embedding -->|"store"| node_vector_db
node_chunker -->|"corpus"| node_bm25
node_api_router -->|"query"| node_reranker
node_reranker -->|"vector search"| node_vector_db
node_reranker -->|"keyword search"| node_bm25
node_reranker -->|"context"| node_llm
node_llm -->|"answer"| node_api_router
node_api_router -->|"payloads"| node_schemas
node_backend_main -->|"mount"| node_api_router
node_backend_main -->|"load"| node_config
node_backend_main -->|"ingest"| node_crawler
node_frontend_main -->|"render"| node_frontend_app
node_frontend_app -->|"call api"| node_api_router
node_compose -->|"service"| node_backend_main
node_compose -->|"service"| node_frontend_main
node_compose -->|"service"| node_redis
node_vector_db -->|"persist"| node_chroma_store

click node_backend_main "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/main.py"
click node_api_router "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/api/router.py"
click node_config "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/config.py"
click node_schemas "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/models/schemas.py"
click node_crawler "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/crawler/crawler.py"
click node_parser "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/parser/parser.py"
click node_chunker "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/chunking/chunker.py"
click node_embedding "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/embeddings/embedding.py"
click node_vector_db "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/retrieval/db.py"
click node_bm25 "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/retrieval/db.py"
click node_reranker "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/retrieval/reranker.py"
click node_llm "https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/llm/llm.py"
click node_frontend_app "https://github.com/piyush271005/ai-document-search-engine/blob/main/frontend/src/App.jsx"
click node_frontend_main "https://github.com/piyush271005/ai-document-search-engine/blob/main/frontend/src/main.jsx"
click node_chroma_store "https://github.com/piyush271005/ai-document-search-engine/tree/main/chroma_db"
click node_compose "https://github.com/piyush271005/ai-document-search-engine/blob/main/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_backend_main,node_api_router,node_config,node_schemas,node_crawler,node_parser,node_chunker,node_embedding,node_vector_db,node_bm25,node_reranker,node_llm toneBlue
class node_frontend_app,node_frontend_main toneAmber
class node_chroma_store toneMint
class node_compose,node_redis toneRose
class node_docs_source toneNeutral
```

## Components

### Backend

Python control plane for ingestion, retrieval, and answer generation.

- [Main](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/main.py): app entry · Backend application startup and wiring. · `backend/main.py`
- [API router](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/api/router.py): http api · HTTP endpoints that expose search and ingestion operations. · `backend/api/router.py`
- [Config](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/config.py): runtime settings · Environment-driven limits, model selection, and storage paths. · `backend/config.py`
- [Schemas](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/models/schemas.py): api models · Request and response shapes for the backend boundary. · `backend/models/schemas.py`
- [Crawler](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/crawler/crawler.py): ingestion · BFS crawl over documentation pages with depth and page limits. · `backend/crawler/crawler.py`
- [Parser](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/parser/parser.py): html cleaning · Cleans HTML into structured text and removes boilerplate. · `backend/parser/parser.py`
- [Chunker](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/chunking/chunker.py): text splitting · Breaks parsed content into paragraph-aware source chunks. · `backend/chunking/chunker.py`
- [Embedding](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/embeddings/embedding.py): vectorization · Converts chunks into dense vectors with local sentence-transformer models. · `backend/embeddings/embedding.py`
- [Vector DB](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/retrieval/db.py): chroma adapter · Persists embeddings, chunk text, and metadata in ChromaDB. · `backend/retrieval/db.py`
- [BM25](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/retrieval/db.py): keyword index · In-memory keyword corpus for exact-match retrieval. · `backend/retrieval/db.py`
- [RRF reranker](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/retrieval/reranker.py): fusion ranking · Merges vector and keyword results using reciprocal rank fusion. · `backend/retrieval/reranker.py`
- [LLM layer](https://github.com/piyush271005/ai-document-search-engine/blob/main/backend/llm/llm.py): provider abstraction · Chooses among Gemini, OpenAI, Ollama, or mock generation providers. · `backend/llm/llm.py`

### Frontend

React client for query entry and result display.

- [App](https://github.com/piyush271005/ai-document-search-engine/blob/main/frontend/src/App.jsx): react ui · Client UI for query input and answer presentation. · `frontend/src/App.jsx`
- [UI entry](https://github.com/piyush271005/ai-document-search-engine/blob/main/frontend/src/main.jsx): react bootstrap · Frontend bootstrap and rendering entrypoint. · `frontend/src/main.jsx`

### Storage

Local persistent vector store and corpus state.

- [Chroma store](https://github.com/piyush271005/ai-document-search-engine/tree/main/chroma_db): local index · On-disk embedded ChromaDB persistence for the indexed corpus. · `chroma_db`

### Deployment

Container composition and service wiring.

- [Compose](https://github.com/piyush271005/ai-document-search-engine/blob/main/docker-compose.yml): service orchestration · Container topology for backend, frontend, and Redis. · `docker-compose.yml`
- **Redis**: support service · Auxiliary runtime service used for queueing or caching.

### Other components

- **Docs site**: external source · Technical documentation site being crawled and indexed.

## Connections

- Docs site → Crawler: crawl · Fetch documentation pages for indexing.
- Crawler → Parser: html · Pass crawled pages for cleanup and structure extraction.
- Parser → Chunker: text · Feed cleaned content into chunking.
- Chunker → Embedding: chunks · Generate vectors for each chunk.
- Embedding → Vector DB: store · Persist embeddings and metadata in ChromaDB.
- Chunker → BM25: corpus · Populate the keyword search corpus.
- API router → RRF reranker: query · Route search requests into hybrid retrieval.
- RRF reranker → Vector DB: vector search · Retrieve semantically similar chunks.
- RRF reranker → BM25: keyword search · Retrieve exact-match candidates.
- RRF reranker → LLM layer: context · Send top-ranked chunks into answer generation.
- LLM layer → API router: answer · Return grounded response payloads to the API.
- API router → Schemas: payloads · Validate and serialize API messages.
- Main → API router: mount · Start and wire HTTP routes.
- Main → Config: load · Read runtime configuration at startup.
- Main → Crawler: ingest · Launch ingestion pipeline orchestration.
- UI entry → App: render · Bootstrap the React client.
- App → API router: call api · Send user queries and receive answers.
- Compose → Main: service · Run the backend container.
- Compose → UI entry: service · Run the frontend container.
- Compose → Redis: service · Provision supporting Redis service.
- Vector DB → Chroma store: persist · Store index state on disk.