faisalq05/llm-rag-web-parser

1. This repository is a TypeScript/Express backend for a RAG pipeline: it crawls websites, extracts text, embeds chunks, stores them in Qdrant, and answers questions over that indexed content with an LLM.

2. The runtime entry path is src/app/server.ts → src/app/app.ts → src/app/container.ts. That split suggests a clean separation between process startup, Express app assembly, and dependency wiring.

3. Configuration is centralized under src/config (env.ts, cors.ts, logger.ts, index.ts). Environment variables drive ports and external service endpoints, while logging and CORS are treated as app-level concerns rather than per-route logic.

4. The HTTP edge is handled by shared middleware in src/common/middlewares: request validation, rate limiting, mTLS enforcement, 404 handling, and a global error middleware. This means requests are normalized and rejected at the boundary before they reach business logic.

5. Domain HTTP handling lives in src/modules/url, with a controller, routes, and service layer. The exposed API is centered on URL indexing and retrieval operations such as parse/index, chat, list indexed URLs, and delete.

6. The indexing path is implemented in src/modules/ai/services/scraper.service.ts. The crawler uses a BFS-style crawl over internal links, strips non-content DOM elements, and produces cleaned text chunks with overlap for downstream embedding.

7. Embedding and vector storage are split into dedicated AI services: embedding.service.ts for Gemini embeddings and qdrant.service.ts for vector persistence/search. This is the core ingestion boundary: text chunks become vectors, then are written to Qdrant for semantic retrieval.

8. The chat path is a retrieval-then-generation flow. A user query is embedded, Qdrant returns top matching chunks, and the LLM service (openai.service.ts in the tree, but described in the README as Groq/Llama) generates an answer from those retrieved sources and optional conversation history.

9. Persistent relational storage is present via src/common/database/connect.ts, with Prisma-specific error handling in src/common/utils/prisma-error.ts. That indicates PostgreSQL is used for app metadata such as indexed URL records and possibly user-related state, separate from Qdrant’s vector store.

10. Shared utilities and exception classes in src/common provide consistent API responses, async handler wrapping, and typed error mapping (bad-request, unauthorized, forbidden, not-found). This keeps controllers thin and pushes cross-cutting concerns into reusable primitives.

11. src/seeder contains seed scripts, which suggests the repository supports bootstrapping database state during development or deployment. That fits with the dockerized workflow described in the README.

12. Infrastructure-wise, the repo is designed to run both locally and in containers via Docker/Docker Compose. The external dependencies that materially shape the architecture are PostgreSQL, Qdrant, Google Gemini for embeddings, and Groq-hosted Llama for generation.