# faisalq05/llm-rag-web-parser architecture

> Architecture diagram of the GitHub repository faisalq05/llm-rag-web-parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/faisalq05/llm-rag-web-parser
- Repository: https://github.com/faisalq05/llm-rag-web-parser

## Overview

1. This repository is a TypeScript/Express backend for a RAG pipeline: it crawls websites, extracts text, embeds chunks, stores them in Qdrant, and answers questions over that indexed content with an LLM.

2. The runtime entry path is `src/app/server.ts` → `src/app/app.ts` → `src/app/container.ts`. That split suggests a clean separation between process startup, Express app assembly, and dependency wiring.

3. Configuration is centralized under `src/config` (`env.ts`, `cors.ts`, `logger.ts`, `index.ts`). Environment variables drive ports and external service endpoints, while logging and CORS are treated as app-level concerns rather than per-route logic.

4. The HTTP edge is handled by shared middleware in `src/common/middlewares`: request validation, rate limiting, mTLS enforcement, 404 handling, and a global error middleware. This means requests are normalized and rejected at the boundary before they reach business logic.

5. Domain HTTP handling lives in `src/modules/url`, with a controller, routes, and service layer. The exposed API is centered on URL indexing and retrieval operations such as parse/index, chat, list indexed URLs, and delete.

6. The indexing path is implemented in `src/modules/ai/services/scraper.service.ts`. The crawler uses a BFS-style crawl over internal links, strips non-content DOM elements, and produces cleaned text chunks with overlap for downstream embedding.

7. Embedding and vector storage are split into dedicated AI services: `embedding.service.ts` for Gemini embeddings and `qdrant.service.ts` for vector persistence/search. This is the core ingestion boundary: text chunks become vectors, then are written to Qdrant for semantic retrieval.

8. The chat path is a retrieval-then-generation flow. A user query is embedded, Qdrant returns top matching chunks, and the LLM service (`openai.service.ts` in the tree, but described in the README as Groq/Llama) generates an answer from those retrieved sources and optional conversation history.

9. Persistent relational storage is present via `src/common/database/connect.ts`, with Prisma-specific error handling in `src/common/utils/prisma-error.ts`. That indicates PostgreSQL is used for app metadata such as indexed URL records and possibly user-related state, separate from Qdrant’s vector store.

10. Shared utilities and exception classes in `src/common` provide consistent API responses, async handler wrapping, and typed error mapping (`bad-request`, `unauthorized`, `forbidden`, `not-found`). This keeps controllers thin and pushes cross-cutting concerns into reusable primitives.

11. `src/seeder` contains seed scripts, which suggests the repository supports bootstrapping database state during development or deployment. That fits with the dockerized workflow described in the README.

12. Infrastructure-wise, the repo is designed to run both locally and in containers via Docker/Docker Compose. The external dependencies that materially shape the architecture are PostgreSQL, Qdrant, Google Gemini for embeddings, and Groq-hosted Llama for generation.

Main entry points: [`src/app/app.ts`](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/app/app.ts) (App), [`src/modules/url/services/url.service.ts`](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/url/services/url.service.ts) (URL Service), [`src/config/index.ts`](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/index.ts) (Config Index), [`src/common/database/connect.ts`](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/database/connect.ts) (Database)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_app["Runtime"]
  node_n_server(("Server<br/>entrypoint<br/>[server.ts]"))
  node_n_app["App<br/>[app.ts]"]
  node_n_container["Container<br/>wiring<br/>[container.ts]"]
end

subgraph group_grp_http["HTTP Edge"]
  node_n_validate["Validate<br/>middleware"]
  node_n_rate_limit["Rate Limit<br/>middleware"]
  node_n_mtls["mTLS<br/>middleware<br/>[mtls.middleware.ts]"]
  node_n_not_found["404 Handler<br/>middleware"]
  node_n_error_mw["Error MW<br/>middleware"]
end

subgraph group_grp_config["Config"]
  node_n_env["Env<br/>config<br/>[env.ts]"]
  node_n_cors["CORS<br/>config<br/>[cors.ts]"]
  node_n_logger["Logger<br/>config<br/>[logger.ts]"]
  node_n_config_index["Config Index<br/>config export<br/>[index.ts]"]
end

subgraph group_grp_domain["Domain"]
  node_n_url_controller["URL Controller<br/>http controller<br/>[url.controller.ts]"]
  node_n_url_routes["URL Routes<br/>[url.routes.ts]"]
  node_n_url_service["URL Service<br/>[url.service.ts]"]
end

subgraph group_grp_ai["AI Pipeline"]
  node_n_scraper["Scraper<br/>crawler service<br/>[scraper.service.ts]"]
  node_n_embed["Embeddings<br/>ai service"]
  node_n_qdrant[("Qdrant<br/>vector store<br/>[qdrant.service.ts]")]
  node_n_openai["LLM<br/>generation service<br/>[openai.service.ts]"]
end

subgraph group_grp_common["Shared Core"]
  node_n_db[("Database<br/>postgres connector<br/>[connect.ts]")]
  node_n_prisma_err["Prisma Errors<br/>error mapper<br/>[prisma-error.ts]"]
  node_n_response["Response Utils<br/>http utils<br/>[response.ts]"]
  node_n_exceptions["Exceptions<br/>error classes<br/>[app.exception.ts]"]
end

subgraph group_grp_seed["Seeding"]
  node_n_seeder["Seeder<br/>bootstrap script<br/>[seed.ts]"]
  node_n_user_seeder["User Seeder<br/>seed module<br/>[user.seeder.ts]"]
end

subgraph group_grp_ext["External"]
  node_n_postgres[("PostgreSQL<br/>external db")]
  node_n_gemini(("Gemini<br/>external embeddings"))
  node_n_groq(("Groq Llama<br/>external llm"))
end

node_n_server -->|"starts"| node_n_app
node_n_app -->|"loads"| node_n_container
node_n_container -->|"uses"| node_n_config_index
node_n_config_index -->|"includes"| node_n_env
node_n_config_index -->|"includes"| node_n_cors
node_n_config_index -->|"includes"| node_n_logger
node_n_app -->|"applies"| node_n_rate_limit
node_n_app -->|"applies"| node_n_mtls
node_n_app -->|"applies"| node_n_validate
node_n_app -->|"mounts"| node_n_url_routes
node_n_url_routes -->|"delegates"| node_n_url_controller
node_n_url_controller -->|"calls"| node_n_url_service
node_n_url_service -->|"crawls"| node_n_scraper
node_n_scraper -->|"embeds"| node_n_embed
node_n_embed -->|"requests"| node_n_gemini
node_n_url_service -->|"stores/searches"| node_n_qdrant
node_n_qdrant -->|"retrieves context for"| node_n_openai
node_n_openai -->|"requests"| node_n_groq
node_n_url_service -->|"reads/writes"| node_n_db
node_n_db -->|"connects to"| node_n_postgres
node_n_db -->|"maps errors with"| node_n_prisma_err
node_n_url_service -->|"formats"| node_n_response
node_n_url_service -->|"throws"| node_n_exceptions
node_n_app -->|"finalizes with"| node_n_not_found
node_n_app -->|"finalizes with"| node_n_error_mw
node_n_seeder -->|"runs"| node_n_user_seeder
node_n_seeder -->|"initializes"| node_n_db

click node_n_server "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/app/server.ts"
click node_n_app "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/app/app.ts"
click node_n_container "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/app/container.ts"
click node_n_env "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/env.ts"
click node_n_cors "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/cors.ts"
click node_n_logger "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/logger.ts"
click node_n_config_index "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/index.ts"
click node_n_validate "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/validate.middleware.ts"
click node_n_rate_limit "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/rate-limit.middleware.ts"
click node_n_mtls "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/mtls.middleware.ts"
click node_n_not_found "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/notFound.middleware.ts"
click node_n_error_mw "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/error.middleware.ts"
click node_n_url_controller "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/url/controller/url.controller.ts"
click node_n_url_routes "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/url/routes/url.routes.ts"
click node_n_url_service "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/url/services/url.service.ts"
click node_n_scraper "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/scraper.service.ts"
click node_n_embed "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/embedding.service.ts"
click node_n_qdrant "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/qdrant.service.ts"
click node_n_openai "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/openai.service.ts"
click node_n_db "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/database/connect.ts"
click node_n_prisma_err "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/utils/prisma-error.ts"
click node_n_response "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/utils/response.ts"
click node_n_exceptions "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/exceptions/app.exception.ts"
click node_n_seeder "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/seeder/seed.ts"
click node_n_user_seeder "https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/seeder/user.seeder.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_server,node_n_app,node_n_container,node_n_seeder,node_n_user_seeder toneBlue
class node_n_validate,node_n_rate_limit,node_n_mtls,node_n_not_found,node_n_error_mw,node_n_postgres,node_n_gemini,node_n_groq toneAmber
class node_n_env,node_n_cors,node_n_logger,node_n_config_index toneMint
class node_n_url_controller,node_n_url_routes,node_n_url_service toneRose
class node_n_scraper,node_n_embed,node_n_qdrant,node_n_openai toneIndigo
class node_n_db,node_n_prisma_err,node_n_response,node_n_exceptions toneTeal
```

## Components

### Runtime

Process startup and application assembly

- [Server](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/app/server.ts): entrypoint · Process bootstrap and listen lifecycle · `src/app/server.ts`
- [App](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/app/app.ts): express app · Express application assembly · `src/app/app.ts`
- [Container](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/app/container.ts): wiring · Dependency injection and service wiring · `src/app/container.ts`

### HTTP Edge

Middleware and request handling boundaries

- [Validate](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/validate.middleware.ts): middleware · Request validation boundary · `src/common/middlewares/validate.middleware.ts`
- [Rate Limit](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/rate-limit.middleware.ts): middleware · Request throttling boundary · `src/common/middlewares/rate-limit.middleware.ts`
- [mTLS](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/mtls.middleware.ts): middleware · Client certificate enforcement · `src/common/middlewares/mtls.middleware.ts`
- [404 Handler](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/notFound.middleware.ts): middleware · Fallback for unmatched routes · `src/common/middlewares/notFound.middleware.ts`
- [Error MW](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/middlewares/error.middleware.ts): middleware · Global error translation and response · `src/common/middlewares/error.middleware.ts`

### Config

Environment and app-wide settings

- [Env](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/env.ts): config · Environment variable loading and validation · `src/config/env.ts`
- [CORS](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/cors.ts): config · Cross-origin policy settings · `src/config/cors.ts`
- [Logger](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/logger.ts): config · Application logging configuration · `src/config/logger.ts`
- [Config Index](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/config/index.ts): config export · Central config aggregation · `src/config/index.ts`

### Domain

URL indexing and chat API

- [URL Controller](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/url/controller/url.controller.ts): http controller · API orchestration for indexing and chat · `src/modules/url/controller/url.controller.ts`
- [URL Routes](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/url/routes/url.routes.ts): routes · HTTP route definitions for URL workflows · `src/modules/url/routes/url.routes.ts`
- [URL Service](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/url/services/url.service.ts): service · Domain operations over indexed URLs · `src/modules/url/services/url.service.ts`

### AI Pipeline

Crawl, embed, store, and generate answers

- [Scraper](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/scraper.service.ts): crawler service · BFS crawl, clean pages, and chunk text · `src/modules/ai/services/scraper.service.ts`
- [Embeddings](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/embedding.service.ts): ai service · Creates vector embeddings for chunks · `src/modules/ai/services/embedding.service.ts`
- [Qdrant](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/qdrant.service.ts): vector store · Stores and searches embedded chunks · `src/modules/ai/services/qdrant.service.ts`
- [LLM](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/modules/ai/services/openai.service.ts): generation service · Generates answers from retrieved context · `src/modules/ai/services/openai.service.ts`

### Shared Core

Database, errors, and response utilities

- [Database](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/database/connect.ts): postgres connector · Relational persistence connection · `src/common/database/connect.ts`
- [Prisma Errors](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/utils/prisma-error.ts): error mapper · Maps Prisma failures to app errors · `src/common/utils/prisma-error.ts`
- [Response Utils](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/utils/response.ts): http utils · Standard API response helpers · `src/common/utils/response.ts`
- [Exceptions](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/common/exceptions/app.exception.ts): error classes · Typed application error hierarchy · `src/common/exceptions/app.exception.ts`

### Seeding

Bootstrapping data scripts

- [Seeder](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/seeder/seed.ts): bootstrap script · Database seeding entrypoint · `src/seeder/seed.ts`
- [User Seeder](https://github.com/faisalq05/llm-rag-web-parser/blob/main/src/seeder/user.seeder.ts): seed module · Seeds baseline user records · `src/seeder/user.seeder.ts`

### External

Infrastructure and model dependencies

- **PostgreSQL**: external db · Relational metadata storage
- **Gemini**: external embeddings · Hosted embedding model endpoint
- **Groq Llama**: external llm · Hosted chat generation endpoint

## Connections

- Server → App: starts · Bootstraps the Express app
- App → Container: loads · Resolves app dependencies and services
- Container → Config Index: uses · Pulls centralized configuration
- Config Index → Env: includes · Environment-driven runtime settings
- Config Index → CORS: includes · CORS policy settings
- Config Index → Logger: includes · Logging configuration
- App → Rate Limit: applies · Protects incoming requests
- App → mTLS: applies · Enforces client identity
- App → Validate: applies · Validates request payloads
- App → URL Routes: mounts · Registers URL API routes
- URL Routes → URL Controller: delegates · Routes forward to controller handlers
- URL Controller → URL Service: calls · Controller delegates domain logic
- URL Service → Scraper: crawls · Fetches and chunks web content
- Scraper → Embeddings: embeds · Prepares chunks for vectorization
- Embeddings → Gemini: requests · Uses hosted embedding model
- URL Service → Qdrant: stores/searches · Persists vectors and retrieves matches
- Qdrant → LLM: retrieves context for · Provides top chunks for answer generation
- LLM → Groq Llama: requests · Uses hosted generation model
- URL Service → Database: reads/writes · Persists URL metadata and app state
- Database → PostgreSQL: connects to · Relational backend
- Database → Prisma Errors: maps errors with · Converts database failures into app errors
- URL Service → Response Utils: formats · Builds consistent API responses
- URL Service → Exceptions: throws · Uses typed domain and HTTP errors
- App → 404 Handler: finalizes with · Handles unmatched routes
- App → Error MW: finalizes with · Handles uncaught application errors
- Seeder → User Seeder: runs · Seeder entrypoint orchestrates seed modules
- Seeder → Database: initializes · Seeds the relational database