# alooha-hub/hiast_rag architecture

> Architecture diagram of the GitHub repository alooha-hub/hiast_rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alooha-hub/hiast_rag
- Repository: https://github.com/alooha-hub/hiast_rag

## Overview

1. **Repo shape**
   - This monorepo contains three independently built subsystems:
     - `Backend_AI/`: a Python FastAPI service for conversational RAG.
     - `Backend_Web/`: a .NET 8 backend using Clean Architecture.
     - `hiast-frontend/`: a Next.js app for public site + admin UI.
   - The main architectural boundaries are: AI search/chat, business/content API, and browser-facing UI.

2. **AI backend (`Backend_AI`)**
   - This is a separate AI microservice, not part of the .NET backend.
   - It exposes conversation and data-management endpoints and is centered around retrieval-augmented generation.
   - Core subsystems include scraping (`url_scraper.py`), embedding/vector storage (`embedder.py`), conversational orchestration (`rag_chain.py`), token/history control (`token_manager.py`), and request/response models (`models.py`).

3. **AI data flow**
   - Content is scraped from URLs, converted into embeddings, and stored in a vector database.
   - User questions go through a RAG pipeline that retrieves relevant chunks, applies conversation memory, and calls a Google generative model.
   - The repo also includes multiple RAG strategies in `rag_abstractions.py` / `rag_chain.py`, so the answer path is pluggable rather than fixed.

4. **AI persistence and external dependencies**
   - The AI stack depends on ChromaDB-style vector retrieval and Google GenAI via API key.
   - It also uses HTML/document parsing tooling for scraping and extraction.
   - Test assets such as `squad_2000_samples.json` and `results.json` show that retrieval quality and strategy comparisons are part of the design.

5. **Web backend architecture (`Backend_Web`)**
   - This backend is clearly layered:
     - **Domain**: entities, enums, and shared base types.
     - **Application**: DTOs, CQRS commands/queries, mapper profiles, service interfaces, serializers, response models.
     - **Infrastructure**: EF Core DbContext, migrations, repository/unit-of-work implementations, domain services, seeding.
     - **Presentation**: ASP.NET Core controllers, middleware, auth, and API surface.
   - The architecture tests explicitly enforce that inner layers do not depend on outer layers.

6. **Web backend request path**
   - Requests enter `Presentation`, where controllers are split into `Areas/Admin` and `Areas/User`.
   - Controllers do not appear to access the database directly; they call services defined in `Application/IServices`.
   - Services are implemented in `Infrastructure/Services` and use repository/unit-of-work abstractions plus EF Core for persistence.

7. **Web backend API partitioning**
   - There is a strong read/write boundary:
     - `User` area: public, mostly read-only endpoints for browsing/filtering content.
     - `Admin` area: authenticated CRUD endpoints for managing content.
   - `AuthenticationController` and base controller types centralize auth context and response formatting.
   - Responses are standardized through `ApiResponse` and a custom JSON serializer/action result path.

8. **Web backend domain model**
   - The domain is content-heavy and multilingual.
   - Many entities have translation companions, such as `Program`/`ProgramTranslation`, `Page`/`PageTranslation`, `Book`/`BookTranslation`, etc.
   - The model also includes admissions, courses, media, FAQs, testimonials, slides, locations, languages, and training-related content.
   - This indicates a CMS-like backend with localized content and related media management.

9. **Web backend infrastructure**
   - Persistence is backed by Entity Framework Core with SQL Server migrations already checked in.
   - `DataSeeder` suggests the system can bootstrap initial content/reference data.
   - `wwwroot/Images` indicates the backend may also serve or store uploaded static assets.
   - JWT auth and ASP.NET Identity are part of the backend security boundary.

10. **Frontend architecture (`hiast-frontend`)**
    - The frontend is a Next.js app using the App Router structure.
    - It contains:
      - public pages for site content,
      - admin pages for CRUD workflows,
      - shared UI components,
      - API client modules under `lib/`,
      - localization files under `locales/`.
    - The frontend is not just presentational; it acts as the main consumer of the web backend APIs.

11. **Frontend data flow**
    - Public pages fetch content through `lib/public*.js` and entity-specific API modules.
    - Admin pages use corresponding admin API helpers and forms/tables for CRUD interaction.
    - Middleware and auth utilities suggest route protection for admin sections.
    - The presence of `AIChatbox.jsx` and `ConnectionTest.jsx` suggests the UI also integrates with the AI service for conversational features and service connectivity checks.

12. **Cross-cutting concerns**
    - Localization is a first-class concern across backend and frontend:
      - backend entities have translation tables,
      - frontend has `en.json` and `ar.json`,
      - language switchers exist in both public and admin UI.
    - Authentication is centralized in the .NET backend and consumed by the frontend.
    - Media/file handling is also a shared concern: backend services and DTOs model uploads, while frontend has dedicated media forms and viewers.

13. **Testing and quality gates**
    - `Backend_Web/ArchitectureTests` validates Clean Architecture dependency rules.
    - `Backend_AI/test/` includes RAG-specific tests and dataset tooling.
    - This repo therefore checks both structural boundaries and AI behavior quality, not just functional correctness.

14. **How to draw the architecture**
    - A good diagram should show three separate runtime boxes:
      - browser/Next.js frontend,
      - .NET content/API backend,
      - Python RAG service.
    - The browser talks primarily to `Backend_Web`, and likely to `Backend_AI` for chat/AI features.
    - `Backend_Web` talks to SQL Server and file storage/static assets.
    - `Backend_AI` talks to a vector database, scraping sources, and Google’s generative AI API.

Main entry points: [`Backend_AI/src/rag_chain.py`](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/rag_chain.py) (RAG chain), [`Backend_Web/Infrastructure`](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Infrastructure) (Infrastructure), [`hiast-frontend/lib`](https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/lib) (API clients), [`Backend_Web/Application`](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Application) (Application)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Next.js UI"]
  node_fe_public_ui["Public site<br/>Next.js pages"]
  node_fe_admin_ui["Admin UI<br/>Next.js admin"]
  node_fe_ui_components["UI components<br/>React components"]
  node_fe_api_clients["API clients<br/>frontend data access"]
  node_fe_i18n["Localization<br/>locale assets"]
end

subgraph group_web_backend[".NET content API"]
  node_web_presentation["Presentation<br/>ASP.NET API"]
  node_web_user_area["User area<br/>public controllers"]
  node_web_admin_area["Admin area<br/>crud controllers"]
  node_web_application["Application<br/>CQRS + DTOs"]
  node_web_domain["Domain<br/>entities + rules"]
  node_web_infrastructure["Infrastructure<br/>EF Core persistence"]
  node_web_db[("SQL Server<br/>relational store")]
  node_web_files["Static files<br/>uploaded media"]
end

subgraph group_ai_backend["Python RAG service"]
  node_ai_api["FastAPI app<br/>Python API<br/>[main.py]"]
  node_ai_rag_chain["RAG chain<br/>chat orchestration<br/>[rag_chain.py]"]
  node_ai_rag_abstractions["RAG strategies<br/>strategy interfaces"]
  node_ai_embedder["Embeddings<br/>vector pipeline<br/>[embedder.py]"]
  node_ai_scraper["URL scraper<br/>content ingestion<br/>[url_scraper.py]"]
  node_ai_token_mgr["Token manager<br/>memory control<br/>[token_manager.py]"]
  node_ai_models["Models<br/>schemas<br/>[models.py]"]
  node_ai_vector_db[("Vector DB<br/>Chroma-style store<br/>[scrapedData.json]")]
  node_ai_genai(("Google GenAI<br/>external model<br/>[system_prompt.json]"))
end

node_fe_public_ui -->|"fetches content"| node_fe_api_clients
node_fe_admin_ui -->|"CRUD/auth"| node_fe_api_clients
node_fe_ui_components -->|"renders"| node_fe_public_ui
node_fe_ui_components -->|"renders"| node_fe_admin_ui
node_fe_i18n -->|"localizes"| node_fe_public_ui
node_fe_i18n -->|"localizes"| node_fe_admin_ui
node_fe_api_clients -->|"calls API"| node_web_presentation
node_fe_api_clients -->|"chat calls"| node_ai_api
node_web_presentation -->|"routes read"| node_web_user_area
node_web_presentation -->|"routes manage"| node_web_admin_area
node_web_user_area -->|"uses services"| node_web_application
node_web_admin_area -->|"uses services"| node_web_application
node_web_application -->|"depends on"| node_web_domain
node_web_application -.->|"contracts only"| node_web_infrastructure
node_web_infrastructure -->|"persists"| node_web_domain
node_web_infrastructure -->|"stores data"| node_web_db
node_web_infrastructure -->|"serves files"| node_web_files
node_ai_api -->|"binds schemas"| node_ai_models
node_ai_api -->|"invokes chat"| node_ai_rag_chain
node_ai_rag_chain -->|"selects strategy"| node_ai_rag_abstractions
node_ai_rag_chain -->|"controls memory"| node_ai_token_mgr
node_ai_rag_chain -->|"retrieves chunks"| node_ai_embedder
node_ai_embedder -->|"indexes sources"| node_ai_scraper
node_ai_embedder -->|"writes vectors"| node_ai_vector_db
node_ai_scraper -->|"ingests content"| node_ai_vector_db
node_ai_rag_chain -->|"generates answer"| node_ai_genai

click node_fe_public_ui "https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/app"
click node_fe_admin_ui "https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/app/admin"
click node_fe_ui_components "https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/components"
click node_fe_api_clients "https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/lib"
click node_fe_i18n "https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/locales"
click node_web_presentation "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation"
click node_web_user_area "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation/Areas/User/Controllers"
click node_web_admin_area "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation/Areas/Admin"
click node_web_application "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Application"
click node_web_domain "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Domain"
click node_web_infrastructure "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Infrastructure"
click node_web_db "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Infrastructure/Migrations"
click node_web_files "https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation/wwwroot/Images"
click node_ai_api "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/main.py"
click node_ai_rag_chain "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/rag_chain.py"
click node_ai_rag_abstractions "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/rag_abstractions.py"
click node_ai_embedder "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/embedder.py"
click node_ai_scraper "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/url_scraper.py"
click node_ai_token_mgr "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/token_manager.py"
click node_ai_models "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/models.py"
click node_ai_vector_db "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/scrapedData.json"
click node_ai_genai "https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/system_prompt.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fe_public_ui,node_fe_admin_ui,node_fe_ui_components,node_fe_api_clients,node_fe_i18n toneBlue
class node_web_presentation,node_web_user_area,node_web_admin_area,node_web_application,node_web_domain,node_web_infrastructure,node_web_db,node_web_files toneAmber
class node_ai_api,node_ai_rag_chain,node_ai_rag_abstractions,node_ai_embedder,node_ai_scraper,node_ai_token_mgr,node_ai_models,node_ai_vector_db,node_ai_genai toneMint
```

## Components

### Next.js UI

Browser-facing public site, admin UI, and AI chat entrypoints.

- [Public site](https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/app): Next.js pages · Public-facing site content and browsing flows. · `hiast-frontend/app`
- [Admin UI](https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/app/admin): Next.js admin · Authenticated CRUD screens for managing site content. · `hiast-frontend/app/admin`
- [UI components](https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/components): React components · Shared layout, page sections, admin tables/forms, language controls, and chat widget. · `hiast-frontend/components`
- [API clients](https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/lib): frontend data access · Browser/server API helpers for public, admin, and auth flows. · `hiast-frontend/lib`
- [Localization](https://github.com/alooha-hub/hiast_rag/tree/main/hiast-frontend/locales): locale assets · English and Arabic translations consumed by the UI. · `hiast-frontend/locales`

### .NET content API

Clean Architecture backend for content, auth, and media management.

- [Presentation](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation): ASP.NET API · HTTP surface with controllers, middleware, and auth-aware entrypoints. · `Backend_Web/Presentation`
- [User area](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation/Areas/User/Controllers): public controllers · Read-oriented endpoints for browsing site content. · `Backend_Web/Presentation/Areas/User/Controllers`
- [Admin area](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation/Areas/Admin): crud controllers · Authenticated CRUD endpoints for content management. · `Backend_Web/Presentation/Areas/Admin`
- [Application](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Application): CQRS + DTOs · Use cases, DTOs, service interfaces, mapping, serialization, and response shaping. · `Backend_Web/Application`
- [Domain](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Domain): entities + rules · Core content entities, translations, enums, and identity models. · `Backend_Web/Domain`
- [Infrastructure](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Infrastructure): EF Core persistence · DbContext, repositories, unit of work, services, seed data, and migrations. · `Backend_Web/Infrastructure`
- [SQL Server](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Infrastructure/Migrations): relational store · Primary relational persistence target for the web backend. · `Backend_Web/Infrastructure/Migrations`
- [Static files](https://github.com/alooha-hub/hiast_rag/tree/main/Backend_Web/Presentation/wwwroot/Images): uploaded media · File and image storage surface for media and uploads. · `Backend_Web/Presentation/wwwroot/Images`

### Python RAG service

Standalone conversational retrieval service with scraping, embeddings, and chat orchestration.

- [FastAPI app](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/main.py): Python API · Service entrypoint exposing chat and data-management endpoints. · `Backend_AI/main.py`
- [RAG chain](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/rag_chain.py): chat orchestration · Pluggable retrieval-and-generation strategies and response assembly. · `Backend_AI/src/rag_chain.py`
- [RAG strategies](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/rag_abstractions.py): strategy interfaces · Abstractions for alternate retrieval and answer-generation approaches. · `Backend_AI/src/rag_abstractions.py`
- [Embeddings](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/embedder.py): vector pipeline · Chunk embedding and vector database integration. · `Backend_AI/src/embedder.py`
- [URL scraper](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/url_scraper.py): content ingestion · Scrapes and extracts source content before indexing. · `Backend_AI/src/url_scraper.py`
- [Token manager](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/token_manager.py): memory control · Conversation history and token-budget management for prompts. · `Backend_AI/src/token_manager.py`
- [Models](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/models.py): schemas · Request, response, and configuration models for the AI service. · `Backend_AI/src/models.py`
- [Vector DB](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/scrapedData.json): Chroma-style store · Persistent retrieval index for scraped content embeddings. · `Backend_AI/scrapedData.json`
- [Google GenAI](https://github.com/alooha-hub/hiast_rag/blob/main/Backend_AI/src/system_prompt.json): external model · Generative model used to produce conversational answers. · `Backend_AI/src/system_prompt.json`

## Connections

- Public site → API clients: fetches content · Public pages call frontend API helpers for site data.
- Admin UI → API clients: CRUD/auth · Admin screens use API clients for authenticated management flows.
- UI components → Public site: renders · Shared components compose the public pages.
- UI components → Admin UI: renders · Shared admin widgets compose CRUD screens.
- Localization → Public site: localizes · Locale files drive public language switching.
- Localization → Admin UI: localizes · Locale files also support admin UI language switching.
- API clients → Presentation: calls API · Frontend helpers talk primarily to the .NET backend.
- API clients → FastAPI app: chat calls · Chatbox and connectivity checks can reach the AI service.
- Presentation → User area: routes read · Presentation dispatches public read endpoints into the user area.
- Presentation → Admin area: routes manage · Presentation dispatches authenticated CRUD endpoints into the admin area.
- User area → Application: uses services · Controllers depend on application services and DTOs rather than data access.
- Admin area → Application: uses services · Admin controllers call application-layer service interfaces.
- Application → Domain: depends on · Use cases and DTO mappings are built on domain entities and rules.
- Application → Infrastructure: contracts only · Application defines interfaces consumed by infrastructure implementations.
- Infrastructure → Domain: persists · EF Core and services operate on domain entities.
- Infrastructure → SQL Server: stores data · Migrations and DbContext target the SQL database.
- Infrastructure → Static files: serves files · File service and static assets support uploaded media.
- FastAPI app → Models: binds schemas · API endpoints use request/response and configuration models.
- FastAPI app → RAG chain: invokes chat · The service entrypoint delegates question answering to the RAG pipeline.
- RAG chain → RAG strategies: selects strategy · Answering can switch between pluggable RAG approaches.
- RAG chain → Token manager: controls memory · Conversation history and prompt budget are managed before generation.
- RAG chain → Embeddings: retrieves chunks · RAG uses vector retrieval to fetch relevant context.
- Embeddings → URL scraper: indexes sources · Scraped content is turned into embeddings for later retrieval.
- Embeddings → Vector DB: writes vectors · Embeddings are stored in the vector database/index.
- URL scraper → Vector DB: ingests content · Source content is prepared for retrieval indexing.
- RAG chain → Google GenAI: generates answer · Retrieved context and prompt state are sent to the model provider.
