How alooha-hub/hiast_rag works
1. Repo shape
- This monorepo contains three independently built subsystems:
- Backend_AI/: a Python FastAPI service for conversational RAG.
- Backend_Web/: a .NET 8 backend using Clean Architecture.
- hiast-frontend/: a Next.js app for public site + admin UI.
- The main architectural boundaries are: AI search/chat, business/content API, and browser-facing UI.
2. AI backend (Backend_AI)
- This is a separate AI microservice, not part of the .NET backend.
- It exposes conversation and data-management endpoints and is centered around retrieval-augmented generation.
- Core subsystems include scraping (url_scraper.py), embedding/vector storage (embedder.py), conversational orchestration (rag_chain.py), token/history control (token_manager.py), and request/response models (models.py).
3. AI data flow
- Content is scraped from URLs, converted into embeddings, and stored in a vector database.
- User questions go through a RAG pipeline that retrieves relevant chunks, applies conversation memory, and calls a Google generative model.
- The repo also includes multiple RAG strategies in rag_abstractions.py / rag_chain.py, so the answer path is pluggable rather than fixed.
4. AI persistence and external dependencies
- The AI stack depends on ChromaDB-style vector retrieval and Google GenAI via API key.
- It also uses HTML/document parsing tooling for scraping and extraction.
- Test assets such as squad_2000_samples.json and results.json show that retrieval quality and strategy comparisons are part of the design.
5. Web backend architecture (Backend_Web)
- This backend is clearly layered:
- Domain: entities, enums, and shared base types.
- Application: DTOs, CQRS commands/queries, mapper profiles, service interfaces, serializers, response models.
- Infrastructure: EF Core DbContext, migrations, repository/unit-of-work implementations, domain services, seeding.
- Presentation: ASP.NET Core controllers, middleware, auth, and API surface.
- The architecture tests explicitly enforce that inner layers do not depend on outer layers.
6. Web backend request path
- Requests enter Presentation, where controllers are split into Areas/Admin and Areas/User.
- Controllers do not appear to access the database directly; they call services defined in Application/IServices.
- Services are implemented in Infrastructure/Services and use repository/unit-of-work abstractions plus EF Core for persistence.
7. Web backend API partitioning
- There is a strong read/write boundary:
- User area: public, mostly read-only endpoints for browsing/filtering content.
- Admin area: authenticated CRUD endpoints for managing content.
- AuthenticationController and base controller types centralize auth context and response formatting.
- Responses are standardized through ApiResponse and a custom JSON serializer/action result path.
8. Web backend domain model
- The domain is content-heavy and multilingual.
- Many entities have translation companions, such as Program/ProgramTranslation, Page/PageTranslation, Book/BookTranslation, etc.
- The model also includes admissions, courses, media, FAQs, testimonials, slides, locations, languages, and training-related content.
- This indicates a CMS-like backend with localized content and related media management.
9. Web backend infrastructure
- Persistence is backed by Entity Framework Core with SQL Server migrations already checked in.
- DataSeeder suggests the system can bootstrap initial content/reference data.
- wwwroot/Images indicates the backend may also serve or store uploaded static assets.
- JWT auth and ASP.NET Identity are part of the backend security boundary.
10. Frontend architecture (hiast-frontend)
- The frontend is a Next.js app using the App Router structure.
- It contains:
- public pages for site content,
- admin pages for CRUD workflows,
- shared UI components,
- API client modules under lib/,
- localization files under locales/.
- The frontend is not just presentational; it acts as the main consumer of the web backend APIs.
11. Frontend data flow
- Public pages fetch content through lib/public*.js and entity-specific API modules.
- Admin pages use corresponding admin API helpers and forms/tables for CRUD interaction.
- Middleware and auth utilities suggest route protection for admin sections.
- The presence of AIChatbox.jsx and ConnectionTest.jsx suggests the UI also integrates with the AI service for conversational features and service connectivity checks.
12. Cross-cutting concerns
- Localization is a first-class concern across backend and frontend:
- backend entities have translation tables,
- frontend has en.json and ar.json,
- language switchers exist in both public and admin UI.
- Authentication is centralized in the .NET backend and consumed by the frontend.
- Media/file handling is also a shared concern: backend services and DTOs model uploads, while frontend has dedicated media forms and viewers.
13. Testing and quality gates
- Backend_Web/ArchitectureTests validates Clean Architecture dependency rules.
- Backend_AI/test/ includes RAG-specific tests and dataset tooling.
- This repo therefore checks both structural boundaries and AI behavior quality, not just functional correctness.
14. How to draw the architecture
- A good diagram should show three separate runtime boxes:
- browser/Next.js frontend,
- .NET content/API backend,
- Python RAG service.
- The browser talks primarily to Backend_Web, and likely to Backend_AI for chat/AI features.
- Backend_Web talks to SQL Server and file storage/static assets.
- Backend_AI talks to a vector database, scraping sources, and Google’s generative AI API.
Main entry points: rag_chain.py RAG chain · Infrastructure Infrastructure · lib API clients · Application Application
Components 22
Next.js UI
Browser-facing public site, admin UI, and AI chat entrypoints.
- Public site · Public-facing site content and browsing flows.
hiast-frontend/app - Admin UI · Authenticated CRUD screens for managing site content.
hiast-frontend/app/admin - UI components · Shared layout, page sections, admin tables/forms, language controls, and chat widget.
hiast-frontend/components - API clients · Browser/server API helpers for public, admin, and auth flows.
hiast-frontend/lib - Localization · English and Arabic translations consumed by the UI.
hiast-frontend/locales
.NET content API
Clean Architecture backend for content, auth, and media management.
- Presentation · HTTP surface with controllers, middleware, and auth-aware entrypoints.
Backend_Web/Presentation - User area · Read-oriented endpoints for browsing site content.
Backend_Web/Presentation/Areas/User/Controllers - Admin area · Authenticated CRUD endpoints for content management.
Backend_Web/Presentation/Areas/Admin - Application · Use cases, DTOs, service interfaces, mapping, serialization, and response shaping.
Backend_Web/Application - Domain · Core content entities, translations, enums, and identity models.
Backend_Web/Domain - Infrastructure · DbContext, repositories, unit of work, services, seed data, and migrations.
Backend_Web/Infrastructure - SQL Server · Primary relational persistence target for the web backend.
Backend_Web/Infrastructure/Migrations - Static files · File and image storage surface for media and uploads.
Backend_Web/Presentation/wwwroot/Images
Python RAG service
Standalone conversational retrieval service with scraping, embeddings, and chat orchestration.
- FastAPI app · Service entrypoint exposing chat and data-management endpoints.
Backend_AI/main.py - RAG chain · Pluggable retrieval-and-generation strategies and response assembly.
Backend_AI/src/rag_chain.py - RAG strategies · Abstractions for alternate retrieval and answer-generation approaches.
Backend_AI/src/rag_abstractions.py - Embeddings · Chunk embedding and vector database integration.
Backend_AI/src/embedder.py - URL scraper · Scrapes and extracts source content before indexing.
Backend_AI/src/url_scraper.py - Token manager · Conversation history and token-budget management for prompts.
Backend_AI/src/token_manager.py - Models · Request, response, and configuration models for the AI service.
Backend_AI/src/models.py - Vector DB · Persistent retrieval index for scraped content embeddings.
Backend_AI/scrapedData.json - Google GenAI · Generative model used to produce conversational answers.
Backend_AI/src/system_prompt.json
Connections 26
- Public site → API clients: fetches content · Public pages call frontend API helpers for site data.
- Admin UI → API clients: CRUD/auth · Admin screens use API clients for authenticated management flows.
- UI components → Public site: renders · Shared components compose the public pages.
- UI components → Admin UI: renders · Shared admin widgets compose CRUD screens.
- Localization → Public site: localizes · Locale files drive public language switching.
- Localization → Admin UI: localizes · Locale files also support admin UI language switching.
- API clients → Presentation: calls API · Frontend helpers talk primarily to the .NET backend.
- API clients → FastAPI app: chat calls · Chatbox and connectivity checks can reach the AI service.
- Presentation → User area: routes read · Presentation dispatches public read endpoints into the user area.
- Presentation → Admin area: routes manage · Presentation dispatches authenticated CRUD endpoints into the admin area.
- User area → Application: uses services · Controllers depend on application services and DTOs rather than data access.
- Admin area → Application: uses services · Admin controllers call application-layer service interfaces.
- Application → Domain: depends on · Use cases and DTO mappings are built on domain entities and rules.
- Application → Infrastructure: contracts only · Application defines interfaces consumed by infrastructure implementations.
- Infrastructure → Domain: persists · EF Core and services operate on domain entities.
- Infrastructure → SQL Server: stores data · Migrations and DbContext target the SQL database.
- Infrastructure → Static files: serves files · File service and static assets support uploaded media.
- FastAPI app → Models: binds schemas · API endpoints use request/response and configuration models.
- FastAPI app → RAG chain: invokes chat · The service entrypoint delegates question answering to the RAG pipeline.
- RAG chain → RAG strategies: selects strategy · Answering can switch between pluggable RAG approaches.
- RAG chain → Token manager: controls memory · Conversation history and prompt budget are managed before generation.
- RAG chain → Embeddings: retrieves chunks · RAG uses vector retrieval to fetch relevant context.
- Embeddings → URL scraper: indexes sources · Scraped content is turned into embeddings for later retrieval.
- Embeddings → Vector DB: writes vectors · Embeddings are stored in the vector database/index.
- URL scraper → Vector DB: ingests content · Source content is prepared for retrieval indexing.
- RAG chain → Google GenAI: generates answer · Retrieved context and prompt state are sent to the model provider.