# adnanzamanniloy/contexforge architecture

> Architecture diagram of the GitHub repository adnanzamanniloy/contexforge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adnanzamanniloy/contexforge
- Repository: https://github.com/adnanzamanniloy/contexforge

## Overview

1. **Overall shape**
   - ContextForge is a two-part RAG system: a **Python backend** that ingests and indexes knowledge, and a **React/Vite frontend** that lets users upload/connect sources and chat against them.
   - The backend is the architectural core; the frontend is a thin client over API endpoints.

2. **User-facing interface**
   - `frontend/src` contains the workspace UI: repo input, upload panel, chat box, message rendering, and source viewing.
   - `frontend/src/services/api.js` and `frontend/src/hooks/useChat.js` indicate the UI talks to the backend over HTTP and manages chat interaction state on the client.
   - This layer does not own retrieval or generation logic; it only gathers user input and displays answers plus citations/sources.

3. **API boundary**
   - `backend/app/main.py` is the backend entrypoint, with route modules split by concern: `github`, `ingest`, and `query`.
   - `backend/app/schemas` defines request/response shapes, while `backend/app/services` contains the orchestration logic used by those routes.
   - This boundary separates transport/API concerns from the RAG pipeline implementation in `backend/core`.

4. **Ingestion sources**
   - The ingestion subsystem supports multiple source types through loaders in `backend/core/ingestion`: GitHub repositories, web pages, PDFs, DOCX, and plain text.
   - `backend/app/routes/ingest.py` likely accepts source descriptors and dispatches into `IngestService`, which selects the appropriate loader and starts the pipeline.
   - `backend/app/routes/github.py` suggests GitHub-specific handling is exposed as a first-class flow rather than a generic file upload only.

5. **Processing pipeline**
   - Raw documents are normalized in `backend/core/processing` using a cleaner, deduplicator, and metadata extractor.
   - The pipeline is designed to turn heterogeneous source content into a consistent internal representation before chunking and embedding.
   - `backend/core/pipeline.py` and `backend/core/orchestrator.py` likely coordinate these steps end-to-end, making this the main internal control flow for ingestion.

6. **Chunking and document preparation**
   - `backend/core/chunking` contains both `text_chunker.py` and `code_chunker.py`, which is important because the system explicitly handles repositories and code-aware splitting.
   - This is a key architectural boundary: code and prose are chunked differently so retrieval can preserve semantic and structural context.
   - The chunker output feeds embedding and indexing, not the UI directly.

7. **Indexing and storage**
   - The storage layer is split into `backend/core/storage/faiss_store.py` for dense vector retrieval and `backend/core/storage/bm25_index.py` for lexical retrieval.
   - This means the system maintains at least two retrieval indexes: one semantic and one keyword-based.
   - The storage boundary is the persistence layer of the RAG system; it sits between ingestion and query-time retrieval.

8. **Retrieval strategy**
   - `backend/core/retrieval` contains BM25, dense retrieval, hybrid retrieval, reciprocal-rank fusion, reranking, and HyDE.
   - This indicates query-time retrieval is multi-stage: generate/search candidates, combine sparse and dense results, then rerank the shortlist.
   - `backend/core/interfaces/retriever.py` suggests these strategies share a common abstraction, making retrieval pluggable.

9. **Embeddings and model abstraction**
   - `backend/core/embedding` includes `bge_embedder.py` and `voyage_embedder.py`, so the system can swap embedding providers/models.
   - `backend/core/interfaces/embedder.py` and `backend/core/interfaces/llm.py` show the project uses explicit interfaces to keep the rest of the pipeline provider-agnostic.
   - This is important architecturally because model choice is isolated from orchestration, indexing, and API code.

10. **Answer generation**
   - `backend/core/generation` contains provider implementations for Gemini and Groq, plus a fallback model and a prompt builder.
   - The query flow likely retrieves context, builds a grounded prompt, then sends it to an LLM through the generation interface.
   - The fallback layer suggests resilience when a primary LLM provider is unavailable or unsuitable.

11. **Query path end-to-end**
   - `backend/app/routes/query.py` and `backend/app/services/query_service.py` are the entry point for chat/search requests.
   - The typical flow is: user question → retrieval over indexed sources → optional reranking/HyDE → prompt construction → LLM response → response back to frontend.
   - The UI’s source viewer and chat components imply answers are surfaced with supporting context, not just raw generation.

12. **Observability, scripts, and validation**
   - `backend/observability/tracer.py` suggests tracing is built into the backend, likely to inspect pipeline stages and model calls.
   - `scripts/ingest_repo.py` and `scripts/query_cli.py` provide operational entry points outside the API, useful for offline ingestion or CLI-based querying.
   - Tests are split between unit tests for chunking, embedding cache, HyDE, reranking, and storage/retrieval, plus integration tests for the API, reflecting the main architectural seams.

13. **Deployment and environment**
   - `docker-compose.yml` and the backend/frontend split indicate the system is intended to run as separate services during development.
   - `backend/app/config/settings_example.py` and `frontend/.env.example` show configuration is environment-driven, which is typical for API keys, model/provider selection, and service endpoints.
   - The repo is therefore best diagrammed as: browser UI → backend API → ingestion/processing/index/storage → retrieval/generation providers, with observability and scripts as supporting side channels.

Main entry points: [`backend/tests`](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/tests) (Tests), [`backend/app/services`](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/app/services) (Services), [`backend/core/retrieval`](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/retrieval) (Retrieval), [`backend/core/pipeline.py`](https://github.com/adnanzamanniloy/contexforge/blob/main/backend/core/pipeline.py) (Pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_frontend_app["UI shell<br/>React app<br/>[App.jsx]"]
  node_frontend_chat["Chat UI<br/>client component<br/>[ChatBox.jsx]"]
  node_frontend_sources["Sources UI<br/>client component<br/>[RepoInput.jsx]"]
  node_frontend_api["API client<br/>HTTP service<br/>[api.js]"]
  node_frontend_chat_hook["Chat state<br/>state hook<br/>[useChat.js]"]
end

subgraph group_api["API Layer"]
  node_api_main["Server app<br/>FastAPI entry<br/>[main.py]"]
  node_api_routes["Routes"]
  node_api_services["Services<br/>orchestration"]
  node_api_schemas["Schemas<br/>contracts"]
end

subgraph group_core["RAG Core"]
  node_ingestion["Ingestion<br/>loaders"]
  node_processing["Processing<br/>normalize"]
  node_chunking["Chunking<br/>splitters"]
  node_storage[("Indexes<br/>storage")]
  node_retrieval["Retrieval<br/>search"]
  node_embedding["Embeddings<br/>model adapters"]
  node_generation["Generation<br/>LLM adapters"]
  node_pipeline["Pipeline<br/>orchestrator<br/>[pipeline.py]"]
end

subgraph group_ops["Ops"]
  node_obs["Tracing<br/>observability<br/>[tracer.py]"]
  node_cli["CLI tools<br/>scripts<br/>[ingest_repo.py]"]
  node_tests["Tests<br/>verification"]
end

node_frontend_app -->|"renders"| node_frontend_chat
node_frontend_app -->|"renders"| node_frontend_sources
node_frontend_chat -->|"uses"| node_frontend_chat_hook
node_frontend_chat_hook -->|"calls"| node_frontend_api
node_frontend_sources -->|"calls"| node_frontend_api
node_api_main -->|"registers"| node_api_routes
node_api_routes -->|"delegates"| node_api_services
node_api_routes -->|"validates"| node_api_schemas
node_api_services -->|"orchestrates"| node_pipeline
node_pipeline -->|"loads"| node_ingestion
node_ingestion -->|"normalizes"| node_processing
node_processing -->|"prepares"| node_chunking
node_chunking -->|"embeds"| node_embedding
node_embedding -->|"indexes"| node_storage
node_api_services -->|"queries"| node_retrieval
node_retrieval -->|"reads"| node_storage
node_retrieval -->|"grounds"| node_generation
node_generation -->|"returns"| node_api_services
node_pipeline -.->|"traces"| node_obs
node_cli -->|"drives"| node_pipeline
node_tests -.->|"covers"| node_chunking
node_tests -.->|"covers"| node_retrieval
node_tests -.->|"covers"| node_storage
node_tests -.->|"covers"| node_api_main

click node_frontend_app "https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/App.jsx"
click node_frontend_chat "https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/components/ChatBox.jsx"
click node_frontend_sources "https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/components/RepoInput.jsx"
click node_frontend_api "https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/services/api.js"
click node_frontend_chat_hook "https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/hooks/useChat.js"
click node_api_main "https://github.com/adnanzamanniloy/contexforge/blob/main/backend/app/main.py"
click node_api_routes "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/app/routes"
click node_api_services "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/app/services"
click node_api_schemas "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/app/schemas"
click node_ingestion "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/ingestion"
click node_processing "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/processing"
click node_chunking "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/chunking"
click node_storage "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/storage"
click node_retrieval "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/retrieval"
click node_embedding "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/embedding"
click node_generation "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/generation"
click node_pipeline "https://github.com/adnanzamanniloy/contexforge/blob/main/backend/core/pipeline.py"
click node_obs "https://github.com/adnanzamanniloy/contexforge/blob/main/backend/observability/tracer.py"
click node_cli "https://github.com/adnanzamanniloy/contexforge/blob/main/scripts/ingest_repo.py"
click node_tests "https://github.com/adnanzamanniloy/contexforge/tree/main/backend/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_app,node_frontend_chat,node_frontend_sources,node_frontend_api,node_frontend_chat_hook toneBlue
class node_api_main,node_api_routes,node_api_services,node_api_schemas toneAmber
class node_ingestion,node_processing,node_chunking,node_storage,node_retrieval,node_embedding,node_generation,node_pipeline toneMint
class node_obs,node_cli,node_tests toneRose
```

## Components

### Frontend

React/Vite user interface for upload, chat, and source viewing.

- [UI shell](https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/App.jsx): React app · Main client application and page composition. · `frontend/src/App.jsx`
- [Chat UI](https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/components/ChatBox.jsx): client component · Chat box, message rendering, and conversation input. · `frontend/src/components/ChatBox.jsx`
- [Sources UI](https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/components/RepoInput.jsx): client component · Repository input, upload panel, and source viewer. · `frontend/src/components/RepoInput.jsx`
- [API client](https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/services/api.js): HTTP service · Frontend-side backend communication and request helpers. · `frontend/src/services/api.js`
- [Chat state](https://github.com/adnanzamanniloy/contexforge/blob/main/frontend/src/hooks/useChat.js): state hook · Client chat interaction and UI state management. · `frontend/src/hooks/useChat.js`

### API Layer

FastAPI routes, schemas, and service orchestration.

- [Server app](https://github.com/adnanzamanniloy/contexforge/blob/main/backend/app/main.py): FastAPI entry · Backend application entrypoint and route registration. · `backend/app/main.py`
- [Routes](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/app/routes): HTTP routes · GitHub, ingest, and query endpoints. · `backend/app/routes`
- [Services](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/app/services): orchestration · Use-case orchestration for ingest and query flows. · `backend/app/services`
- [Schemas](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/app/schemas): contracts · Request and response models for API boundaries. · `backend/app/schemas`

### RAG Core

Ingestion, processing, indexing, retrieval, and generation pipeline.

- [Ingestion](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/ingestion): loaders · Source loaders for repositories, web pages, PDFs, DOCX, and text. · `backend/core/ingestion`
- [Processing](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/processing): normalize · Cleaning, deduplication, and metadata extraction. · `backend/core/processing`
- [Chunking](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/chunking): splitters · Code-aware and text-aware document splitting. · `backend/core/chunking`
- [Indexes](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/storage): storage · Dense and lexical retrieval indexes. · `backend/core/storage`
- [Retrieval](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/retrieval): search · BM25, dense, hybrid, fusion, reranking, and HyDE. · `backend/core/retrieval`
- [Embeddings](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/embedding): model adapters · Pluggable embedding providers and interfaces. · `backend/core/embedding`
- [Generation](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/core/generation): LLM adapters · Prompt building and provider-backed answer generation. · `backend/core/generation`
- [Pipeline](https://github.com/adnanzamanniloy/contexforge/blob/main/backend/core/pipeline.py): orchestrator · End-to-end ingestion and query pipeline coordination. · `backend/core/pipeline.py`

### Ops

CLI, tracing, tests, and deployment support.

- [Tracing](https://github.com/adnanzamanniloy/contexforge/blob/main/backend/observability/tracer.py): observability · Backend tracing for pipeline and model calls. · `backend/observability/tracer.py`
- [CLI tools](https://github.com/adnanzamanniloy/contexforge/blob/main/scripts/ingest_repo.py): scripts · Offline repository ingestion and query entry points. · `scripts/ingest_repo.py`
- [Tests](https://github.com/adnanzamanniloy/contexforge/tree/main/backend/tests): verification · Unit and integration coverage around the main seams. · `backend/tests`

## Connections

- UI shell → Chat UI: renders · The app composes the chat workspace.
- UI shell → Sources UI: renders · The app composes upload and source views.
- Chat UI → Chat state: uses · Chat UI delegates state and interactions to the hook.
- Chat state → API client: calls · Chat state issues query requests over HTTP.
- Sources UI → API client: calls · Upload and repo actions go through the API client.
- Server app → Routes: registers · The server mounts route modules.
- Routes → Services: delegates · Routes forward work to service layer orchestration.
- Routes → Schemas: validates · Routes use typed request and response models.
- Services → Pipeline: orchestrates · Services invoke the core pipeline for ingest and query.
- Pipeline → Ingestion: loads · Pipeline starts by loading source content.
- Ingestion → Processing: normalizes · Raw content is cleaned and enriched.
- Processing → Chunking: prepares · Processed documents are split into retrievable chunks.
- Chunking → Embeddings: embeds · Chunks are converted into vectors.
- Embeddings → Indexes: indexes · Embeddings are written to vector and lexical stores.
- Services → Retrieval: queries · Query service performs candidate search and reranking.
- Retrieval → Indexes: reads · Retrievers search the stored indexes.
- Retrieval → Generation: grounds · Retrieved context feeds grounded answer generation.
- Generation → Services: returns · Generated answers flow back to the query service.
- Pipeline → Tracing: traces · Pipeline stages and model calls can be observed.
- CLI tools → Pipeline: drives · Scripts reuse the same core pipeline outside the API.
- Tests → Chunking: covers · Chunking behavior is validated directly.
- Tests → Retrieval: covers · Retrieval and reranking behavior is validated directly.
- Tests → Indexes: covers · Storage and retrieval integration is validated.
- Tests → Server app: covers · API behavior is validated end-to-end.
