How adnanzamanniloy/contexforge works
1. Overall shape
- ContextForge is a two-part RAG system: a Python backend that ingests and indexes knowledge, and a React/Vite frontend that lets users upload/connect sources and chat against them.
- The backend is the architectural core; the frontend is a thin client over API endpoints.
2. User-facing interface
- frontend/src contains the workspace UI: repo input, upload panel, chat box, message rendering, and source viewing.
- frontend/src/services/api.js and frontend/src/hooks/useChat.js indicate the UI talks to the backend over HTTP and manages chat interaction state on the client.
- This layer does not own retrieval or generation logic; it only gathers user input and displays answers plus citations/sources.
3. API boundary
- backend/app/main.py is the backend entrypoint, with route modules split by concern: github, ingest, and query.
- backend/app/schemas defines request/response shapes, while backend/app/services contains the orchestration logic used by those routes.
- This boundary separates transport/API concerns from the RAG pipeline implementation in backend/core.
4. Ingestion sources
- The ingestion subsystem supports multiple source types through loaders in backend/core/ingestion: GitHub repositories, web pages, PDFs, DOCX, and plain text.
- backend/app/routes/ingest.py likely accepts source descriptors and dispatches into IngestService, which selects the appropriate loader and starts the pipeline.
- backend/app/routes/github.py suggests GitHub-specific handling is exposed as a first-class flow rather than a generic file upload only.
5. Processing pipeline
- Raw documents are normalized in backend/core/processing using a cleaner, deduplicator, and metadata extractor.
- The pipeline is designed to turn heterogeneous source content into a consistent internal representation before chunking and embedding.
- backend/core/pipeline.py and backend/core/orchestrator.py likely coordinate these steps end-to-end, making this the main internal control flow for ingestion.
6. Chunking and document preparation
- backend/core/chunking contains both text_chunker.py and code_chunker.py, which is important because the system explicitly handles repositories and code-aware splitting.
- This is a key architectural boundary: code and prose are chunked differently so retrieval can preserve semantic and structural context.
- The chunker output feeds embedding and indexing, not the UI directly.
7. Indexing and storage
- The storage layer is split into backend/core/storage/faiss_store.py for dense vector retrieval and backend/core/storage/bm25_index.py for lexical retrieval.
- This means the system maintains at least two retrieval indexes: one semantic and one keyword-based.
- The storage boundary is the persistence layer of the RAG system; it sits between ingestion and query-time retrieval.
8. Retrieval strategy
- backend/core/retrieval contains BM25, dense retrieval, hybrid retrieval, reciprocal-rank fusion, reranking, and HyDE.
- This indicates query-time retrieval is multi-stage: generate/search candidates, combine sparse and dense results, then rerank the shortlist.
- backend/core/interfaces/retriever.py suggests these strategies share a common abstraction, making retrieval pluggable.
9. Embeddings and model abstraction
- backend/core/embedding includes bge_embedder.py and voyage_embedder.py, so the system can swap embedding providers/models.
- backend/core/interfaces/embedder.py and backend/core/interfaces/llm.py show the project uses explicit interfaces to keep the rest of the pipeline provider-agnostic.
- This is important architecturally because model choice is isolated from orchestration, indexing, and API code.
10. Answer generation
- backend/core/generation contains provider implementations for Gemini and Groq, plus a fallback model and a prompt builder.
- The query flow likely retrieves context, builds a grounded prompt, then sends it to an LLM through the generation interface.
- The fallback layer suggests resilience when a primary LLM provider is unavailable or unsuitable.
11. Query path end-to-end
- backend/app/routes/query.py and backend/app/services/query_service.py are the entry point for chat/search requests.
- The typical flow is: user question → retrieval over indexed sources → optional reranking/HyDE → prompt construction → LLM response → response back to frontend.
- The UI’s source viewer and chat components imply answers are surfaced with supporting context, not just raw generation.
12. Observability, scripts, and validation
- backend/observability/tracer.py suggests tracing is built into the backend, likely to inspect pipeline stages and model calls.
- scripts/ingest_repo.py and scripts/query_cli.py provide operational entry points outside the API, useful for offline ingestion or CLI-based querying.
- Tests are split between unit tests for chunking, embedding cache, HyDE, reranking, and storage/retrieval, plus integration tests for the API, reflecting the main architectural seams.
13. Deployment and environment
- docker-compose.yml and the backend/frontend split indicate the system is intended to run as separate services during development.
- backend/app/config/settings_example.py and frontend/.env.example show configuration is environment-driven, which is typical for API keys, model/provider selection, and service endpoints.
- The repo is therefore best diagrammed as: browser UI → backend API → ingestion/processing/index/storage → retrieval/generation providers, with observability and scripts as supporting side channels.
Main entry points: tests Tests · services Services · retrieval Retrieval · pipeline.py Pipeline
Components 20
Frontend
React/Vite user interface for upload, chat, and source viewing.
- UI shell · Main client application and page composition.
frontend/src/App.jsx - Chat UI · Chat box, message rendering, and conversation input.
frontend/src/components/ChatBox.jsx - Sources UI · Repository input, upload panel, and source viewer.
frontend/src/components/RepoInput.jsx - API client · Frontend-side backend communication and request helpers.
frontend/src/services/api.js - Chat state · Client chat interaction and UI state management.
frontend/src/hooks/useChat.js
API Layer
FastAPI routes, schemas, and service orchestration.
- Server app · Backend application entrypoint and route registration.
backend/app/main.py - Routes · GitHub, ingest, and query endpoints.
backend/app/routes - Services · Use-case orchestration for ingest and query flows.
backend/app/services - Schemas · Request and response models for API boundaries.
backend/app/schemas
RAG Core
Ingestion, processing, indexing, retrieval, and generation pipeline.
- Ingestion · Source loaders for repositories, web pages, PDFs, DOCX, and text.
backend/core/ingestion - Processing · Cleaning, deduplication, and metadata extraction.
backend/core/processing - Chunking · Code-aware and text-aware document splitting.
backend/core/chunking - Indexes · Dense and lexical retrieval indexes.
backend/core/storage - Retrieval · BM25, dense, hybrid, fusion, reranking, and HyDE.
backend/core/retrieval - Embeddings · Pluggable embedding providers and interfaces.
backend/core/embedding - Generation · Prompt building and provider-backed answer generation.
backend/core/generation - Pipeline · End-to-end ingestion and query pipeline coordination.
backend/core/pipeline.py
Connections 24
- UI shell → Chat UI: renders · The app composes the chat workspace.
- UI shell → Sources UI: renders · The app composes upload and source views.
- Chat UI → Chat state: uses · Chat UI delegates state and interactions to the hook.
- Chat state → API client: calls · Chat state issues query requests over HTTP.
- Sources UI → API client: calls · Upload and repo actions go through the API client.
- Server app → Routes: registers · The server mounts route modules.
- Routes → Services: delegates · Routes forward work to service layer orchestration.
- Routes → Schemas: validates · Routes use typed request and response models.
- Services → Pipeline: orchestrates · Services invoke the core pipeline for ingest and query.
- Pipeline → Ingestion: loads · Pipeline starts by loading source content.
- Ingestion → Processing: normalizes · Raw content is cleaned and enriched.
- Processing → Chunking: prepares · Processed documents are split into retrievable chunks.
- Chunking → Embeddings: embeds · Chunks are converted into vectors.
- Embeddings → Indexes: indexes · Embeddings are written to vector and lexical stores.
- Services → Retrieval: queries · Query service performs candidate search and reranking.
- Retrieval → Indexes: reads · Retrievers search the stored indexes.
- Retrieval → Generation: grounds · Retrieved context feeds grounded answer generation.
- Generation → Services: returns · Generated answers flow back to the query service.
- Pipeline → Tracing: traces · Pipeline stages and model calls can be observed.
- CLI tools → Pipeline: drives · Scripts reuse the same core pipeline outside the API.
- Tests → Chunking: covers · Chunking behavior is validated directly.
- Tests → Retrieval: covers · Retrieval and reranking behavior is validated directly.
- Tests → Indexes: covers · Storage and retrieval integration is validated.
- Tests → Server app: covers · API behavior is validated end-to-end.