How csharikrishna/cortexrag works
1. What this repo is
- processor/ is the main runtime package for a production RAG backend.
- The system is organized around two primary flows: document ingestion/indexing and query retrieval/answering.
- It is designed to run as a containerized service with external storage/search components.
2. Runtime and deployment boundary
- docker-compose.yml is the top-level orchestration entrypoint.
- The backend service is built from the processor/dockerfile* images and launched through processor/entrypoint.py / entrypoint.sh.
- The README indicates the service depends materially on Weaviate for vector search and Neo4j for graph storage, with configuration driven by environment variables in processor/core/config.py.
3. API surface
- processor/api_server.py is the main API layer, exposing document upload and RAG query endpoints.
- The README calls out /upload for ingestion and /v1/rag for retrieval/generation.
- Static/UI assets in processor/static/ and processor/templates/ provide a lightweight web interface and sample pages, but the architecture is primarily API-first.
4. Document ingestion pipeline
- Incoming files are handled by processor/loaders/ with separate loaders for PDF and DOCX.
- The extraction layer in processor/extractors/ breaks documents into structured pieces: section slicing, boundary detection, metadata extraction, and reference extraction.
- This pipeline converts raw files into normalized document units suitable for chunking and indexing.
5. Chunking and normalization
- processor/indexing/chunking.py and related helpers in processor/utils/ handle chunk formation and cleanup.
- Utilities such as chunk_classifier, chunk_rewriter, and postprocessing indicate the system does more than naive splitting: it tries to preserve semantic boundaries and improve chunk quality before indexing.
- This is an important architectural boundary because retrieval quality depends on these normalized chunks, not raw source text.
6. Embedding and vector storage
- processor/indexing/embedding.py produces dense embeddings, using the model described in the README (BAAI/bge-large-en-v1.5).
- processor/databases/weaviate_store.py is the vector store adapter, and Weaviate is the primary dense retrieval backend.
- The ingestion flow writes chunk vectors and associated metadata into Weaviate so the retrieval layer can perform semantic search.
7. Graph storage and relationship modeling
- processor/databases/graphdb.py connects the pipeline to Neo4j.
- The readme’s “knowledge graph integration” plus the presence of reference extraction suggests ingestion also materializes document relationships, links, and multi-hop structure into the graph.
- This creates a second persistence tier alongside the vector index: Weaviate for similarity search, Neo4j for relational traversal.
8. Retrieval orchestration
- processor/retrieval/engine.py is the main coordinator for query-time execution.
- query_analyzer.py classifies queries by complexity, which supports the repo’s low/high/complex routing strategy.
- retrieval_store.py, cache.py, and reranker.py show the query path is multi-stage: candidate retrieval, optional cache reuse, then cross-encoder-style reranking before final context selection.
9. Answer assembly
- context_assembler.py is the boundary where ranked evidence becomes the final prompt/context passed to the model layer.
- quality.py and evaluator.py indicate the system measures retrieval or context quality before returning results.
- The README’s mention of an external LLM provider (OLLAMA_URL, OLLAMA_MODEL) means generation is delegated to a configurable model service rather than embedded in the API layer.
10. State, jobs, and lightweight persistence
- processor/databases/jobs.db suggests local SQLite-backed job or pipeline state tracking.
- This is separate from the search indexes and likely used for ingestion status, bookkeeping, or operational metadata.
- It is a small but important boundary: ephemeral/local operational state vs. the durable search stores.
11. Integration and operational tooling
- check_weaviate.py and service_connectivity_test.py are health/connectivity utilities for external dependencies.
- evaluate_retrieval.py, load_test_retrieval.py, and processor/tests/ provide validation, benchmarking, and regression coverage for the retrieval stack.
- processor/mcp_retrieval_tool.py indicates an additional integration surface for exposing retrieval capabilities to external tooling or agents.
12. Architectural summary
- The core diagram should show: Client/UI → API server → ingestion pipeline → loaders/extractors/chunking → Weaviate + Neo4j + local job state.
- For queries: Client/UI → API server → query analyzer → retrieval engine → Weaviate/Neo4j/cache → reranker → context assembler → LLM provider.
- The main architectural theme is a dual-store RAG system where vector similarity and graph relationships are both first-class retrieval signals.
Main entry points: api_server.py API Server · engine.py Retrieval Engine · data_extractor.py Extractors · retrieval_store.py Retrieval Store
Components 20
Runtime & API
Container entrypoint and API surface for the RAG service.
- Compose · Top-level service orchestration entrypoint.
docker-compose.yml - API Server · Main ingestion and RAG query API layer.
processor/api_server.py - Entrypoint · Container startup and process launch.
processor/entrypoint.py - Config · Environment-driven runtime and integration configuration.
processor/core/config.py - UI Assets · Lightweight web interface and sample pages.
processor/static/rag_ui.html
Ingestion Pipeline
Document loading, extraction, chunking, embedding, and indexing.
- Loaders · Document type adapters for incoming files.
processor/loaders/loader.py - Extractors · Structured document parsing and reference extraction.
processor/extractors/data_extractor.py - Chunking · Forms normalized semantic chunks from extracted content.
processor/indexing/chunking.py - Embeddings · Produces dense embeddings for indexed chunks.
processor/indexing/embedding.py
Query & Answering
Query-time routing, retrieval, reranking, and context assembly.
- Analyzer · Classifies query complexity and selects retrieval strategy.
processor/retrieval/query_analyzer.py - Retrieval Engine · Coordinates query-time retrieval and answer preparation.
processor/retrieval/engine.py - Retrieval Store · Retrieves candidates from search backends.
processor/retrieval/retrieval_store.py - Cache · Reusable intermediate retrieval results.
processor/retrieval/cache.py - Reranker · Re-scores retrieved candidates before final selection.
processor/retrieval/reranker.py - Context Assembler · Builds the final evidence context for generation.
processor/retrieval/context_assembler.py - Quality Checks · Measures retrieval or context quality.
processor/retrieval/quality.py
Storage & Integrations
External and local persistence used by the system.
- Weaviate Store · Primary semantic retrieval and vector persistence layer.
processor/databases/weaviate_store.py - Graph DB · Neo4j integration for document relationships and traversal.
processor/databases/graphdb.py - Jobs DB · Operational job and pipeline state tracking.
processor/databases/jobs.db - LLM Provider · External model service used for answer generation.
Connections 21
- Compose → Entrypoint: starts · Orchestrates the service container.
- Entrypoint → API Server: launches · Boots the main API process.
- API Server → Config: reads · Loads runtime settings and provider endpoints.
- API Server → Loaders: ingests · Accepts uploaded documents for processing.
- Loaders → Extractors: passes · Raw files are normalized into structured content.
- Extractors → Chunking: segments · Document structure is converted into chunks.
- Chunking → Embeddings: encodes · Chunks are prepared for vector embedding.
- Embeddings → Weaviate Store: stores · Writes dense vectors and metadata to the vector index.
- Extractors → Graph DB: links · Extracted references and relationships are materialized in the graph.
- API Server → Jobs DB: tracks · Records ingestion status and operational state.
- API Server → Analyzer: routes · Classifies inbound questions by complexity.
- Analyzer → Retrieval Engine: selects · Hands off to the appropriate retrieval strategy.
- Retrieval Engine → Retrieval Store: fetches · Pulls candidate evidence from search backends.
- Retrieval Store → Weaviate Store: searches · Uses semantic similarity search.
- Retrieval Store → Graph DB: traverses · Uses graph relationships for multi-hop evidence.
- Retrieval Engine → Cache: reuses · Checks for reusable query artifacts.
- Retrieval Engine → Reranker: ranks · Refines candidate ordering before answer assembly.
- Reranker → Context Assembler: feeds · Top evidence is assembled into final context.
- Context Assembler → Quality Checks: checks · Context quality is evaluated before returning results.
- Context Assembler → LLM Provider: prompts · Final context is sent to the external LLM provider.
- API Server → UI Assets: serves · Exposes a lightweight browser interface.