# ginnezhang/novasearch architecture

> Architecture diagram of the GitHub repository ginnezhang/novasearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ginnezhang/novasearch
- Repository: https://github.com/ginnezhang/novasearch

## Overview

##### What this project is (type + purpose)
**NovaSearch** is a **backend-first “enterprise copilot + retrieval engine”** implemented in Python. It exposes a **FastAPI** service that orchestrates:
- **Ingestion/ETL**: chunk documents, extract entities, build a **knowledge graph**, and index content
- **Hybrid retrieval**: sparse + dense + graph retrieval with fusion and reranking
- **Agentic reasoning**: query understanding, decomposition, memory, and controlled LLM generation
- **Persistence layer**: Postgres/pgvector, Redis, Neo4j (per README + docker-compose)

##### Offline/async ingestion + indexing
2) **Online query-time serving** (API → agent → tri-engine retrieval → rerank → grounded generation)

##### Serving / API layer
- `api/main.py`, `api/schemas.py`: FastAPI endpoints + request/response models

##### Agent & orchestration
- `agent/copilot_agent.py`: the reasoning loop / tool-using agent that calls retrieval, memory, and LLMs

##### Core shared services
- `core/memory.py`: Redis-backed conversation/session memory (context memory)
- `core/db_init.py`: DB initialization wiring (Postgres/Neo4j/etc)

##### Ingestion pipeline
- `ingestion/chunking/semantic_chunker.py`: chunking strategy (sliding window / semantic)
- `ingestion/graph_build/kg_builder.py`: entity linking / KG build and Neo4j population

##### Retrieval system
- `retrieval/hybrid_search.py`: orchestrator for “tri-engine fusion”
- `retrieval/sparse/keyword_search.py`: BM25/Elasticsearch-like keyword retrieval
- `retrieval/dense/vector_search.py`: FAISS/PGVector-like dense retrieval
- `retrieval/reranker/*`: reranking + fusion methods
- `cross_encoder.py`: deep reranker
- `rrf_fusion.py`: reciprocal rank fusion / blending

##### Infra / runtime environment
- `docker-compose.yml`: local Postgres+pgvector, Redis, Neo4j
- `.env.example`: configuration surfaces (API keys, DB creds)

##### Client → Serving layer
- **Clients** (browser/internal app/CLI) calling REST endpoints
- **FastAPI Service** (from `api/`): request validation (`schemas.py`), streaming responses (mentioned in README)

##### Orchestration / reasoning layer (core runtime)
- **Copilot Agent** (`agent/copilot_agent.py`) containing:
- Query understanding (rewrite / decomposition)
- Tool selection (calls hybrid retrieval, graph traversal)
- Controlled LLM generation (grounded outputs + citations)

- **Memory service** (`core/memory.py`) backed by **Redis**
- Read/write of conversation state, threads, session context

- **Configuration + DB init** (`core/db_init.py`) as the wiring to Postgres/Neo4j/Redis

- **LLM providers** as *external services*
- OpenAI/Anthropic/local models as outbound calls from agent

##### Retrieval layer (tri-engine)
Model as three retrieval “engines” that the orchestrator queries **in parallel**:

##### Sparse Retrieval (`retrieval/sparse/keyword_search.py`)
- backend: Elasticsearch/BM25 (README). Even if ES isn’t in compose, show it as a logical dependency.

##### Dense Retrieval (`retrieval/dense/vector_search.py`)
- backend: **Postgres + pgvector** and/or FAISS (README)
- An “Embedding model” dependency for indexing + query embeddings (BGE/E5/OpenAI embeddings)

##### Graph Retrieval / KG Reasoning
- backed by **Neo4j**
- query uses entity linking / traversal logic (partly in ingestion + possibly agent-time)

- **Hybrid Search Coordinator** (`retrieval/hybrid_search.py`) aggregating results
- **Fusion** (`retrieval/reranker/rrf_fusion.py`)
- **Reranker** (`retrieval/reranker/cross_encoder.py`) to reorder and filter

##### Storage / persistence layer
- **PostgreSQL + pgvector**: document metadata, embeddings, chunks, maybe audit logs
- **Neo4j**: entities/relations + SOP/medical/legal knowledge constraints
- **Redis**: session memory / state / caching

##### Ingestion / indexing pipeline (offline)
A separate “Ingestion Worker / Job” (even if it currently runs in-process) that does:
- Input sources: enterprise docs, PDFs, SOP pages, tables, images (README says multimodal)
- **Chunking** (`semantic_chunker.py`)
- **Embedding generation** (external embedding model API)
- Write vectors + chunks → Postgres/pgvector (and/or FAISS)
- **KG Builder** (`kg_builder.py`) → Neo4j population
- (Optional) Keyword index build → Elasticsearch/BM25 index

This ingestion side is crucial: it explains where the retrieval engines get their indexes.

##### Hybrid retrieval fans out
- → Sparse search (BM25/ES)
- → Dense search (pgvector/FAISS) + embedding model call for query vector
- → Graph retrieval (Neo4j traversal / constraints)

##### (Optionally) store doc registry/metadata in Postgres

- **Orchestrator pattern**: FastAPI + agent coordinates multiple retrieval subsystems
- **Polyglot persistence**: Postgres/pgvector + Redis + Neo4j (+ optionally Elasticsearch)
- **Two-plane architecture**: ingestion/indexing plane vs serving/query plane
- **Evidence-grounded generation**: LLM is downstream of retrieval + KG constraints, not a standalone responder
- **Fusion + reranking stage**: explicitly show “candidate generation” then “scoring/ranking” before final answer

Shows: Clients → NovaSearch API Service → (Redis, Postgres/pgvector, Neo4j, Elasticsearch) + External LLMs.

Shows internal modules: `api/*`, `agent/*`, `retrieval/*`, `core/*` and their calls.

- Query-time DFD (fan-out retrieval + reranking + generation)
- Ingestion-time DFD (chunk → embed → index; entity link → Neo4j)

This set will be the most “accurate” representation of both the **code boundaries** and the **runtime architecture** implied by Docker/README.

## Diagram (Mermaid)

```mermaid
flowchart TB

%% =========================================================
%% 1) Context / Container (C4 L2)
%% =========================================================
subgraph C1["Context / Containers (C4 L2)"]
direction TB

Clients["Clients\n(Browser / Internal App / CLI)"]:::external

subgraph Nova["NovaSearch Platform"]
direction TB
FastAPI["FastAPI API Service\n(api/main.py)"]:::svc
end

subgraph Stores["Polyglot Persistence"]
direction TB
Postgres[("PostgreSQL + pgvector")]:::store
Redis[("Redis")]:::store
Neo4j[("Neo4j")]:::store
Elastic[("Elasticsearch / BM25 Index\n(logical)")]:::store
end

LLMCloud["LLM Providers\n(OpenAI / Anthropic / Local)"]:::cloud
EmbedCloud["Embedding Model API\n(BGE/E5/OpenAI embeddings)"]:::cloud

Clients -->|"REST/streaming"| FastAPI
FastAPI -->|"reads/writes"| Redis
FastAPI -->|"reads/writes"| Postgres
FastAPI -->|"graph_queries"| Neo4j
FastAPI -->|"keyword_queries"| Elastic
FastAPI -.->|"generation_calls"| LLMCloud
FastAPI -.->|"embedding_calls"| EmbedCloud
end

%% =========================================================
%% 2) Component diagram (C4 L3) - API Service internals
%% =========================================================
subgraph C2["API Service Components (C4 L3)"]
direction TB

subgraph API["Serving / API Layer"]
direction TB
Schemas["Request/Response Schemas\n(api/schemas.py)"]:::module
end

subgraph Orchestrator["Orchestration / Reasoning Layer"]
direction TB
Agent["Copilot Agent\n(agent/copilot_agent.py)"]:::svc
Memory["Memory Service\n(core/memory.py)"]:::module
DBInit["DB/Store Init & Wiring\n(core/db_init.py)"]:::module
end

subgraph Retrieval["Retrieval Layer (Tri-engine + Ranking)"]
direction TB
Hybrid["Hybrid Search Coordinator\n(retrieval/hybrid_search.py)"]:::svc

subgraph Engines["Candidate Generation (Parallel)"]
direction TB
Sparse["Sparse Retrieval Engine\n(retrieval/sparse/keyword_search.py)"]:::module
Dense["Dense Retrieval Engine\n(retrieval/dense/vector_search.py)"]:::module
Graph["Graph Retrieval / KG Reasoning\n(Neo4j traversal)"]:::module
end

subgraph Rank["Fusion + Reranking"]
direction TB
RRF["RRF Fusion\n(retrieval/reranker/rrf_fusion.py)"]:::module
CrossEnc["Cross-Encoder Reranker\n(retrieval/reranker/cross_encoder.py)"]:::module
end
end

FastAPI -->|"validate"| Schemas
FastAPI -->|"orchestrate"| Agent

Agent -->|"load/store_context"| Memory
Memory -->|"session_state"| Redis

Agent -->|"init_clients"| DBInit
DBInit -->|"connect"| Postgres
DBInit -->|"connect"| Neo4j
DBInit -->|"connect"| Redis

Agent -->|"retrieve"| Hybrid
Hybrid -->|"fanout"| Sparse
Hybrid -->|"fanout"| Dense
Hybrid -->|"fanout"| Graph

Sparse -->|"keyword_index"| Elastic
Dense -->|"vector_store"| Postgres
Dense -.->|"query_embedding"| EmbedCloud
Graph -->|"traversal"| Neo4j

Hybrid -->|"candidates"| RRF
RRF -->|"fused_ranks"| CrossEnc
CrossEnc -->|"top_evidence"| Agent

Agent -.->|"grounded_generation"| LLMCloud
Agent -->|"stream_answer+citation"| FastAPI
end

%% =========================================================
%% 3) Data-flow diagrams (Online Query + Offline Ingestion)
%% =========================================================
subgraph C3["Data Flows"]
direction TB

subgraph Q1["Online Query-time Flow (Synchronous)"]
direction TB
QClient["Client Query"]:::external
QAPI["FastAPI Endpoint"]:::svc
QAgent["Copilot Agent Loop"]:::svc
QMem["Redis Memory Read/Write"]:::store
QHybrid["Hybrid Retrieval"]:::svc
QRank["Fusion + Rerank"]:::module
QGen["Evidence-grounded LLM Generation"]:::cloud
QResp["Streaming Response + Citations"]:::external

QClient -->|"1)request"| QAPI
QAPI -->|"2)invoke"| QAgent
QAgent <-->|"3)context"| QMem
QAgent -->|"4)retrieve"| QHybrid
QHybrid -->|"5a)sparse"| Sparse
QHybrid -->|"5b)dense"| Dense
QHybrid -->|"5c)graph"| Graph
QHybrid -->|"6)aggregate"| QRank
QRank -->|"7)evidence"| QAgent
QAgent -.->|"8)generate"| QGen
QGen -.->|"9)tokens"| QAgent
QAgent -->|"10)stream"| QAPI
QAPI -->|"11)response"| QResp
end

subgraph I1["Offline / Async Ingestion & Indexing Flow (Batch)"]
direction TB
DocSources["Document Sources\n(PDFs / SOP pages / tables / images)"]:::source
IngestJob["Ingestion Worker / Job\n(ingestion/)"]:::svc
Chunker["Semantic Chunker\n(ingestion/chunking/semantic_chunker.py)"]:::module
Embedder["Embedding Generation"]:::cloud
KGBuilder["KG Builder / Entity Linking\n(ingestion/graph_build/kg_builder.py)"]:::module
KeyIndex["Keyword Index Build\n(BM25/Elasticsearch)"]:::module

DocSources -.->|"1)ingest"| IngestJob
IngestJob -.->|"2)chunk"| Chunker
Chunker -.->|"3)embed_calls"| Embedder
Embedder -.->|"4)vectors+chunks"| Postgres
IngestJob -.->|"5)entities/relations"| KGBuilder
KGBuilder -.->|"6)upsert_graph"| Neo4j
IngestJob -.->|"7)keywords"| KeyIndex
KeyIndex -.->|"8)index_docs"| Elastic
end

subgraph T1["Testing / Confidence"]
direction TB
E2E["End-to-end System Test\n(tests/test_e2e.py)"]:::test
E2E -->|"exercises_query_flow"| FastAPI
E2E -.->|"requires_stores"| Postgres
E2E -.->|"requires_stores"| Redis
E2E -.->|"requires_stores"| Neo4j
end

subgraph Infra["Infra / Runtime Environment"]
direction TB
Compose["Docker Compose\n(docker-compose.yml)"]:::infra
Env["Environment Template\n(.env.example)"]:::infra
Compose -->|"boots"| Postgres
Compose -->|"boots"| Redis
Compose -->|"boots"| Neo4j
Compose -.->|"optional"| Elastic
Env -.->|"configures"| FastAPI
Env -.->|"configures"| Agent
end

end

%% =========================================================
%% Click events (component_mapping)
%% =========================================================
click FastAPI "https://github.com/ginnezhang/novasearch/blob/main/api/main.py"
click Schemas "https://github.com/ginnezhang/novasearch/blob/main/api/schemas.py"
click Agent "https://github.com/ginnezhang/novasearch/blob/main/agent/copilot_agent.py"
click Memory "https://github.com/ginnezhang/novasearch/blob/main/core/memory.py"
click DBInit "https://github.com/ginnezhang/novasearch/blob/main/core/db_init.py"
click Hybrid "https://github.com/ginnezhang/novasearch/blob/main/retrieval/hybrid_search.py"
click Sparse "https://github.com/ginnezhang/novasearch/blob/main/retrieval/sparse/keyword_search.py"
click Dense "https://github.com/ginnezhang/novasearch/blob/main/retrieval/dense/vector_search.py"
click CrossEnc "https://github.com/ginnezhang/novasearch/blob/main/retrieval/reranker/cross_encoder.py"
click RRF "https://github.com/ginnezhang/novasearch/blob/main/retrieval/reranker/rrf_fusion.py"
click IngestJob "https://github.com/ginnezhang/novasearch/tree/main/ingestion/"
click Chunker "https://github.com/ginnezhang/novasearch/blob/main/ingestion/chunking/semantic_chunker.py"
click KGBuilder "https://github.com/ginnezhang/novasearch/blob/main/ingestion/graph_build/kg_builder.py"
click Compose "https://github.com/ginnezhang/novasearch/blob/main/docker-compose.yml"
click Env "https://github.com/ginnezhang/novasearch/blob/main/.env.example"
click E2E "https://github.com/ginnezhang/novasearch/blob/main/tests/test_e2e.py"

%% =========================================================
%% Styles
%% =========================================================
classDef svc fill:#2B6CB0,stroke:#1A365D,color:#FFFFFF,stroke-width:1px
classDef module fill:#3182CE,stroke:#1A365D,color:#FFFFFF,stroke-width:1px
classDef store fill:#DD6B20,stroke:#7B341E,color:#FFFFFF,stroke-width:1px
classDef cloud fill:#718096,stroke:#2D3748,color:#FFFFFF,stroke-width:1px
classDef external fill:#4A5568,stroke:#1A202C,color:#FFFFFF,stroke-width:1px
classDef source fill:#2F855A,stroke:#22543D,color:#FFFFFF,stroke-width:1px
classDef infra fill:#805AD5,stroke:#44337A,color:#FFFFFF,stroke-width:1px
classDef test fill:#D53F8C,stroke:#702459,color:#FFFFFF,stroke-width:1px
```
