How ginnezhang/novasearch works
What this project is (type + purpose)
NovaSearch is a backend-first “enterprise copilot + retrieval engine” implemented in Python. It exposes a FastAPI service that orchestrates:
• Ingestion/ETL: chunk documents, extract entities, build a knowledge graph, and index content
• Hybrid retrieval: sparse + dense + graph retrieval with fusion and reranking
• Agentic reasoning: query understanding, decomposition, memory, and controlled LLM generation
• Persistence layer: Postgres/pgvector, Redis, Neo4j (per README + docker-compose)
Offline/async ingestion + indexing
2) Online query-time serving (API → agent → tri-engine retrieval → rerank → grounded generation)
Serving / API layer
• api/main.py, api/schemas.py: FastAPI endpoints + request/response models
Agent & orchestration
• agent/copilot_agent.py: the reasoning loop / tool-using agent that calls retrieval, memory, and LLMs
Core shared services
• core/memory.py: Redis-backed conversation/session memory (context memory)
• core/db_init.py: DB initialization wiring (Postgres/Neo4j/etc)
Ingestion pipeline
• ingestion/chunking/semantic_chunker.py: chunking strategy (sliding window / semantic)
• ingestion/graph_build/kg_builder.py: entity linking / KG build and Neo4j population
Retrieval system
• retrieval/hybrid_search.py: orchestrator for “tri-engine fusion”
• retrieval/sparse/keyword_search.py: BM25/Elasticsearch-like keyword retrieval
• retrieval/dense/vector_search.py: FAISS/PGVector-like dense retrieval
• retrieval/reranker/*: reranking + fusion methods
• cross_encoder.py: deep reranker
• rrf_fusion.py: reciprocal rank fusion / blending
Infra / runtime environment
• docker-compose.yml: local Postgres+pgvector, Redis, Neo4j
• .env.example: configuration surfaces (API keys, DB creds)
Client → Serving layer
• Clients (browser/internal app/CLI) calling REST endpoints
• FastAPI Service (from api/): request validation (schemas.py), streaming responses (mentioned in README)
Orchestration / reasoning layer (core runtime)
• Copilot Agent (agent/copilot_agent.py) containing:
• Query understanding (rewrite / decomposition)
• Tool selection (calls hybrid retrieval, graph traversal)
• Controlled LLM generation (grounded outputs + citations)
• Memory service (core/memory.py) backed by Redis
• Read/write of conversation state, threads, session context
• Configuration + DB init (core/db_init.py) as the wiring to Postgres/Neo4j/Redis
• LLM providers as *external services*
• OpenAI/Anthropic/local models as outbound calls from agent
Retrieval layer (tri-engine)
Model as three retrieval “engines” that the orchestrator queries in parallel:
Sparse Retrieval (retrieval/sparse/keyword_search.py)
• backend: Elasticsearch/BM25 (README). Even if ES isn’t in compose, show it as a logical dependency.
Dense Retrieval (retrieval/dense/vector_search.py)
• backend: Postgres + pgvector and/or FAISS (README)
• An “Embedding model” dependency for indexing + query embeddings (BGE/E5/OpenAI embeddings)
Graph Retrieval / KG Reasoning
• backed by Neo4j
• query uses entity linking / traversal logic (partly in ingestion + possibly agent-time)
• Hybrid Search Coordinator (retrieval/hybrid_search.py) aggregating results
• Fusion (retrieval/reranker/rrf_fusion.py)
• Reranker (retrieval/reranker/cross_encoder.py) to reorder and filter
Storage / persistence layer
• PostgreSQL + pgvector: document metadata, embeddings, chunks, maybe audit logs
• Neo4j: entities/relations + SOP/medical/legal knowledge constraints
• Redis: session memory / state / caching
Ingestion / indexing pipeline (offline)
A separate “Ingestion Worker / Job” (even if it currently runs in-process) that does:
• Input sources: enterprise docs, PDFs, SOP pages, tables, images (README says multimodal)
• Chunking (semantic_chunker.py)
• Embedding generation (external embedding model API)
• Write vectors + chunks → Postgres/pgvector (and/or FAISS)
• KG Builder (kg_builder.py) → Neo4j population
• (Optional) Keyword index build → Elasticsearch/BM25 index
This ingestion side is crucial: it explains where the retrieval engines get their indexes.
Hybrid retrieval fans out
• → Sparse search (BM25/ES)
• → Dense search (pgvector/FAISS) + embedding model call for query vector
• → Graph retrieval (Neo4j traversal / constraints)
(Optionally) store doc registry/metadata in Postgres
• Orchestrator pattern: FastAPI + agent coordinates multiple retrieval subsystems
• Polyglot persistence: Postgres/pgvector + Redis + Neo4j (+ optionally Elasticsearch)
• Two-plane architecture: ingestion/indexing plane vs serving/query plane
• Evidence-grounded generation: LLM is downstream of retrieval + KG constraints, not a standalone responder
• Fusion + reranking stage: explicitly show “candidate generation” then “scoring/ranking” before final answer
Shows: Clients → NovaSearch API Service → (Redis, Postgres/pgvector, Neo4j, Elasticsearch) + External LLMs.
Shows internal modules: api/*, agent/*, retrieval/*, core/* and their calls.
• Query-time DFD (fan-out retrieval + reranking + generation)
• Ingestion-time DFD (chunk → embed → index; entity link → Neo4j)
This set will be the most “accurate” representation of both the code boundaries and the runtime architecture implied by Docker/README.