bhavy-ranka/graphitti

Purpose and Entry Points

Graphitti builds a temporary, graph-native knowledge base from a crawled website, then answers questions using routed text and graph retrieval. The primary backend entry point is graphitti/api/app.py; interactive use is through graphitti/streamlit_app.py. The API serves an interactive graph view from graphitti/api/static/graph.html.

Ingestion Control Flow

An ingest request orchestrates crawl-to-graph construction. By default it resets prior graph and text-index state, enforcing a single active crawled corpus. The central workflow is implemented in graphitti/orchestration/orchestrator.py, using crawl output from graphitti/crawling/crawler.py and graph loading in graphitti/graph/loader.py.

Crawl and Local Extraction

The crawler performs breadth-first browser crawling with Playwright in graphitti/crawling/crawler.py. Page text is segmented into sentence-aware chunks by graphitti/extraction/chunking.py. graphitti/extraction/nlp_pipeline.py and graphitti/extraction/triple_extractor.py derive entity-relation-entity triples locally with NLTK POS tagging, NER, grammar rules, SVO/passive handling, pronoun heuristics, and confidence scoring; extraction does not call an LLM.

Entity and Graph Quality Boundary

Extracted triples use typed schemas in graphitti/extraction/schemas.py, then pass through direction normalization and alias/entity merging in graphitti/extraction/direction.py and graphitti/extraction/entity_resolution.py. Graph conflict detection marks contested relationships before persistence in graphitti/graph/consistency.py. This separates noisy web-text interpretation from graph storage and retrieval.

Graph and Text Persistence

graphitti/graph/store.py provides Neo4j and in-memory graph-store implementations. graphitti/graph/loader.py batch-loads nodes and edges through Neo4j UNWIND/MERGE operations with rollback behavior. Raw chunks remain addressable in graphitti/retrieval/text_index.py for textual evidence and citations. Neo4j is the material external persistence dependency; Playwright/Chromium is required for crawling.

Query Routing and Retrieval

Questions enter intent classification and decomposition chains in graphitti/routing/chains.py, with strategy selection and confidence fallback in graphitti/routing/query_router.py. Low-confidence routing selects hybrid retrieval. Available evidence sources are BM25 (graphitti/retrieval/bm25.py), dense vector search with Sentence-Transformers or TF-IDF fallback (graphitti/retrieval/dense.py), and entity/multi-hop graph traversal (graphitti/retrieval/graph_retrieval.py). graphitti/retrieval/base.py registers strategies and combines rankings with reciprocal-rank fusion.

Answer Orchestration and Presentation

A LangGraph state machine coordinates routing, retrieval, and final synthesis in graphitti/orchestration/orchestrator.py. graphitti/orchestration/synthesis_chain.py produces provenance-backed answers from retrieved graph/text evidence. FastAPI exposes ingest, query, and graph-facing operations in graphitti/api/app.py; Streamlit drives those APIs and refreshes the visualized graph after each reset-and-ingest cycle.