How idea-finai/tog-2 works
Project Type and Purpose
• This is a research‐oriented full‐stack Python application that performs knowledge‐guided, retrieval‐augmented LLM reasoning (Think‐on‐Graph 2.0).
• It consists of three logical layers:
a. Data and knowledge‐graph setup (Wikidata submodule + datasets)
b. Retrieval + reasoning pipeline (ToG-2 folder)
c. Evaluation suite (eval folder)
Top‐Level Components
• CLI Orchestrator (main_tog2.py)
• Named Entity Recognition (ner.py) via Azure API
• Retrieval Engine (search.py) using embedding models & BM25
• Graph Client (client.py + server_urls.txt) communicating with a local Wikidata HTTP service
• Prompt Library (prompt_list.py) for exploration & reasoning stages
• LLM Interface (OpenAI or local LLM calls within utils.py)
• Result Serializer (search.save_2_jsonl_simplier → JSONL output)
• Evaluation Script (eval/eval.py + utils.py) computing Exact Match
• Knowledge‐Graph Stack (Wikidata/simple_wikidata_db):
• Preprocessing stage (preprocess_dump.py + worker/writer/reader)
• Index building (db_deploy/build_index.py via shell scripts)
• Local HTTP server (db_deploy/server.py)
Relationships & Data Flow
a. User invokes main_tog2.py with CLI flags →
b. NER Module: sends text to Azure, tags entities →
c. Retrieval Module: for each entity/query, calls embedding/BM25 search →
d. Graph Client: for retrieval results, issues HTTP queries to the local Wikidata service →
e. Search Graph: builds exploration trees to given depth/width, prunes relations →
f. Prompt Generator: builds prompts from prompt_list.py →
g. LLM Module: sends prompts to OpenAI/gpt‐API or local LLM →
h. Aggregator: collects reasoning outputs, applies self-consistency threshold →
i. Serializer: writes predictions to JSONL →
j. Eval Script: consumes JSONL + ground‐truth data → calculates Exact Match.
Architectural Patterns & Key Technologies
• Modular pipeline: clear separation of concerns for NER, retrieval, graph search, prompting, LLM calls, and evaluation.
• Microservice for knowledge graph: local HTTP service exposing KG index.
• Asynchronous Preprocessing: reader, worker, writer processes for dump preprocessing.
• Shell‐scripted orchestration (build_index.sh, start_server.sh).
• Core libraries & frameworks: Python 3, OpenAI API, Azure Cognitive Services, embedding models (bge, minilm, colbert), BM25, JSON.