How internscience/graphgen works
1. What this repo is
GraphGen is a configurable synthetic-data generation framework centered on knowledge graphs. The core product is a pipeline that ingests source data, extracts structured knowledge, builds a graph, uses that graph to generate QA-style training data, and optionally evaluates the result.
2. Top-level execution paths
There are two main entry points:
- graphgen/run.py and the package CLI for batch pipeline execution.
- webui/app.py for the Gradio-based demo UI.
Both ultimately drive the same underlying pipeline and model components; the web UI is a thin front end over the same generation system.
3. Pipeline orchestration layer
The main orchestration lives in graphgen/engine.py and the graphgen/operators/* services. This layer reads YAML configs, resolves node dependencies, and executes a DAG-like workflow. Operators are separated by task stage: read, preprocess, chunk, extract, partition, build KG, search, generate, rephrase, filter, evaluate, and quiz/judge-related steps.
4. Core architectural boundary: abstract bases vs concrete models
graphgen/bases/* defines the repository’s key contracts: reader, splitter, tokenizer, searcher, extractor, generator, partitioner, evaluator, storage, LLM wrapper, and operator abstractions. graphgen/models/* contains the concrete implementations plugged into those interfaces. This separation is the main extensibility boundary in the repo.
5. Input ingestion and preprocessing
Source data enters through the reader subsystem in graphgen/models/reader/*, which supports text, CSV, JSON/JSONL, PDF, parquet, pickle, RDF, and Hugging Face datasets. Text is then normalized and chunked via graphgen/models/splitter/*, with tokenization support in graphgen/models/tokenizer/*. There is also a preprocessing step for tasks like coreference resolution.
6. Knowledge construction path
The extraction layer (graphgen/models/extractor/*) converts source chunks into structured knowledge. That output is used by the KG builders in graphgen/models/kg_builder/* and the KG templates in graphgen/templates/kg/*. The repo supports both text-centric and multimodal KG building, with graph persistence abstracted behind graphgen/storage/graph/* using networkx or kuzu.
7. Graph partitioning and sampling
Once a graph exists, graphgen/models/partitioner/* slices it into useful subgraphs or neighborhoods for downstream generation. The available strategies include BFS/DFS variants, Leiden-based community detection, anchor-based BFS, ECE, triple, and quintuple partitioning. This is important because the chosen partitioning determines the granularity and reasoning scope of generated data.
8. LLM access layer
Model interaction is isolated in graphgen/models/llm/*. It supports API-based backends (openai, ollama, generic HTTP, Azure/OpenAI-style endpoints) and local inference wrappers (transformers, vLLM, TGI, SGLang, TensorRT). graphgen/common/init_llm.py wires the configured synthesizer and trainee models, and the repo’s environment variables make this a primary deployment boundary.
9. Generation, rephrasing, and question shaping
The generation subsystem (graphgen/models/generator/*) is one of the core outputs of the repo. It includes atomic, aggregated, CoT, multi-hop, multi-answer, multi-choice, fill-in-the-blank, true/false, VQA, and quiz-oriented generators. Rephrasing is handled separately in graphgen/models/rephraser/*, which is used to diversify outputs without changing underlying facts. The prompt logic for each format lives in graphgen/templates/generation/* and graphgen/templates/rephrasing/*.
10. Search and external knowledge enrichment
graphgen/models/searcher/* adds retrieval from external sources. It supports:
- Web search: Google and Bing
- Knowledge graph search: Wikipedia
- Domain databases: UniProt, NCBI, RNAcentral, InterPro
This makes GraphGen not just a local extraction pipeline but also a knowledge augmentation system.
11. Evaluation and quality control
Evaluation is a separate subsystem in graphgen/models/evaluator/* and graphgen/operators/evaluate/*. It includes QA-level metrics like length, MTLD, reward, and unified evaluation, plus KG-oriented structure and triple accuracy evaluation. There is also filtering in graphgen/models/filter/* and graphgen/operators/filter/* to enforce output quality before data export.
12. Persistence and intermediate state
The repository uses two storage axes:
- Graph storage: networkx or kuzu
- Key-value storage: JSON or RocksDB
These stores let the pipeline persist intermediate artifacts, cache large generation runs, and support distributed or multi-step workflows. That persistence boundary is especially important because many operators depend on earlier graph state rather than raw source files.
13. Templates as the prompt/logic layer
graphgen/templates/* holds the prompt templates and schema definitions that shape the behavior of extraction, generation, search judgment, statement judgment, KG summarization, and style-controlled rephrasing. In architecture terms, this is where domain-specific prompting is separated from execution logic.
14. Repo support files and examples
examples/* provides runnable end-to-end configs and shell scripts for the major workflows: generation, extraction, search, filtering, evaluation, and baseline reproduction. baselines/* contains alternative methods for comparison, and tests/* covers end-to-end and integration behavior across readers, splitters, partitioners, LLM wrappers, and generation/evaluation flows. This makes the repository both a framework and a reproducible benchmark suite.
15. Operational/deployment surfaces
The repo includes a Dockerfile, CI workflows, and Hugging Face / ModelScope sync automation. In practice the system can run locally, in a container, or as a demo service, but the core architecture remains config-driven and backend-agnostic.
Main entry points: engine.py Engine · read.py Read ops · extract_service.py Extract ops · build_kg_service.py KG build
Components 29
Entry points
Batch CLI and demo UI surfaces that invoke the same pipeline.
Orchestration
Config-driven workflow engine and operator services.
- Engine · Loads configs and coordinates workflow execution.
graphgen/engine.py - Read ops · Input scanning and dataset ingestion services.
graphgen/operators/read/read.py - Preprocess · Text normalization and coreference resolution.
graphgen/operators/preprocess/resolute_coreference.py - Chunk ops · Splits and token-aware chunking workflow.
graphgen/operators/chunk/chunk_service.py - Extract ops · Structured information extraction services.
graphgen/operators/extract/extract_service.py - KG build · Builds and persists knowledge graphs from extracted content.
graphgen/operators/build_kg/build_kg_service.py - Partition ops · Slices graphs into subgraphs for downstream tasks.
graphgen/operators/partition/partition_service.py - Search ops · Retrieval and external knowledge enrichment.
graphgen/operators/search/search_service.py - Generate ops · QA-style synthetic data generation workflows.
graphgen/operators/generate/generate_service.py - Rephrase ops · Style-controlled rewriting and diversification.
graphgen/operators/rephrase/rephrase_service.py - Filter ops · Quality checks and output filtering.
graphgen/operators/filter/filter_service.py - Evaluate ops · Evaluation workflows for QA, KG, and triples.
graphgen/operators/evaluate/evaluate_service.py - Judge ops · Judgement logic for search and statement validation.
graphgen/operators/judge/judge_service.py - Quiz ops · Quiz-oriented generation and scoring flow.
graphgen/operators/quiz/quiz_service.py
Contracts
Abstract bases that define the extensibility boundary.
- Base APIs · Repository-wide interfaces for readers, splitters, LLMs, graph storage, generation, and evaluation.
graphgen/bases
Models
Concrete implementations plugged into the framework interfaces.
- LLM layer · API and local model wrappers for generation-time inference.
graphgen/models/llm - Ingest models · Readers, splitters, tokenizers, and preprocessing helpers.
graphgen/models/reader - KG models · Extraction and knowledge-graph construction implementations.
graphgen/models/extractor - Partition models · Graph partitioning strategies for downstream subgraph selection.
graphgen/models/partitioner - Generator models · Atomic, multi-hop, VQA, quiz, and other QA generators.
graphgen/models/generator - Search models · Web, KG, and domain database search backends.
graphgen/models/searcher - Eval models · Quality and accuracy evaluators for QA, KG, and triples.
graphgen/models/evaluator
Storage
Persistent state for graphs and key-value artifacts.
- Storage backends · Graph and key-value persistence implementations.
graphgen/storage
Templates
Prompt and schema layer shaping extraction, generation, and evaluation.
- KG templates · Templates for extraction, KG summarization, and multimodal KG prompts.
graphgen/templates/kg - Gen templates · Question and answer generation prompt templates.
graphgen/templates/generation - Rephrase templates · Style-controlled rephrasing prompt variants.
graphgen/templates/rephrasing - Eval templates · Templates for KG accuracy and judgment flows.
graphgen/templates/evaluation
Connections 34
- CLI run → Engine: executes · CLI starts config-driven workflow execution.
- Web UI → Engine: invokes · UI triggers the same pipeline through a thin front end.
- Engine → Base APIs: binds · Engine resolves components through abstract interfaces.
- Engine → Read ops: orchestrates · Workflow begins with dataset ingestion.
- Read ops → Ingest models: uses · Readers, splitters, tokenizers, and preprocessors handle inputs.
- Read ops → Storage backends: persists · Input artifacts and intermediate state can be cached.
- Engine → Preprocess: orders · Preprocessing is a distinct pipeline stage.
- Preprocess → Ingest models: uses · Normalization and coreference operate on ingested text.
- Engine → Chunk ops: orders · Chunks are created before extraction and generation.
- Chunk ops → Ingest models: uses · Chunking depends on splitters and tokenizers.
- Engine → Extract ops: orchestrates · Structured extraction is a main pipeline stage.
- Extract ops → KG templates: prompts · Extraction behavior is shaped by prompt schemas.
- Extract ops → KG models: produces · Extracted knowledge feeds KG construction.
- Engine → KG build: orchestrates · Knowledge graph building is executed after extraction.
- KG build → KG models: uses · Concrete KG builders assemble text and multimodal graphs.
- KG build → Storage backends: writes · Built graphs are persisted for later stages.
- Engine → Partition ops: orchestrates · Graph partitioning prepares subgraphs for generation.
- Partition ops → Partition models: uses · BFS, DFS, Leiden, and other strategies select graph slices.
- Engine → Search ops: orchestrates · Retrieval can enrich or validate generation inputs.
- Search ops → Search models: uses · Web, wiki, and domain search backends provide evidence.
- Engine → Generate ops: orchestrates · Generation is the central synthetic-data output stage.
- Generate ops → LLM layer: calls · Generators rely on configured LLM backends.
- Generate ops → Gen templates: prompts · Question formats are defined by generation templates.
- Engine → Rephrase ops: orchestrates · Optional rewriting diversifies outputs.
- Rephrase ops → LLM layer: calls · Rephrasing also uses the LLM access layer.
- Rephrase ops → Rephrase templates: prompts · Style and tone variants come from prompt templates.
- Engine → Filter ops: orchestrates · Filtering removes low-quality outputs before export.
- Engine → Evaluate ops: orchestrates · Evaluation is a separate pipeline branch.
- Evaluate ops → Eval models: uses · Metric implementations score QA, KG, and triple outputs.
- Evaluate ops → Eval templates: prompts · Judgment and accuracy checks use evaluation prompts.
- Engine → Judge ops: supports · Judgement services validate statements and search results.
- Judge ops → LLM layer: calls · Judgement is typically LLM-backed.
- Engine → Quiz ops: supports · Quiz-specific workflows extend generation.
- Quiz ops → Gen templates: prompts · Quiz generation reuses question templates.