# internscience/graphgen architecture

> Architecture diagram of the GitHub repository internscience/graphgen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internscience/graphgen
- Repository: https://github.com/internscience/graphgen

## Overview

1. **What this repo is**
   GraphGen is a configurable synthetic-data generation framework centered on knowledge graphs. The core product is a pipeline that ingests source data, extracts structured knowledge, builds a graph, uses that graph to generate QA-style training data, and optionally evaluates the result.

2. **Top-level execution paths**
   There are two main entry points:
   - `graphgen/run.py` and the package CLI for batch pipeline execution.
   - `webui/app.py` for the Gradio-based demo UI.
   Both ultimately drive the same underlying pipeline and model components; the web UI is a thin front end over the same generation system.

3. **Pipeline orchestration layer**
   The main orchestration lives in `graphgen/engine.py` and the `graphgen/operators/*` services. This layer reads YAML configs, resolves node dependencies, and executes a DAG-like workflow. Operators are separated by task stage: read, preprocess, chunk, extract, partition, build KG, search, generate, rephrase, filter, evaluate, and quiz/judge-related steps.

4. **Core architectural boundary: abstract bases vs concrete models**
   `graphgen/bases/*` defines the repository’s key contracts: reader, splitter, tokenizer, searcher, extractor, generator, partitioner, evaluator, storage, LLM wrapper, and operator abstractions. `graphgen/models/*` contains the concrete implementations plugged into those interfaces. This separation is the main extensibility boundary in the repo.

5. **Input ingestion and preprocessing**
   Source data enters through the reader subsystem in `graphgen/models/reader/*`, which supports text, CSV, JSON/JSONL, PDF, parquet, pickle, RDF, and Hugging Face datasets. Text is then normalized and chunked via `graphgen/models/splitter/*`, with tokenization support in `graphgen/models/tokenizer/*`. There is also a preprocessing step for tasks like coreference resolution.

6. **Knowledge construction path**
   The extraction layer (`graphgen/models/extractor/*`) converts source chunks into structured knowledge. That output is used by the KG builders in `graphgen/models/kg_builder/*` and the KG templates in `graphgen/templates/kg/*`. The repo supports both text-centric and multimodal KG building, with graph persistence abstracted behind `graphgen/storage/graph/*` using `networkx` or `kuzu`.

7. **Graph partitioning and sampling**
   Once a graph exists, `graphgen/models/partitioner/*` slices it into useful subgraphs or neighborhoods for downstream generation. The available strategies include BFS/DFS variants, Leiden-based community detection, anchor-based BFS, ECE, triple, and quintuple partitioning. This is important because the chosen partitioning determines the granularity and reasoning scope of generated data.

8. **LLM access layer**
   Model interaction is isolated in `graphgen/models/llm/*`. It supports API-based backends (`openai`, `ollama`, generic HTTP, Azure/OpenAI-style endpoints) and local inference wrappers (`transformers`, `vLLM`, `TGI`, `SGLang`, TensorRT). `graphgen/common/init_llm.py` wires the configured synthesizer and trainee models, and the repo’s environment variables make this a primary deployment boundary.

9. **Generation, rephrasing, and question shaping**
   The generation subsystem (`graphgen/models/generator/*`) is one of the core outputs of the repo. It includes atomic, aggregated, CoT, multi-hop, multi-answer, multi-choice, fill-in-the-blank, true/false, VQA, and quiz-oriented generators. Rephrasing is handled separately in `graphgen/models/rephraser/*`, which is used to diversify outputs without changing underlying facts. The prompt logic for each format lives in `graphgen/templates/generation/*` and `graphgen/templates/rephrasing/*`.

10. **Search and external knowledge enrichment**
    `graphgen/models/searcher/*` adds retrieval from external sources. It supports:
    - Web search: Google and Bing
    - Knowledge graph search: Wikipedia
    - Domain databases: UniProt, NCBI, RNAcentral, InterPro
    This makes GraphGen not just a local extraction pipeline but also a knowledge augmentation system.

11. **Evaluation and quality control**
    Evaluation is a separate subsystem in `graphgen/models/evaluator/*` and `graphgen/operators/evaluate/*`. It includes QA-level metrics like length, MTLD, reward, and unified evaluation, plus KG-oriented structure and triple accuracy evaluation. There is also filtering in `graphgen/models/filter/*` and `graphgen/operators/filter/*` to enforce output quality before data export.

12. **Persistence and intermediate state**
    The repository uses two storage axes:
    - Graph storage: `networkx` or `kuzu`
    - Key-value storage: JSON or RocksDB
    These stores let the pipeline persist intermediate artifacts, cache large generation runs, and support distributed or multi-step workflows. That persistence boundary is especially important because many operators depend on earlier graph state rather than raw source files.

13. **Templates as the prompt/logic layer**
    `graphgen/templates/*` holds the prompt templates and schema definitions that shape the behavior of extraction, generation, search judgment, statement judgment, KG summarization, and style-controlled rephrasing. In architecture terms, this is where domain-specific prompting is separated from execution logic.

14. **Repo support files and examples**
    `examples/*` provides runnable end-to-end configs and shell scripts for the major workflows: generation, extraction, search, filtering, evaluation, and baseline reproduction. `baselines/*` contains alternative methods for comparison, and `tests/*` covers end-to-end and integration behavior across readers, splitters, partitioners, LLM wrappers, and generation/evaluation flows. This makes the repository both a framework and a reproducible benchmark suite.

15. **Operational/deployment surfaces**
    The repo includes a `Dockerfile`, CI workflows, and Hugging Face / ModelScope sync automation. In practice the system can run locally, in a container, or as a demo service, but the core architecture remains config-driven and backend-agnostic.

Main entry points: [`graphgen/engine.py`](https://github.com/internscience/graphgen/blob/main/graphgen/engine.py) (Engine), [`graphgen/operators/read/read.py`](https://github.com/internscience/graphgen/blob/main/graphgen/operators/read/read.py) (Read ops), [`graphgen/operators/extract/extract_service.py`](https://github.com/internscience/graphgen/blob/main/graphgen/operators/extract/extract_service.py) (Extract ops), [`graphgen/operators/build_kg/build_kg_service.py`](https://github.com/internscience/graphgen/blob/main/graphgen/operators/build_kg/build_kg_service.py) (KG build)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entry points"]
  node_run_cli["CLI run<br/>entrypoint<br/>[run.py]"]
  node_web_ui["Web UI<br/>entrypoint<br/>[app.py]"]
end

subgraph group_orchestration["Orchestration"]
  node_engine["Engine<br/>orchestrator<br/>[engine.py]"]
  node_read_ops["Read ops<br/>operator suite<br/>[read.py]"]
  node_prep_ops["Preprocess<br/>operator"]
  node_chunk_ops["Chunk ops<br/>operator<br/>[chunk_service.py]"]
  node_extract_ops["Extract ops<br/>operator suite<br/>[extract_service.py]"]
  node_build_ops["KG build<br/>operator suite"]
  node_partition_ops["Partition ops<br/>operator"]
  node_search_ops["Search ops<br/>operator<br/>[search_service.py]"]
  node_generate_ops["Generate ops<br/>operator suite"]
  node_rephrase_ops["Rephrase ops<br/>operator"]
  node_filter_ops["Filter ops<br/>operator<br/>[filter_service.py]"]
  node_evaluate_ops["Evaluate ops<br/>operator suite"]
  node_judge_ops["Judge ops<br/>operator<br/>[judge_service.py]"]
  node_quiz_ops["Quiz ops<br/>operator<br/>[quiz_service.py]"]
end

subgraph group_contracts["Contracts"]
  node_base_contracts["Base APIs<br/>abstractions"]
end

subgraph group_models["Models"]
  node_llm_models["LLM layer<br/>model adapters"]
  node_ingest_models["Ingest models<br/>input adapters"]
  node_kg_models["KG models<br/>graph builders"]
  node_partition_models["Partition models<br/>graph sampling"]
  node_gen_models["Generator models<br/>synthetic QA"]
  node_search_models["Search models<br/>retrieval adapters"]
  node_eval_models["Eval models<br/>metrics"]
end

subgraph group_storage["Storage"]
  node_storage_models[("Storage backends<br/>persistence")]
end

subgraph group_templates["Templates"]
  node_kg_templates["KG templates<br/>prompt schemas"]
  node_gen_templates["Gen templates<br/>prompt schemas"]
  node_rephrase_templates["Rephrase templates<br/>prompt schemas"]
  node_eval_templates["Eval templates<br/>prompt schemas"]
end

node_run_cli -->|"executes"| node_engine
node_web_ui -->|"invokes"| node_engine
node_engine -->|"binds"| node_base_contracts
node_engine -->|"orchestrates"| node_read_ops
node_read_ops -->|"uses"| node_ingest_models
node_read_ops -.->|"persists"| node_storage_models
node_engine -->|"orders"| node_prep_ops
node_prep_ops -->|"uses"| node_ingest_models
node_engine -->|"orders"| node_chunk_ops
node_chunk_ops -->|"uses"| node_ingest_models
node_engine -->|"orchestrates"| node_extract_ops
node_extract_ops -->|"prompts"| node_kg_templates
node_extract_ops -->|"produces"| node_kg_models
node_engine -->|"orchestrates"| node_build_ops
node_build_ops -->|"uses"| node_kg_models
node_build_ops -->|"writes"| node_storage_models
node_engine -->|"orchestrates"| node_partition_ops
node_partition_ops -->|"uses"| node_partition_models
node_engine -->|"orchestrates"| node_search_ops
node_search_ops -->|"uses"| node_search_models
node_engine -->|"orchestrates"| node_generate_ops
node_generate_ops -->|"calls"| node_llm_models
node_generate_ops -->|"prompts"| node_gen_templates
node_engine -->|"orchestrates"| node_rephrase_ops
node_rephrase_ops -->|"calls"| node_llm_models
node_rephrase_ops -->|"prompts"| node_rephrase_templates
node_engine -->|"orchestrates"| node_filter_ops
node_engine -->|"orchestrates"| node_evaluate_ops
node_evaluate_ops -->|"uses"| node_eval_models
node_evaluate_ops -->|"prompts"| node_eval_templates
node_engine -.->|"supports"| node_judge_ops
node_judge_ops -->|"calls"| node_llm_models
node_engine -.->|"supports"| node_quiz_ops
node_quiz_ops -->|"prompts"| node_gen_templates

click node_run_cli "https://github.com/internscience/graphgen/blob/main/graphgen/run.py"
click node_web_ui "https://github.com/internscience/graphgen/blob/main/webui/app.py"
click node_engine "https://github.com/internscience/graphgen/blob/main/graphgen/engine.py"
click node_read_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/read/read.py"
click node_prep_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/preprocess/resolute_coreference.py"
click node_chunk_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/chunk/chunk_service.py"
click node_extract_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/extract/extract_service.py"
click node_build_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/build_kg/build_kg_service.py"
click node_partition_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/partition/partition_service.py"
click node_search_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/search/search_service.py"
click node_generate_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/generate/generate_service.py"
click node_rephrase_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/rephrase/rephrase_service.py"
click node_filter_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/filter/filter_service.py"
click node_evaluate_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/evaluate/evaluate_service.py"
click node_judge_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/judge/judge_service.py"
click node_quiz_ops "https://github.com/internscience/graphgen/blob/main/graphgen/operators/quiz/quiz_service.py"
click node_base_contracts "https://github.com/internscience/graphgen/tree/main/graphgen/bases"
click node_llm_models "https://github.com/internscience/graphgen/tree/main/graphgen/models/llm"
click node_ingest_models "https://github.com/internscience/graphgen/tree/main/graphgen/models/reader"
click node_kg_models "https://github.com/internscience/graphgen/tree/main/graphgen/models/extractor"
click node_partition_models "https://github.com/internscience/graphgen/tree/main/graphgen/models/partitioner"
click node_gen_models "https://github.com/internscience/graphgen/tree/main/graphgen/models/generator"
click node_search_models "https://github.com/internscience/graphgen/tree/main/graphgen/models/searcher"
click node_eval_models "https://github.com/internscience/graphgen/tree/main/graphgen/models/evaluator"
click node_storage_models "https://github.com/internscience/graphgen/tree/main/graphgen/storage"
click node_kg_templates "https://github.com/internscience/graphgen/tree/main/graphgen/templates/kg"
click node_gen_templates "https://github.com/internscience/graphgen/tree/main/graphgen/templates/generation"
click node_rephrase_templates "https://github.com/internscience/graphgen/tree/main/graphgen/templates/rephrasing"
click node_eval_templates "https://github.com/internscience/graphgen/tree/main/graphgen/templates/evaluation"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_run_cli,node_web_ui toneBlue
class node_engine,node_read_ops,node_prep_ops,node_chunk_ops,node_extract_ops,node_build_ops,node_partition_ops,node_search_ops,node_generate_ops,node_rephrase_ops,node_filter_ops,node_evaluate_ops,node_judge_ops,node_quiz_ops toneAmber
class node_base_contracts toneMint
class node_llm_models,node_ingest_models,node_kg_models,node_partition_models,node_gen_models,node_search_models,node_eval_models toneRose
class node_storage_models toneIndigo
class node_kg_templates,node_gen_templates,node_rephrase_templates,node_eval_templates toneTeal
```

## Components

### Entry points

Batch CLI and demo UI surfaces that invoke the same pipeline.

- [CLI run](https://github.com/internscience/graphgen/blob/main/graphgen/run.py): entrypoint · Batch execution path for pipeline runs. · `graphgen/run.py`
- [Web UI](https://github.com/internscience/graphgen/blob/main/webui/app.py): entrypoint · Gradio-based front end over the same generation system. · `webui/app.py`

### Orchestration

Config-driven workflow engine and operator services.

- [Engine](https://github.com/internscience/graphgen/blob/main/graphgen/engine.py): orchestrator · Loads configs and coordinates workflow execution. · `graphgen/engine.py`
- [Read ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/read/read.py): operator suite · Input scanning and dataset ingestion services. · `graphgen/operators/read/read.py`
- [Preprocess](https://github.com/internscience/graphgen/blob/main/graphgen/operators/preprocess/resolute_coreference.py): operator · Text normalization and coreference resolution. · `graphgen/operators/preprocess/resolute_coreference.py`
- [Chunk ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/chunk/chunk_service.py): operator · Splits and token-aware chunking workflow. · `graphgen/operators/chunk/chunk_service.py`
- [Extract ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/extract/extract_service.py): operator suite · Structured information extraction services. · `graphgen/operators/extract/extract_service.py`
- [KG build](https://github.com/internscience/graphgen/blob/main/graphgen/operators/build_kg/build_kg_service.py): operator suite · Builds and persists knowledge graphs from extracted content. · `graphgen/operators/build_kg/build_kg_service.py`
- [Partition ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/partition/partition_service.py): operator · Slices graphs into subgraphs for downstream tasks. · `graphgen/operators/partition/partition_service.py`
- [Search ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/search/search_service.py): operator · Retrieval and external knowledge enrichment. · `graphgen/operators/search/search_service.py`
- [Generate ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/generate/generate_service.py): operator suite · QA-style synthetic data generation workflows. · `graphgen/operators/generate/generate_service.py`
- [Rephrase ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/rephrase/rephrase_service.py): operator · Style-controlled rewriting and diversification. · `graphgen/operators/rephrase/rephrase_service.py`
- [Filter ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/filter/filter_service.py): operator · Quality checks and output filtering. · `graphgen/operators/filter/filter_service.py`
- [Evaluate ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/evaluate/evaluate_service.py): operator suite · Evaluation workflows for QA, KG, and triples. · `graphgen/operators/evaluate/evaluate_service.py`
- [Judge ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/judge/judge_service.py): operator · Judgement logic for search and statement validation. · `graphgen/operators/judge/judge_service.py`
- [Quiz ops](https://github.com/internscience/graphgen/blob/main/graphgen/operators/quiz/quiz_service.py): operator · Quiz-oriented generation and scoring flow. · `graphgen/operators/quiz/quiz_service.py`

### Contracts

Abstract bases that define the extensibility boundary.

- [Base APIs](https://github.com/internscience/graphgen/tree/main/graphgen/bases): abstractions · Repository-wide interfaces for readers, splitters, LLMs, graph storage, generation, and evaluation. · `graphgen/bases`

### Models

Concrete implementations plugged into the framework interfaces.

- [LLM layer](https://github.com/internscience/graphgen/tree/main/graphgen/models/llm): model adapters · API and local model wrappers for generation-time inference. · `graphgen/models/llm`
- [Ingest models](https://github.com/internscience/graphgen/tree/main/graphgen/models/reader): input adapters · Readers, splitters, tokenizers, and preprocessing helpers. · `graphgen/models/reader`
- [KG models](https://github.com/internscience/graphgen/tree/main/graphgen/models/extractor): graph builders · Extraction and knowledge-graph construction implementations. · `graphgen/models/extractor`
- [Partition models](https://github.com/internscience/graphgen/tree/main/graphgen/models/partitioner): graph sampling · Graph partitioning strategies for downstream subgraph selection. · `graphgen/models/partitioner`
- [Generator models](https://github.com/internscience/graphgen/tree/main/graphgen/models/generator): synthetic QA · Atomic, multi-hop, VQA, quiz, and other QA generators. · `graphgen/models/generator`
- [Search models](https://github.com/internscience/graphgen/tree/main/graphgen/models/searcher): retrieval adapters · Web, KG, and domain database search backends. · `graphgen/models/searcher`
- [Eval models](https://github.com/internscience/graphgen/tree/main/graphgen/models/evaluator): metrics · Quality and accuracy evaluators for QA, KG, and triples. · `graphgen/models/evaluator`

### Storage

Persistent state for graphs and key-value artifacts.

- [Storage backends](https://github.com/internscience/graphgen/tree/main/graphgen/storage): persistence · Graph and key-value persistence implementations. · `graphgen/storage`

### Templates

Prompt and schema layer shaping extraction, generation, and evaluation.

- [KG templates](https://github.com/internscience/graphgen/tree/main/graphgen/templates/kg): prompt schemas · Templates for extraction, KG summarization, and multimodal KG prompts. · `graphgen/templates/kg`
- [Gen templates](https://github.com/internscience/graphgen/tree/main/graphgen/templates/generation): prompt schemas · Question and answer generation prompt templates. · `graphgen/templates/generation`
- [Rephrase templates](https://github.com/internscience/graphgen/tree/main/graphgen/templates/rephrasing): prompt schemas · Style-controlled rephrasing prompt variants. · `graphgen/templates/rephrasing`
- [Eval templates](https://github.com/internscience/graphgen/tree/main/graphgen/templates/evaluation): prompt schemas · Templates for KG accuracy and judgment flows. · `graphgen/templates/evaluation`

## Connections

- CLI run → Engine: executes · CLI starts config-driven workflow execution.
- Web UI → Engine: invokes · UI triggers the same pipeline through a thin front end.
- Engine → Base APIs: binds · Engine resolves components through abstract interfaces.
- Engine → Read ops: orchestrates · Workflow begins with dataset ingestion.
- Read ops → Ingest models: uses · Readers, splitters, tokenizers, and preprocessors handle inputs.
- Read ops → Storage backends: persists · Input artifacts and intermediate state can be cached.
- Engine → Preprocess: orders · Preprocessing is a distinct pipeline stage.
- Preprocess → Ingest models: uses · Normalization and coreference operate on ingested text.
- Engine → Chunk ops: orders · Chunks are created before extraction and generation.
- Chunk ops → Ingest models: uses · Chunking depends on splitters and tokenizers.
- Engine → Extract ops: orchestrates · Structured extraction is a main pipeline stage.
- Extract ops → KG templates: prompts · Extraction behavior is shaped by prompt schemas.
- Extract ops → KG models: produces · Extracted knowledge feeds KG construction.
- Engine → KG build: orchestrates · Knowledge graph building is executed after extraction.
- KG build → KG models: uses · Concrete KG builders assemble text and multimodal graphs.
- KG build → Storage backends: writes · Built graphs are persisted for later stages.
- Engine → Partition ops: orchestrates · Graph partitioning prepares subgraphs for generation.
- Partition ops → Partition models: uses · BFS, DFS, Leiden, and other strategies select graph slices.
- Engine → Search ops: orchestrates · Retrieval can enrich or validate generation inputs.
- Search ops → Search models: uses · Web, wiki, and domain search backends provide evidence.
- Engine → Generate ops: orchestrates · Generation is the central synthetic-data output stage.
- Generate ops → LLM layer: calls · Generators rely on configured LLM backends.
- Generate ops → Gen templates: prompts · Question formats are defined by generation templates.
- Engine → Rephrase ops: orchestrates · Optional rewriting diversifies outputs.
- Rephrase ops → LLM layer: calls · Rephrasing also uses the LLM access layer.
- Rephrase ops → Rephrase templates: prompts · Style and tone variants come from prompt templates.
- Engine → Filter ops: orchestrates · Filtering removes low-quality outputs before export.
- Engine → Evaluate ops: orchestrates · Evaluation is a separate pipeline branch.
- Evaluate ops → Eval models: uses · Metric implementations score QA, KG, and triple outputs.
- Evaluate ops → Eval templates: prompts · Judgment and accuracy checks use evaluation prompts.
- Engine → Judge ops: supports · Judgement services validate statements and search results.
- Judge ops → LLM layer: calls · Judgement is typically LLM-backed.
- Engine → Quiz ops: supports · Quiz-specific workflows extend generation.
- Quiz ops → Gen templates: prompts · Quiz generation reuses question templates.
