# yifanfeng97/hyper-extract architecture

> Architecture diagram of the GitHub repository yifanfeng97/hyper-extract, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yifanfeng97/hyper-extract
- Repository: https://github.com/yifanfeng97/hyper-extract

## Overview

1. **Repository purpose**
   Hyper-Extract is a Python 3.11+ package for turning unstructured documents into structured “Knowledge Abstracts.” The core architecture is centered on extracting text into strongly typed outputs such as models, lists, sets, graphs, hypergraphs, temporal graphs, and spatial/spatio-temporal graphs.

2. **Primary runtime boundary**
   The main runtime is a Python library plus a CLI. Users either call the package API directly or invoke `he`/`he-mcp` from the command line. There is no separate backend service in the repo; the system runs locally and talks to external model providers over network APIs.

3. **Top-level pipeline**
   The dominant data flow is: input document → template selection/configuration → extraction method → LLM/embedding provider calls → typed knowledge abstract → search/show/export actions. This same pipeline is reused across the CLI, Python API, and MCP server.

4. **Type system / knowledge structures**
   `hyperextract/types` defines the canonical data structures that all extraction methods target. These types are the architectural backbone: they provide serialization, merging, search, iteration, slicing, and graph-specific behaviors. The repo treats the output structure as first-class, not as loose JSON.

5. **Template subsystem**
   `hyperextract/templates` contains a large preset library of YAML templates and the template design docs. Templates define language, output schema, identifiers, and extraction intent. The template engine in `hyperextract/utils/template_engine` parses these YAML definitions into runtime objects, validates fields, handles display/output metadata, and drives template discovery/generation.

6. **Extraction methods subsystem**
   `hyperextract/methods` is the algorithm layer. It contains “typical” extractors such as ATOM, ITEXT2KG, ITEXT2KG*, and KG-Gen, plus RAG-oriented methods such as GraphRAG, LightRAG, Hyper-RAG, Hypergraph-RAG, and Cog-RAG. These methods are the strategy boundary between a template/type specification and the actual LLM-driven extraction process.

7. **Model/provider integration**
   `hyperextract/utils/client.py` and the provider-related docs indicate a pluggable provider system for both LLM and embedding backends. The repo supports OpenAI-compatible endpoints, Anthropic Claude, Alibaba Bailian, and local vLLM. This provider layer is important because structured output and embeddings are external dependencies, and the extraction quality depends on them.

8. **CLI subsystem**
   `hyperextract/cli` is the user-facing operational layer. It exposes commands for configuration, listing templates, parsing documents, searching, showing results, exporting, and cleaning knowledge abstracts. The CLI is the main orchestration point for end-to-end document ingestion, query, and maintenance workflows.

9. **MCP server boundary**
   `hyperextract/mcp_server.py` exposes a read/query/export interface through the Model Context Protocol. This is a separate integration boundary for AI assistants and IDE agents: it wraps existing knowledge-abstract capabilities without introducing a new storage layer or extraction pipeline.

10. **Export and local knowledge representation**
   Utilities such as `hyperextract/utils/obsidian.py` and the `export` workflow show that knowledge abstracts can be serialized into external note systems, especially Obsidian vaults with linked Markdown. This means the repo supports a “structured knowledge → portable local artifact” boundary, not just in-memory querying.

11. **Configuration and logging**
   `hyperextract/cli/config.py`, `.env.example`, and `hyperextract/utils/logging.py` indicate standard runtime configuration through environment/file-based settings plus structured logging. This supports credential setup, provider selection, and reproducible CLI behavior.

12. **Docs and examples as architecture probes**
   The extensive `docs/` and `examples/` trees are not just marketing material; they mirror the product’s architecture. They document the template system, provider system, method families, and end-to-end workflows in both English and Chinese, and the examples demonstrate the same flows across languages, domains, and output types.

13. **Testing strategy**
   `tests/` is organized around the same boundaries as the code: types, template engine, CLI, utilities, integration, and MCP. This suggests the project’s correctness model is centered on preserving template parsing, type behavior, provider integration, and extraction workflow consistency rather than on a single monolithic app layer.

14. **External dependencies that shape architecture**
   The architecture is shaped by LLM APIs and embedding endpoints, plus the structured-output capabilities those models must support (`json_schema`/function-calling style behavior). The repo’s design emphasizes provider abstraction so the same extraction pipeline can run against cloud APIs or local inference servers.

Main entry points: [`hyperextract/types/base.py`](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/base.py) (Base Type), [`hyperextract/methods/registry.py`](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/registry.py) (Method Registry), [`hyperextract/utils/client.py`](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/client.py) (Provider Client), [`hyperextract/utils/template_engine/template.py`](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/template_engine/template.py) (Template Engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_api["Package API<br/>python package<br/>[__init__.py]"]
  node_cli["CLI<br/>command entry<br/>[cli.py]"]
  node_cli_cmds["Commands<br/>command modules"]
  node_mcp["MCP Server<br/>protocol bridge<br/>[mcp_server.py]"]
end

subgraph group_domain["Core Domain"]
  node_base_type["Base Type<br/>type system<br/>[base.py]"]
  node_model_type["Model<br/>structured record<br/>[model.py]"]
  node_list_type["List<br/>ordered collection<br/>[list.py]"]
  node_set_type["Set<br/>dedup collection<br/>[set.py]"]
  node_graph_type(("Graph<br/>[graph.py]"))
  node_hypergraph_type{{"Hypergraph<br/>n-ary graph<br/>[hypergraph.py]"}}
  node_temporal_type(("Temporal Graph<br/>time graph<br/>[temporal_graph.py]"))
  node_spatial_type(("Spatial Graph<br/>space graph<br/>[spatial_graph.py]"))
end

subgraph group_templates["Templates"]
  node_template_preset["Preset Templates<br/>yaml presets"]
  node_template_engine["Template Engine<br/>runtime parser<br/>[template.py]"]
  node_template_parsers["Template Parsers<br/>parser suite"]
end

subgraph group_methods["Extraction Methods"]
  node_method_registry["Method Registry<br/>dispatcher<br/>[registry.py]"]
  node_atom_method["ATOM<br/>typical extractor<br/>[atom.py]"]
  node_itext2kg_method["iText2KG<br/>typical extractor<br/>[itext2kg.py]"]
  node_graph_rag_method["GraphRAG<br/>rag extractor<br/>[graph_rag.py]"]
  node_hyper_rag_method["Hyper-RAG<br/>rag extractor<br/>[hyper_rag.py]"]
end

subgraph group_integration["Integration"]
  node_provider_client["Provider Client<br/>llm/embedding client<br/>[client.py]"]
  node_export_obsidian["Obsidian Export<br/>local export<br/>[obsidian.py]"]
  node_logging_util["Logging<br/>runtime support<br/>[logging.py]"]
end

subgraph group_docs["Docs & Examples"]
  node_docs_concepts["Architecture Docs<br/>documentation<br/>[architecture.md]"]
  node_examples["Examples"]
end

node_cli -->|"dispatches"| node_cli_cmds
node_api -->|"exposes"| node_method_registry
node_cli -->|"loads templates"| node_template_engine
node_cli_cmds -->|"selects methods"| node_method_registry
node_mcp -->|"wraps"| node_api
node_method_registry -->|"registers"| node_atom_method
node_method_registry -->|"registers"| node_itext2kg_method
node_method_registry -->|"registers"| node_graph_rag_method
node_method_registry -->|"registers"| node_hyper_rag_method
node_template_preset -->|"parsed by"| node_template_engine
node_template_engine -->|"uses"| node_template_parsers
node_template_engine -->|"configures"| node_method_registry
node_template_engine -->|"targets"| node_base_type
node_atom_method -->|"calls"| node_provider_client
node_itext2kg_method -->|"calls"| node_provider_client
node_graph_rag_method -->|"calls"| node_provider_client
node_hyper_rag_method -->|"calls"| node_provider_client
node_atom_method -->|"produces"| node_base_type
node_itext2kg_method -->|"produces"| node_graph_type
node_graph_rag_method -->|"produces"| node_graph_type
node_hyper_rag_method -->|"produces"| node_hypergraph_type
node_base_type -->|"specializes"| node_model_type
node_base_type -->|"specializes"| node_list_type
node_base_type -->|"specializes"| node_set_type
node_base_type -->|"specializes"| node_graph_type
node_base_type -->|"specializes"| node_hypergraph_type
node_base_type -->|"specializes"| node_temporal_type
node_base_type -->|"specializes"| node_spatial_type
node_base_type -->|"exports to"| node_export_obsidian
node_cli_cmds -->|"exports"| node_export_obsidian
node_cli_cmds -->|"logs"| node_logging_util
node_mcp -->|"offers export"| node_export_obsidian
node_docs_concepts -->|"documents"| node_template_preset
node_docs_concepts -->|"documents"| node_method_registry
node_docs_concepts -->|"documents"| node_provider_client
node_examples -->|"demonstrates"| node_template_preset
node_examples -->|"demonstrates"| node_atom_method
node_examples -->|"demonstrates"| node_graph_rag_method
node_examples -->|"demonstrates"| node_provider_client

click node_api "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/__init__.py"
click node_cli "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/cli/cli.py"
click node_cli_cmds "https://github.com/yifanfeng97/hyper-extract/tree/main/hyperextract/cli/commands"
click node_mcp "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/mcp_server.py"
click node_base_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/base.py"
click node_model_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/model.py"
click node_list_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/list.py"
click node_set_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/set.py"
click node_graph_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/graph.py"
click node_hypergraph_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/hypergraph.py"
click node_temporal_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/temporal_graph.py"
click node_spatial_type "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/spatial_graph.py"
click node_template_preset "https://github.com/yifanfeng97/hyper-extract/tree/main/hyperextract/templates/presets"
click node_template_engine "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/template_engine/template.py"
click node_template_parsers "https://github.com/yifanfeng97/hyper-extract/tree/main/hyperextract/utils/template_engine/parsers"
click node_method_registry "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/registry.py"
click node_atom_method "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/typical/atom.py"
click node_itext2kg_method "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/typical/itext2kg.py"
click node_graph_rag_method "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/rag/graph_rag.py"
click node_hyper_rag_method "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/rag/hyper_rag.py"
click node_provider_client "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/client.py"
click node_export_obsidian "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/obsidian.py"
click node_logging_util "https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/logging.py"
click node_docs_concepts "https://github.com/yifanfeng97/hyper-extract/blob/main/docs/en/concepts/architecture.md"
click node_examples "https://github.com/yifanfeng97/hyper-extract/tree/main/examples"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_cli,node_cli_cmds,node_mcp toneBlue
class node_base_type,node_model_type,node_list_type,node_set_type,node_graph_type,node_hypergraph_type,node_temporal_type,node_spatial_type toneAmber
class node_template_preset,node_template_engine,node_template_parsers toneMint
class node_method_registry,node_atom_method,node_itext2kg_method,node_graph_rag_method,node_hyper_rag_method toneRose
class node_provider_client,node_export_obsidian,node_logging_util toneIndigo
class node_docs_concepts,node_examples toneTeal
```

## Components

### User Interfaces

Library, CLI, and MCP entry points for running extraction and querying results.

- [Package API](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/__init__.py): python package · Public import surface for programmatic use. · `hyperextract/__init__.py`
- [CLI](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/cli/cli.py): command entry · Main command-line orchestration entry point. · `hyperextract/cli/cli.py`
- [Commands](https://github.com/yifanfeng97/hyper-extract/tree/main/hyperextract/cli/commands): command modules · Operational commands for parse, search, show, export, config, and maintenance. · `hyperextract/cli/commands`
- [MCP Server](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/mcp_server.py): protocol bridge · Model Context Protocol boundary for assistant-driven query and export workflows. · `hyperextract/mcp_server.py`

### Core Domain

Canonical knowledge-abstract types and their behaviors.

- [Base Type](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/base.py): type system · Shared serialization, merging, iteration, and search behavior. · `hyperextract/types/base.py`
- [Model](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/model.py): structured record · Attribute-oriented knowledge abstract. · `hyperextract/types/model.py`
- [List](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/list.py): ordered collection · Ordered extracted items with slicing and iteration. · `hyperextract/types/list.py`
- [Set](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/set.py): dedup collection · Deduplicated extracted items and operators. · `hyperextract/types/set.py`
- [Graph](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/graph.py): relation graph · Node-edge knowledge abstraction for relationships. · `hyperextract/types/graph.py`
- [Hypergraph](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/hypergraph.py): n-ary graph · Higher-order relation structure for grouped associations. · `hyperextract/types/hypergraph.py`
- [Temporal Graph](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/temporal_graph.py): time graph · Graph abstraction with temporal structure. · `hyperextract/types/temporal_graph.py`
- [Spatial Graph](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/types/spatial_graph.py): space graph · Graph abstraction with spatial structure. · `hyperextract/types/spatial_graph.py`

### Templates

YAML presets and the runtime template engine that turns them into usable extraction specs.

- [Preset Templates](https://github.com/yifanfeng97/hyper-extract/tree/main/hyperextract/templates/presets): yaml presets · Domain and output presets organized by task and vertical. · `hyperextract/templates/presets`
- [Template Engine](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/template_engine/template.py): runtime parser · Loads, validates, and materializes template definitions. · `hyperextract/utils/template_engine/template.py`
- [Template Parsers](https://github.com/yifanfeng97/hyper-extract/tree/main/hyperextract/utils/template_engine/parsers): parser suite · Field, output, schema, and metadata parsing for YAML templates. · `hyperextract/utils/template_engine/parsers`

### Extraction Methods

Strategy layer for typical extraction and RAG-oriented pipelines.

- [Method Registry](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/registry.py): dispatcher · Selects and exposes extraction strategies. · `hyperextract/methods/registry.py`
- [ATOM](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/typical/atom.py): typical extractor · Representative non-RAG extraction strategy. · `hyperextract/methods/typical/atom.py`
- [iText2KG](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/typical/itext2kg.py): typical extractor · Text-to-knowledge-graph extraction family. · `hyperextract/methods/typical/itext2kg.py`
- [GraphRAG](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/rag/graph_rag.py): rag extractor · Graph-based retrieval-augmented extraction strategy. · `hyperextract/methods/rag/graph_rag.py`
- [Hyper-RAG](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/methods/rag/hyper_rag.py): rag extractor · Hypergraph-oriented retrieval-augmented extraction strategy. · `hyperextract/methods/rag/hyper_rag.py`

### Integration

Provider clients, export paths, and runtime support utilities.

- [Provider Client](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/client.py): llm/embedding client · Pluggable client layer for model and embedding backends. · `hyperextract/utils/client.py`
- [Obsidian Export](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/obsidian.py): local export · Serializes knowledge abstracts into linked Markdown vaults. · `hyperextract/utils/obsidian.py`
- [Logging](https://github.com/yifanfeng97/hyper-extract/blob/main/hyperextract/utils/logging.py): runtime support · Structured logging and runtime diagnostics. · `hyperextract/utils/logging.py`

### Docs & Examples

Reference material and examples that mirror the main architecture.

- [Architecture Docs](https://github.com/yifanfeng97/hyper-extract/blob/main/docs/en/concepts/architecture.md): documentation · Core design, provider, methods, and template concepts. · `docs/en/concepts/architecture.md`
- [Examples](https://github.com/yifanfeng97/hyper-extract/tree/main/examples): usage examples · Cross-language demos for templates, methods, autotypes, and providers. · `examples`

## Connections

- CLI → Commands: dispatches · CLI routes user commands into operational handlers.
- Package API → Method Registry: exposes · Package API makes the extraction method surface available.
- CLI → Template Engine: loads templates · Command workflows resolve templates before extraction.
- Commands → Method Registry: selects methods · Commands choose extraction strategies via the registry.
- MCP Server → Package API: wraps · MCP exposes library capabilities without a separate backend.
- Method Registry → ATOM: registers · Typical extractor family is available through the registry.
- Method Registry → iText2KG: registers · Typical extractor family is available through the registry.
- Method Registry → GraphRAG: registers · RAG extractor family is available through the registry.
- Method Registry → Hyper-RAG: registers · RAG extractor family is available through the registry.
- Preset Templates → Template Engine: parsed by · YAML presets become runtime template objects.
- Template Engine → Template Parsers: uses · Engine delegates validation and schema parsing to parser modules.
- Template Engine → Method Registry: configures · Template metadata drives method selection and execution parameters.
- Template Engine → Base Type: targets · Templates specify the structured output shape to produce.
- ATOM → Provider Client: calls · Extraction requires model or embedding provider access.
- iText2KG → Provider Client: calls · Extraction requires model or embedding provider access.
- GraphRAG → Provider Client: calls · Retrieval-augmented extraction depends on external providers.
- Hyper-RAG → Provider Client: calls · Retrieval-augmented extraction depends on external providers.
- ATOM → Base Type: produces · All extractors emit canonical knowledge-abstract types.
- iText2KG → Graph: produces · Graph-oriented outputs are a central target for this family.
- GraphRAG → Graph: produces · GraphRAG builds graph-structured knowledge abstracts.
- Hyper-RAG → Hypergraph: produces · Hyper-RAG builds hypergraph-structured knowledge abstracts.
- Base Type → Model: specializes · Model is one canonical structured output form.
- Base Type → List: specializes · List is one canonical structured output form.
- Base Type → Set: specializes · Set is one canonical structured output form.
- Base Type → Graph: specializes · Graph is one canonical structured output form.
- Base Type → Hypergraph: specializes · Hypergraph is one canonical structured output form.
- Base Type → Temporal Graph: specializes · Temporal graph extends the canonical output family.
- Base Type → Spatial Graph: specializes · Spatial graph extends the canonical output family.
- Base Type → Obsidian Export: exports to · Knowledge abstracts can be serialized into local note artifacts.
- Commands → Obsidian Export: exports · CLI export commands target external note systems.
- Commands → Logging: logs · Operational commands rely on structured runtime logging.
- MCP Server → Obsidian Export: offers export · Assistant-facing workflows can surface export actions.
- Architecture Docs → Preset Templates: documents · Docs explain the preset library and selection patterns.
- Architecture Docs → Method Registry: documents · Docs describe the method families and registry boundary.
- Architecture Docs → Provider Client: documents · Docs explain provider abstraction and backend choices.
- Examples → Preset Templates: demonstrates · Examples mirror template-driven extraction flows.
- Examples → ATOM: demonstrates · Examples show typical extraction usage.
- Examples → GraphRAG: demonstrates · Examples show RAG-based extraction usage.
- Examples → Provider Client: demonstrates · Examples show different provider backends.