yifanfeng97/hyper-extract

1. Repository purpose

Hyper-Extract is a Python 3.11+ package for turning unstructured documents into structured “Knowledge Abstracts.” The core architecture is centered on extracting text into strongly typed outputs such as models, lists, sets, graphs, hypergraphs, temporal graphs, and spatial/spatio-temporal graphs.

2. Primary runtime boundary

The main runtime is a Python library plus a CLI. Users either call the package API directly or invoke he/he-mcp from the command line. There is no separate backend service in the repo; the system runs locally and talks to external model providers over network APIs.

3. Top-level pipeline

The dominant data flow is: input document → template selection/configuration → extraction method → LLM/embedding provider calls → typed knowledge abstract → search/show/export actions. This same pipeline is reused across the CLI, Python API, and MCP server.

4. Type system / knowledge structures

hyperextract/types defines the canonical data structures that all extraction methods target. These types are the architectural backbone: they provide serialization, merging, search, iteration, slicing, and graph-specific behaviors. The repo treats the output structure as first-class, not as loose JSON.

5. Template subsystem

hyperextract/templates contains a large preset library of YAML templates and the template design docs. Templates define language, output schema, identifiers, and extraction intent. The template engine in hyperextract/utils/template_engine parses these YAML definitions into runtime objects, validates fields, handles display/output metadata, and drives template discovery/generation.

6. Extraction methods subsystem

hyperextract/methods is the algorithm layer. It contains “typical” extractors such as ATOM, ITEXT2KG, ITEXT2KG*, and KG-Gen, plus RAG-oriented methods such as GraphRAG, LightRAG, Hyper-RAG, Hypergraph-RAG, and Cog-RAG. These methods are the strategy boundary between a template/type specification and the actual LLM-driven extraction process.

7. Model/provider integration

hyperextract/utils/client.py and the provider-related docs indicate a pluggable provider system for both LLM and embedding backends. The repo supports OpenAI-compatible endpoints, Anthropic Claude, Alibaba Bailian, and local vLLM. This provider layer is important because structured output and embeddings are external dependencies, and the extraction quality depends on them.

8. CLI subsystem

hyperextract/cli is the user-facing operational layer. It exposes commands for configuration, listing templates, parsing documents, searching, showing results, exporting, and cleaning knowledge abstracts. The CLI is the main orchestration point for end-to-end document ingestion, query, and maintenance workflows.

9. MCP server boundary

hyperextract/mcp_server.py exposes a read/query/export interface through the Model Context Protocol. This is a separate integration boundary for AI assistants and IDE agents: it wraps existing knowledge-abstract capabilities without introducing a new storage layer or extraction pipeline.

10. Export and local knowledge representation

Utilities such as hyperextract/utils/obsidian.py and the export workflow show that knowledge abstracts can be serialized into external note systems, especially Obsidian vaults with linked Markdown. This means the repo supports a “structured knowledge → portable local artifact” boundary, not just in-memory querying.

11. Configuration and logging

hyperextract/cli/config.py, .env.example, and hyperextract/utils/logging.py indicate standard runtime configuration through environment/file-based settings plus structured logging. This supports credential setup, provider selection, and reproducible CLI behavior.

12. Docs and examples as architecture probes

The extensive docs/ and examples/ trees are not just marketing material; they mirror the product’s architecture. They document the template system, provider system, method families, and end-to-end workflows in both English and Chinese, and the examples demonstrate the same flows across languages, domains, and output types.

13. Testing strategy

tests/ is organized around the same boundaries as the code: types, template engine, CLI, utilities, integration, and MCP. This suggests the project’s correctness model is centered on preserving template parsing, type behavior, provider integration, and extraction workflow consistency rather than on a single monolithic app layer.

14. External dependencies that shape architecture

The architecture is shaped by LLM APIs and embedding endpoints, plus the structured-output capabilities those models must support (json_schema/function-calling style behavior). The repo’s design emphasizes provider abstraction so the same extraction pipeline can run against cloud APIs or local inference servers.