# myhhub/knowledgegraph architecture

> Architecture diagram of the GitHub repository myhhub/knowledgegraph, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/myhhub/knowledgegraph
- Repository: https://github.com/myhhub/knowledgegraph

## Overview

This repository is a collection of practical components for building a Chinese knowledge graph, especially movie and encyclopedia knowledge. Its user-facing paths include question-to-answer through REfO/SPARQL and natural-language search through Elasticsearch; its data path spans crawling, extraction, graph conversion, and entity linking. README descriptions establish several capabilities whose code is not sampled, so those are represented at their supported directory paths without speculative wiring. Sampled code confirms MySQL crawler pipelines, a REfO query loop, Elasticsearch search, and Silk/Jena controls; other connections are limited to documented stages.

Main entry points: [`KBQA/patternREfO/query.py`](https://github.com/myhhub/knowledgegraph/blob/master/KBQA/patternREfO/query.py) (REfO question answering), [`ie/struct_to_rdf`](https://github.com/myhhub/knowledgegraph/tree/master/ie/struct_to_rdf) (RDF conversion), [`semantic_search/elasticsearch/utils/views.py`](https://github.com/myhhub/knowledgegraph/blob/master/semantic_search/elasticsearch/utils/views.py) (Query parsing and lookup), [`ie/deepdive/udf`](https://github.com/myhhub/knowledgegraph/tree/master/ie/deepdive/udf) (DeepDive extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Data acquisition"]
  node_baike_crawlers["Baike crawlers<br/>[baidu_baike.py]"]
  node_bulk_crawlers["Bulk Baike crawlers"]
  node_hudong_crawlers["Hudong crawlers<br/>[hudong_baike.py]"]
  node_bulk_hudong["Bulk Hudong crawler<br/>[hudong_baike.py]"]
  node_other_crawlers["News and Weixin"]
  node_movie_mysql_pipeline["Movie records<br/>[pipelines.py]"]
  node_baike_mysql_pipeline["Lemma records<br/>[pipelines.py]"]
  node_hudong_mysql_pipeline["Hudong records<br/>[pipelines.py]"]
end

subgraph group_extraction["Extraction and conversion"]
  node_deepdive["DeepDive extraction"]
  node_cnn_re["Neural relation extraction"]
  node_rdf["RDF conversion"]
end

subgraph group_fusion["Graph storage and fusion"]
  node_silk["Entity linking<br/>[batch_link.py]"]
  node_jena_control["Jena dataset control<br/>[batch_link.py]"]
end

subgraph group_applications["Search and QA"]
  node_ref_query["REfO question answering<br/>[query.py]"]
  node_ref_rules["Question rules<br/>[rules.py]"]
  node_word_tagging["Word tagging<br/>[word_tagging.py]"]
  node_semantic_query["Semantic search<br/>[query.py]"]
  node_search_views["Query parsing and lookup<br/>[views.py]"]
end

node_user(("Questioner"))
node_web(("Web sources"))
node_mysql[("MySQL")]
node_neo4j[("Neo4j")]
node_jena[("Jena / Fuseki")]
node_elasticsearch[("Elasticsearch")]

node_web -->|"provides pages"| node_baike_crawlers
node_web -->|"provides pages"| node_bulk_crawlers
node_web -->|"provides pages"| node_hudong_crawlers
node_web -->|"provides pages"| node_bulk_hudong
node_web -->|"provides content"| node_other_crawlers
node_movie_mysql_pipeline -->|"writes actor/movie"| node_mysql
node_baike_mysql_pipeline -->|"writes lemmas"| node_mysql
node_hudong_mysql_pipeline -->|"writes lemmas"| node_mysql
node_mysql -->|"structured data"| node_rdf
node_mysql -->|"text input"| node_deepdive
node_deepdive -.->|"relation examples"| node_cnn_re
node_rdf -->|"loads graph"| node_neo4j
node_rdf -->|"RDF input"| node_jena_control
node_jena_control -->|"creates or loads"| node_jena
node_silk -->|"controls linking"| node_jena
node_user -->|"asks"| node_ref_query
node_ref_query -->|"tags question"| node_word_tagging
node_ref_query -->|"applies rules"| node_ref_rules
node_ref_query -->|"SPARQL query"| node_jena
node_jena -->|"returns bindings"| node_ref_query
node_user -->|"asks"| node_semantic_query
node_semantic_query -->|"searches"| node_search_views
node_search_views -->|"queries index"| node_elasticsearch
node_elasticsearch -->|"returns hits"| node_search_views

click node_baike_crawlers "https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/baidu_baike/baidu_baike/spiders/baidu_baike.py"
click node_bulk_crawlers "https://github.com/myhhub/knowledgegraph/tree/master/ie/craw/craw_all_baidu/baidu_baike/spiders"
click node_hudong_crawlers "https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/hudong_baike/hudong_baike/spiders/hudong_baike.py"
click node_bulk_hudong "https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/craw_all_hudong/craw_all_hudong/spiders/hudong_baike.py"
click node_other_crawlers "https://github.com/myhhub/knowledgegraph/tree/master/ie/craw"
click node_movie_mysql_pipeline "https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/baidu_baike/baidu_baike/pipelines.py"
click node_baike_mysql_pipeline "https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/craw_all_baidu/baidu_baike/pipelines.py"
click node_hudong_mysql_pipeline "https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/craw_all_hudong/craw_all_hudong/pipelines.py"
click node_deepdive "https://github.com/myhhub/knowledgegraph/tree/master/ie/deepdive/udf"
click node_cnn_re "https://github.com/myhhub/knowledgegraph/tree/master/ie/re_cnn_att"
click node_rdf "https://github.com/myhhub/knowledgegraph/tree/master/ie/struct_to_rdf"
click node_silk "https://github.com/myhhub/knowledgegraph/blob/master/knowledge_fusion/silk/batch_link.py"
click node_jena_control "https://github.com/myhhub/knowledgegraph/blob/master/knowledge_fusion/silk/batch_link.py"
click node_ref_query "https://github.com/myhhub/knowledgegraph/blob/master/KBQA/patternREfO/query.py"
click node_ref_rules "https://github.com/myhhub/knowledgegraph/blob/master/KBQA/patternREfO/utils/rules.py"
click node_word_tagging "https://github.com/myhhub/knowledgegraph/blob/master/KBQA/patternREfO/utils/word_tagging.py"
click node_semantic_query "https://github.com/myhhub/knowledgegraph/blob/master/semantic_search/elasticsearch/query.py"
click node_search_views "https://github.com/myhhub/knowledgegraph/blob/master/semantic_search/elasticsearch/utils/views.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_baike_crawlers,node_bulk_crawlers,node_hudong_crawlers,node_bulk_hudong,node_other_crawlers,node_movie_mysql_pipeline,node_baike_mysql_pipeline,node_hudong_mysql_pipeline toneBlue
class node_deepdive,node_cnn_re,node_rdf,node_mysql,node_neo4j,node_jena,node_elasticsearch toneAmber
class node_silk,node_jena_control toneMint
class node_ref_query,node_ref_rules,node_word_tagging,node_semantic_query,node_search_views toneRose
class node_user,node_web toneIndigo
```

## Components

### Data acquisition

- [Baike crawlers](https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/baidu_baike/baidu_baike/spiders/baidu_baike.py): `ie/craw/baidu_baike/baidu_baike/spiders/baidu_baike.py`
- [Bulk Baike crawlers](https://github.com/myhhub/knowledgegraph/tree/master/ie/craw/craw_all_baidu/baidu_baike/spiders): `ie/craw/craw_all_baidu/baidu_baike/spiders`
- [Hudong crawlers](https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/hudong_baike/hudong_baike/spiders/hudong_baike.py): `ie/craw/hudong_baike/hudong_baike/spiders/hudong_baike.py`
- [Bulk Hudong crawler](https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/craw_all_hudong/craw_all_hudong/spiders/hudong_baike.py): `ie/craw/craw_all_hudong/craw_all_hudong/spiders/hudong_baike.py`
- [News and Weixin](https://github.com/myhhub/knowledgegraph/tree/master/ie/craw): `ie/craw`
- [Movie records](https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/baidu_baike/baidu_baike/pipelines.py): `ie/craw/baidu_baike/baidu_baike/pipelines.py`
- [Lemma records](https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/craw_all_baidu/baidu_baike/pipelines.py): `ie/craw/craw_all_baidu/baidu_baike/pipelines.py`
- [Hudong records](https://github.com/myhhub/knowledgegraph/blob/master/ie/craw/craw_all_hudong/craw_all_hudong/pipelines.py): `ie/craw/craw_all_hudong/craw_all_hudong/pipelines.py`

### Extraction and conversion

- [DeepDive extraction](https://github.com/myhhub/knowledgegraph/tree/master/ie/deepdive/udf): `ie/deepdive/udf`
- [Neural relation extraction](https://github.com/myhhub/knowledgegraph/tree/master/ie/re_cnn_att): `ie/re_cnn_att`
- [RDF conversion](https://github.com/myhhub/knowledgegraph/tree/master/ie/struct_to_rdf): `ie/struct_to_rdf`

### Graph storage and fusion

- [Entity linking](https://github.com/myhhub/knowledgegraph/blob/master/knowledge_fusion/silk/batch_link.py): `knowledge_fusion/silk/batch_link.py`
- [Jena dataset control](https://github.com/myhhub/knowledgegraph/blob/master/knowledge_fusion/silk/batch_link.py): `knowledge_fusion/silk/batch_link.py`

### Search and QA

- [REfO question answering](https://github.com/myhhub/knowledgegraph/blob/master/KBQA/patternREfO/query.py): `KBQA/patternREfO/query.py`
- [Question rules](https://github.com/myhhub/knowledgegraph/blob/master/KBQA/patternREfO/utils/rules.py): `KBQA/patternREfO/utils/rules.py`
- [Word tagging](https://github.com/myhhub/knowledgegraph/blob/master/KBQA/patternREfO/utils/word_tagging.py): `KBQA/patternREfO/utils/word_tagging.py`
- [Semantic search](https://github.com/myhhub/knowledgegraph/blob/master/semantic_search/elasticsearch/query.py): `semantic_search/elasticsearch/query.py`
- [Query parsing and lookup](https://github.com/myhhub/knowledgegraph/blob/master/semantic_search/elasticsearch/utils/views.py): `semantic_search/elasticsearch/utils/views.py`

### Other components

- **Questioner**
- **Web sources**
- **MySQL**
- **Neo4j**
- **Jena / Fuseki**
- **Elasticsearch**

## Connections

- Web sources → Baike crawlers: provides pages
- Web sources → Bulk Baike crawlers: provides pages
- Web sources → Hudong crawlers: provides pages
- Web sources → Bulk Hudong crawler: provides pages
- Web sources → News and Weixin: provides content
- Movie records → MySQL: writes actor/movie
- Lemma records → MySQL: writes lemmas
- Hudong records → MySQL: writes lemmas
- MySQL → RDF conversion: structured data
- MySQL → DeepDive extraction: text input
- DeepDive extraction → Neural relation extraction: relation examples
- RDF conversion → Neo4j: loads graph
- RDF conversion → Jena dataset control: RDF input
- Jena dataset control → Jena / Fuseki: creates or loads
- Entity linking → Jena / Fuseki: controls linking
- Questioner → REfO question answering: asks
- REfO question answering → Word tagging: tags question
- REfO question answering → Question rules: applies rules
- REfO question answering → Jena / Fuseki: SPARQL query
- Jena / Fuseki → REfO question answering: returns bindings
- Questioner → Semantic search: asks
- Semantic search → Query parsing and lookup: searches
- Query parsing and lookup → Elasticsearch: queries index
- Elasticsearch → Query parsing and lookup: returns hits