How myhhub/knowledgegraph works
This repository is a collection of practical components for building a Chinese knowledge graph, especially movie and encyclopedia knowledge. Its user-facing paths include question-to-answer through REfO/SPARQL and natural-language search through Elasticsearch; its data path spans crawling, extraction, graph conversion, and entity linking. README descriptions establish several capabilities whose code is not sampled, so those are represented at their supported directory paths without speculative wiring. Sampled code confirms MySQL crawler pipelines, a REfO query loop, Elasticsearch search, and Silk/Jena controls; other connections are limited to documented stages.
Main entry points: query.py REfO question answering · struct_to_rdf RDF conversion · views.py Query parsing and lookup · udf DeepDive extraction
Components 24
Data acquisition
- Baike crawlers
ie/craw/baidu_baike/baidu_baike/spiders/baidu_baike.py - Bulk Baike crawlers
ie/craw/craw_all_baidu/baidu_baike/spiders - Hudong crawlers
ie/craw/hudong_baike/hudong_baike/spiders/hudong_baike.py - Bulk Hudong crawler
ie/craw/craw_all_hudong/craw_all_hudong/spiders/hudong_baike.py - News and Weixin
ie/craw - Movie records
ie/craw/baidu_baike/baidu_baike/pipelines.py - Lemma records
ie/craw/craw_all_baidu/baidu_baike/pipelines.py - Hudong records
ie/craw/craw_all_hudong/craw_all_hudong/pipelines.py
Extraction and conversion
- DeepDive extraction
ie/deepdive/udf - Neural relation extraction
ie/re_cnn_att - RDF conversion
ie/struct_to_rdf
Graph storage and fusion
- Entity linking
knowledge_fusion/silk/batch_link.py - Jena dataset control
knowledge_fusion/silk/batch_link.py
Search and QA
- REfO question answering
KBQA/patternREfO/query.py - Question rules
KBQA/patternREfO/utils/rules.py - Word tagging
KBQA/patternREfO/utils/word_tagging.py - Semantic search
semantic_search/elasticsearch/query.py - Query parsing and lookup
semantic_search/elasticsearch/utils/views.py
Other components
- Questioner
- Web sources
- MySQL
- Neo4j
- Jena / Fuseki
- Elasticsearch
Connections 24
- Web sources → Baike crawlers: provides pages
- Web sources → Bulk Baike crawlers: provides pages
- Web sources → Hudong crawlers: provides pages
- Web sources → Bulk Hudong crawler: provides pages
- Web sources → News and Weixin: provides content
- Movie records → MySQL: writes actor/movie
- Lemma records → MySQL: writes lemmas
- Hudong records → MySQL: writes lemmas
- MySQL → RDF conversion: structured data
- MySQL → DeepDive extraction: text input
- DeepDive extraction → Neural relation extraction: relation examples
- RDF conversion → Neo4j: loads graph
- RDF conversion → Jena dataset control: RDF input
- Jena dataset control → Jena / Fuseki: creates or loads
- Entity linking → Jena / Fuseki: controls linking
- Questioner → REfO question answering: asks
- REfO question answering → Word tagging: tags question
- REfO question answering → Question rules: applies rules
- REfO question answering → Jena / Fuseki: SPARQL query
- Jena / Fuseki → REfO question answering: returns bindings
- Questioner → Semantic search: asks
- Semantic search → Query parsing and lookup: searches
- Query parsing and lookup → Elasticsearch: queries index
- Elasticsearch → Query parsing and lookup: returns hits