How hankcs/hanlp works
HanLP is a multilingual NLP toolkit with PyTorch and TensorFlow implementations. Users load pretrained components or compose components into pipelines; task models then produce structured linguistic results such as tokenization, tags, parses, semantic roles, AMR, and similarity scores. The graph preserves those documented task families and highlights the shared component, document, and embedding layers. Some task implementations are represented by their exact source directories because their individual files were not sampled; links between those branches are omitted where the excerpts do not establish calls.
Main entry points: pipeline.py Pipeline · multi_task_learning.py Multitask learning · __init__.py HanLP API · tokenizers Tokenization
Components 20
Entry and composition
- HanLP API
hanlp/__init__.py - Pretrained models
hanlp/pretrained - Pipeline
hanlp/components/pipeline.py - Multitask learning
hanlp/components/mtl/multi_task_learning.py
Linguistic analysis
- Tokenization
hanlp/components/tokenizers - POS tagging
hanlp/components/taggers - Named entities
hanlp/components/ner - Dependency parsing
hanlp/components/parsers - Constituency parsing
hanlp/components/parsers/constituency
Semantic tasks
- Semantic parsing
hanlp/components/parsers/biaffine - Semantic roles
hanlp/components/srl - AMR parsing
hanlp/components/amr - Text similarity
hanlp/components/sts - Classification
hanlp/components/classifiers
Shared runtime
- Component base
hanlp/common/component.py - Document model
plugins/hanlp_common/hanlp_common/document.py - Data transforms
hanlp/common/transform.py - Dataset batching
hanlp/common/dataset.py - Embeddings
hanlp/layers/embeddings
Other components
- NLP user
Connections 21
- NLP user → HanLP API: loads or composes
hanlp/__init__.py - HanLP API → Pretrained models: resolves models
hanlp/__init__.py - HanLP API → Pipeline: creates pipeline
hanlp/__init__.py - Pipeline → Component base: wraps components
hanlp/components/pipeline.py - Pipeline → Document model: reads and updates
hanlp/components/pipeline.py - Pipeline → Tokenization: dispatches task
- Pipeline → POS tagging: dispatches task
- Pipeline → Named entities: dispatches task
- Pipeline → Dependency parsing: dispatches task
- Pipeline → Constituency parsing: dispatches task
- Pipeline → Semantic parsing: dispatches task
- Pipeline → Semantic roles: dispatches task
- Pipeline → AMR parsing: dispatches task
- Pipeline → Text similarity: dispatches task
- Pipeline → Classification: dispatches task
- Multitask learning → Tokenization: supports joint tasks
- Multitask learning → POS tagging: supports joint tasks
- Multitask learning → Named entities: supports joint tasks
- Multitask learning → Dependency parsing: supports joint tasks
- Dataset batching → Data transforms: applies transforms
hanlp/common/dataset.py - Embeddings → Data transforms: consumes transformed inputs
hanlp/layers/embeddings/contextual_word_embedding.py