How determined22/zh-ner-tf works
Project Type and Purpose
• The project implements a sequence labeling (NER) system using a BiLSTM-CRF architecture specifically designed for Chinese texts.
• Its goal is to recognize named entities (PERSON, LOCATION, ORGANIZATION) from input sentences with a character-based approach.
• The README explains the model layers, data format, training, testing, and demo procedures.
a. Data Management & Preprocessing:
• Raw and processed data: Represent the directories in ./data_path that include raw files in "original", preprocessed datasets (train_data/test_data), and the vocabulary file (word2id.pkl).
• Data conversion: Include the role of data.py in generating vocabulary mappings and formatting the input data.
b. Model Architecture:
• Embedding / Look-up Layer: Illustrate the transformation of one-hot vectors to character embeddings.
• BiLSTM Layer: Show this layer to handle context from both backward and forward directions.
• CRF Layer: Represent the CRF for structured prediction to enforce valid tag transitions.
• Reference the internal modules (model.py encapsulates the model, main.py launches training/inference, and utils.py for miscellaneous helper functions).
c. Training, Checkpointing, and Evaluation:
• Training Flow: Indicate that main.py enables both training and testing modes with command line parameters such as --mode=train/test.
• Checkpoint Management: Highlight the "data_path_save" directory structure with timestamps and checkpoints, where model weights (data, index, meta files) are saved.
• Evaluation: Include the eval.py module and the provided conlleval_rev.pl script for external metric evaluation.
d. User Interaction:
• Demo Mode: Note the demo mode capability where input sentences are fed and processed to output NER results (as shown in pics/demo.txt and pics/pic2.png).
Relationships and Data Flow
• Then, from preprocessed data into the model pipeline: input (embedding), processed through BiLSTM, and finally passed to the CRF layer for prediction.
• Training control flow from main.py to model training, checkpoint saving, and evaluation (via eval.py).
Architectural Patterns and Design Principles
• Use of a layered model architecture (lookup → BiLSTM → CRF) which follows a typical deep learning design for sequence labeling.
• Separation of concerns: data preprocessing (data.py and data directories), model design (model.py), training orchestration (main.py), and evaluation.
• Use of external evaluation tools and scripting (conlleval_rev.pl) for performance validation.
Technology Stack
• Specify dependencies: Python 3 and TensorFlow 1.2.
• Mention that the project was influenced by another open-source sequence tagging repository.
• Standard libraries used in utils.py if relevant.