# determined22/zh-ner-tf architecture

> Architecture diagram of the GitHub repository determined22/zh-ner-tf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/determined22/zh-ner-tf
- Repository: https://github.com/determined22/zh-ner-tf

## Overview

##### Project Type and Purpose
- The project implements a sequence labeling (NER) system using a BiLSTM-CRF architecture specifically designed for Chinese texts.
- Its goal is to recognize named entities (PERSON, LOCATION, ORGANIZATION) from input sentences with a character-based approach.
- The README explains the model layers, data format, training, testing, and demo procedures.

a. Data Management & Preprocessing:
- Raw and processed data: Represent the directories in ./data_path that include raw files in "original", preprocessed datasets (train_data/test_data), and the vocabulary file (word2id.pkl).
- Data conversion: Include the role of data.py in generating vocabulary mappings and formatting the input data.

b. Model Architecture:
- Embedding / Look-up Layer: Illustrate the transformation of one-hot vectors to character embeddings.
- BiLSTM Layer: Show this layer to handle context from both backward and forward directions.
- CRF Layer: Represent the CRF for structured prediction to enforce valid tag transitions.
- Reference the internal modules (model.py encapsulates the model, main.py launches training/inference, and utils.py for miscellaneous helper functions).

c. Training, Checkpointing, and Evaluation:
- Training Flow: Indicate that main.py enables both training and testing modes with command line parameters such as --mode=train/test.
- Checkpoint Management: Highlight the "data_path_save" directory structure with timestamps and checkpoints, where model weights (data, index, meta files) are saved.
- Evaluation: Include the eval.py module and the provided conlleval_rev.pl script for external metric evaluation.

d. User Interaction:
- Demo Mode: Note the demo mode capability where input sentences are fed and processed to output NER results (as shown in pics/demo.txt and pics/pic2.png).

##### Relationships and Data Flow
- Then, from preprocessed data into the model pipeline: input (embedding), processed through BiLSTM, and finally passed to the CRF layer for prediction.
- Training control flow from main.py to model training, checkpoint saving, and evaluation (via eval.py).

##### Architectural Patterns and Design Principles
- Use of a layered model architecture (lookup → BiLSTM → CRF) which follows a typical deep learning design for sequence labeling.
- Separation of concerns: data preprocessing (data.py and data directories), model design (model.py), training orchestration (main.py), and evaluation.
- Use of external evaluation tools and scripting (conlleval_rev.pl) for performance validation.

##### Technology Stack
- Specify dependencies: Python 3 and TensorFlow 1.2.
- Mention that the project was influenced by another open-source sequence tagging repository.
- Standard libraries used in utils.py if relevant.

## Diagram (Mermaid)

```mermaid
graph TB
  %% Data Layer
  subgraph "Input/Data Sources"
    A["Raw Data (data_path/original)"]:::data
    D["Data Conversion (data.py)"]:::data
    B["Preprocessed Data (train_data/test_data)"]:::data
    C["Vocabulary (word2id.pkl)"]:::data
    A -->|"processes"| D
    D -->|"generates"| B
    D -->|"generates"| C
  end

  %% Model Architecture
  subgraph "Model Architecture"
    H["Model (model.py)"]:::model
    E["Embedding Lookup"]:::model
    F["BiLSTM Layer"]:::model
    G["CRF Layer"]:::model
    B -->|"inputs"| H
    H -->|"performs"| E
    E -->|"flows"| F
    F -->|"outputs"| G
  end

  %% Training & Evaluation
  subgraph "Training & Evaluation"
    I["Training Controller (main.py)"]:::training
    J["Checkpointing (data_path_save)"]:::training
    K["Evaluation (eval.py)"]:::training
    L["Evaluation Script (conlleval_rev.pl)"]:::training
    I -->|"trains"| H
    I -->|"saves"| J
    I -->|"evaluates"| K
    K -->|"uses"| L
  end

  %% User Interaction
  subgraph "User Interaction"
    M["Demo Mode (main.py demo)"]:::ui
    N["Demo Output (pics/demo.txt & pics/pic2.png)"]:::ui
    I -->|"triggers"| M
    M -->|"displays"| N
  end

  %% Utilities
  O["Misc Utilities (utils.py)"]:::util
  O -->|"supports"| I

  %% Class Definitions
  classDef data fill:#f9f,stroke:#333,stroke-width:2px;
  classDef model fill:#bbf,stroke:#333,stroke-width:2px;
  classDef training fill:#fea,stroke:#333,stroke-width:2px;
  classDef ui fill:#dfd,stroke:#333,stroke-width:2px;
  classDef util fill:#efc,stroke:#333,stroke-width:2px;

  %% Click Events
  click A "https://github.com/determined22/zh-ner-tf/tree/master/data_path/original"
  click B "https://github.com/determined22/zh-ner-tf/tree/master/data_path/train_data"
  click C "https://github.com/determined22/zh-ner-tf/blob/master/data_path/word2id.pkl"
  click D "https://github.com/determined22/zh-ner-tf/blob/master/data.py"
  click H "https://github.com/determined22/zh-ner-tf/blob/master/model.py"
  click I "https://github.com/determined22/zh-ner-tf/blob/master/main.py"
  click J "https://github.com/determined22/zh-ner-tf/tree/master/data_path_save"
  click K "https://github.com/determined22/zh-ner-tf/blob/master/eval.py"
  click L "https://github.com/determined22/zh-ner-tf/blob/master/conlleval_rev.pl"
  click N "https://github.com/determined22/zh-ner-tf/tree/master/pics"
  click O "https://github.com/determined22/zh-ner-tf/blob/master/utils.py"
```
