# morizeyao/gpt2-chinese architecture

> Architecture diagram of the GitHub repository morizeyao/gpt2-chinese, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/morizeyao/gpt2-chinese
- Repository: https://github.com/morizeyao/gpt2-chinese

## Overview

##### Main System Components

The system should be represented with these key components:

a) Data Processing Layer
- Tokenization System (tokenizations/)
- Cache Management (cache/)
- Configuration Management (config/)

b) Core Model Layer
- Training Module
- Generation Module
- Evaluation Module

c) Interface Layer
- Script Interface
- Command Line Interface

##### Component Relationships

**[Input Data] → [Preprocessing] → [Training Pipeline] → [Model Storage] → [Generation Pipeline] → [Output Text]**

##### Detailed Architecture Elements

a) Data Processing Components:
- Multiple Tokenizer Options:
- BERT Tokenizer
- Word-level BERT Tokenizer
- BPE Tokenizer
- Vocabulary Management System
- Data Cache System

b) Model Components:
- Model Configuration Handler
- Training Engine
- Generation Engine
- Evaluation System

c) Utility Components:
- Script Management
- Configuration Management
- Cache Management

##### Specific Design Elements

Specialized components:
- Vocabulary Building Pipeline
- Model Configuration System
- Training Data Flow
- Generation Pipeline
- Evaluation System

##### Integration Points

Connections with:
- External data sources
- Model storage
- Output handlers
- Configuration systems

- Component Types
- Integration Points
- Configuration Points

This system design should emphasize the project's focus on Chinese language model training and generation, showing both the training and inference paths clearly while highlighting the unique aspects of Chinese text processing and multiple tokenization strategies.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Legend
        A1[Process Component]:::process
        A2[Data Store]:::data
        A3[Interface Component]:::interface
        A4[Model Component]:::model
        A5[Configuration Component]:::config
    end

    subgraph "Data Processing Layer"
        TS[Tokenization System]:::process
        BT["BERT Tokenizer"]:::process
        WBT["Word-level BERT Tokenizer"]:::process
        BPET["BPE Tokenizer"]:::process
        VM["Vocabulary Management"]:::data
        DC["Data Cache"]:::data
        
        TS --> BT
        TS --> WBT
        TS --> BPET
        BT & WBT & BPET --> VM
        VM --> DC
    end

    subgraph "Core Model Layer"
        MC["Model Configuration"]:::config
        TE["Training Engine"]:::model
        GE["Generation Engine"]:::model
        ES["Evaluation System"]:::model
        
        MC --> TE
        MC --> GE
        MC --> ES
    end

    subgraph "Interface Layer"
        SI["Script Interface"]:::interface
        CLI["Command Line Interface"]:::interface
        
        SI --> TE
        SI --> GE
        CLI --> TE
        CLI --> GE
    end

    InputData[/"Input Data"/]:::data --> TS
    TS --> TE
    TE --> ModelStorage[(Model Storage)]:::data
    ModelStorage --> GE
    GE --> OutputText[/"Output Text"/]:::data
    
    classDef process fill:#a8d1ff,stroke:#333,stroke-width:1px
    classDef data fill:#ffe0b2,stroke:#333,stroke-width:1px
    classDef interface fill:#fff59d,stroke:#333,stroke-width:1px
    classDef model fill:#a5d6a7,stroke:#333,stroke-width:1px
    classDef config fill:#ffccbc,stroke:#333,stroke-width:1px

    click TS "https://github.com/Morizeyao/GPT2-Chinese/tree/old_gpt_2_chinese_before_2021_4_22/tokenizations/"
    click BT "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/tokenizations/tokenization_bert.py"
    click WBT "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/tokenizations/tokenization_bert_word_level.py"
    click BPET "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/tokenizations/bpe_tokenizer.py"
    click VM "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/cache/make_vocab.py"
    click DC "https://github.com/Morizeyao/GPT2-Chinese/tree/old_gpt_2_chinese_before_2021_4_22/cache/"
    click MC "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/config/model_config.json"
    click TE "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/train.py"
    click GE "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/generate.py"
    click ES "https://github.com/Morizeyao/GPT2-Chinese/blob/old_gpt_2_chinese_before_2021_4_22/eval.py"
    click SI "https://github.com/Morizeyao/GPT2-Chinese/tree/old_gpt_2_chinese_before_2021_4_22/scripts/"
```
