# carycha/jieba_fast_dat architecture

> Architecture diagram of the GitHub repository carycha/jieba_fast_dat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/carycha/jieba_fast_dat
- Repository: https://github.com/carycha/jieba_fast_dat

## Overview

##### Project type and purpose
- This is an open-source Python library (jieba_fast_dat) for high-performance Chinese word segmentation and part-of-speech tagging.
- It exposes a Python API (and a CLI entry point) that delegates core algorithms (Viterbi, DAT trie lookup) to optimized C++ code via pybind11.
- It relies on Double-Array Trie (DAT) dictionaries, memory-mapped caching, and drop-in replacement of jieba routines with higher performance.

##### File structure
- Top-level packaging:
- setup.py, pyproject.toml – build and packaging scripts.
- LICENSE, MANIFEST.in – distribution metadata.
- extra_dict/ – auxiliary dictionaries for offline use.
- jieba_fast_dat/ – main source tree:
– __main__.py – CLI launcher.
– _compat.py – Python version shims.
– analyse/ – TF-IDF and TextRank modules.
– finalseg/ – HMM-based final segmentation data files.
– posseg/ – POS tagging modules and HMM tables.
– source/ – C++ headers and implementation for core routines, exposed via pybind11.
- test/ – comprehensive test suite covering Python and C++ bindings, multithreading, caching, segmentation modes.

##### README for additional insights
- Key optimizations: DAT data structure, mmap-based persistent cache, C++ core algorithms, CPU-first design.
- Removed runtime dictionary addition in exchange for DAT-based static loading + automatic cache invalidation.
- Python ≥3.10 requirement.
- Presents usage examples for segmentation, search-engine mode, POS tagging, user dictionary loading.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["CLI Entry Point"]:::python
        API["API Entry Points"]:::python
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        COMPAT["_compat.py"]:::python
        subgraph "Analysis Modules"
            TFIDF["TF–IDF Module"]:::python
            TEXTRANK["TextRank Module"]:::python
            ANALYZER["Analyzer Orchestration"]:::python
        end
        subgraph "FinalSeg Data"
            FS_START["prob_start.p"]:::data
            FS_EMIT["prob_emit.p"]:::data
            FS_TRANS["prob_trans.p"]:::data
        end
        subgraph "POSSegging"
            VITERBI["POS Viterbi Logic"]:::python
            PS_START["posseg/prob_start.p"]:::data
            PS_EMIT["posseg/prob_emit.p"]:::data
            PS_TRANS["posseg/prob_trans.p"]:::data
            PS_CHAR["char_state_tab.p"]:::data
        end
    end

    %% Binding Layer
    BINDING["pybind11 Binding Layer"]:::cpp

    %% Core C++ Layer
    subgraph "Native Core"
        DARTS["Double-Array Trie (darts.h)"]:::cpp
        FUNCTIONS["C++ Core Algorithms"]:::cpp
    end

    %% Data Stores
    subgraph "Dictionaries & Cache"    
        BUILTIN_DICT["Built-in DAT Dictionary"]:::data
        subgraph "Auxiliary Dictionary Files"
            EXTRA_BIG["extra_dict/dict.txt.big"]:::data
            EXTRA_SMALL["extra_dict/dict.txt.small"]:::data
            STOP_WORDS["extra_dict/stop_words.txt"]:::data
            IDF_DATA["extra_dict/idf.txt.big"]:::data
        end
        CACHE_MANAGER["Mmap Cache Manager"]:::data
    end

    %% Build & Test
    subgraph "Build & Test"
        BUILD["Build & Packaging"]:::gray
        METADATA["LICENSE & MANIFEST.in"]:::gray
        TEST["Test Harness"]:::gray
    end

    %% Connections
    CLI -->|"invoke"| API
    API -->|"uses"| COMPAT
    API -->|"calls"| ANALYZER
    API -->|"calls"| TFIDF
    API -->|"calls"| TEXTRANK
    API -->|"calls"| VITERBI
    ANALYZER -->|"loads"| IDF_DATA
    API -->|"checks cache"| CACHE_MANAGER
    CACHE_MANAGER -->|"mmap load/build"| BUILTIN_DICT
    API -->|"pybind call"| BINDING
    BINDING -->|"access"| DARTS
    BINDING -->|"invoke"| FUNCTIONS
    FUNCTIONS -->|"lookup/traverse"| BUILTIN_DICT
    EXTRA_BIG & EXTRA_SMALL & STOP_WORDS -->|"raw data"| CACHE_MANAGER
    TEST -->|"executes APIs & bindings"| API
    TEST -->|"verifies"| FUNCTIONS
    BUILD -->|"packages"| API
    BUILD -->|"compiles"| FUNCTIONS
    BUILD -->|"includes"| METADATA

    %% Click Events
    click CLI "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/__main__.py"
    click API "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/__init__.py"
    click COMPAT "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/_compat.py"
    click TFIDF "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/analyse/tfidf.py"
    click TEXTRANK "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/analyse/textrank.py"
    click ANALYZER "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/analyse/analyzer.py"
    click FS_START "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/finalseg/prob_start.p"
    click FS_EMIT "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/finalseg/prob_emit.p"
    click FS_TRANS "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/finalseg/prob_trans.p"
    click VITERBI "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/posseg/viterbi.py"
    click PS_START "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/posseg/prob_start.p"
    click PS_EMIT "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/posseg/prob_emit.p"
    click PS_TRANS "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/posseg/prob_trans.p"
    click PS_CHAR "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/posseg/char_state_tab.p"
    click BINDING "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/source/jieba_fast_functions_pybind.cpp"
    click DARTS "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/source/darts.h"
    click BUILTIN_DICT "https://github.com/carycha/jieba_fast_dat/blob/master/jieba_fast_dat/dict.txt"
    click EXTRA_BIG "https://github.com/carycha/jieba_fast_dat/blob/master/extra_dict/dict.txt.big"
    click EXTRA_SMALL "https://github.com/carycha/jieba_fast_dat/blob/master/extra_dict/dict.txt.small"
    click STOP_WORDS "https://github.com/carycha/jieba_fast_dat/blob/master/extra_dict/stop_words.txt"
    click IDF_DATA "https://github.com/carycha/jieba_fast_dat/blob/master/extra_dict/idf.txt.big"
    click TEST "https://github.com/carycha/jieba_fast_dat/tree/master/test/"
    click BUILD "https://github.com/carycha/jieba_fast_dat/blob/master/setup.py"
    click METADATA "https://github.com/carycha/jieba_fast_dat/blob/master/MANIFEST.in"

    %% Styling
    classDef python fill:#AED6F1,stroke:#1B4F72,color:#1B2631
    classDef cpp fill:#A9DFBF,stroke:#145A32,color:#0B5345
    classDef data fill:#F9E79F,stroke:#B9770E,color:#784212
    classDef gray fill:#D5D8DC,stroke:#626567,color:#2C3E50
```
