# konsheng/sensitive-lexicon architecture

> Architecture diagram of the GitHub repository konsheng/sensitive-lexicon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/konsheng/sensitive-lexicon
- Repository: https://github.com/konsheng/sensitive-lexicon

## Overview

##### Project type and purpose
- This is an **open-source tool** (a reusable data library) called “Sensitive-lexicon.”
- Its primary function is to provide developers and content-moderation services with up-to-date Chinese sensitive-word lists for filtering or classification.

##### File structure
- Top level:
- LICENSE, README.md – project metadata
- ThirdPartyCompatibleFormats/… – adapters or export formats (e.g., TrChat JSON)
- Vocabulary/… – multiple plain-text lexicon files, each covering a category (pornography, violence, politics, etc.)
- There is no source code or service definitions here; this repo is purely data and format adapters.

##### README for additional insights
- No built-in server or client code; users clone or download the data and then integrate it into their own filtering pipeline.
- Integration is via simple file reads plus a matching algorithm chosen by the integrator (DFA, Trie, RegEx, etc.).

##### Main components
a. Lexicon Data Store
- All text files under Vocabulary/ and JSON files under ThirdPartyCompatibleFormats/TrChat
b. Adapter/Format Conversion Layer
- Code or scripts (in consuming projects) that parse plain text or JSON into in-memory structures
c. Data Structure Builder
- Modules that build a DFA or Trie from the raw word lists
d. Matching/Filtering Engine
- The runtime component that scans incoming text against the built structure to detect sensitive terms
e. Integration API / Client Application
- The host application (e.g., web-service, chat platform, moderation pipeline) that invokes the filtering engine
f. External Systems (optional)
- Logging/metrics, alerting, or database where flagged content is stored

##### Relationships and data flows
- Adapter Layer → Data Structure Builder (“parse & build Trie/DFA”)
- Data Structure Builder → Matching Engine (“initialized structure”)
- Client Application → Matching Engine (“submit text to filter”)
- Matching Engine → Client Application (“return matched results or flags”)
- (Optional) Matching Engine → External Systems (“log incidents”)

##### Architectural patterns and design principles
- **Separation of Concerns**: Data (word lists) is strictly separated from algorithmic modules (parsing, building, matching) and from the integration layer (your host app).
- **Extensibility**: The Adapter Layer is an extensibility point—new formats (CSV, database, remote service) can be added without touching the core engine.
- **Pluggable Matching Algorithms**: You could swap out a DFA for a regex engine or ML-based classifier.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Lexicon Data Store"
        Vocabulary["Vocabulary/*.txt"]:::data
        TrChat["TrChat JSON (SensitiveLexicon.json)"]:::data
    end

    Adapter["Adapter Layer\n(format converters)"]:::processing
    Builder["Data Structure Builder\n(Trie/DFA)"]:::processing
    Engine["Matching/Filtering Engine"]:::processing
    Client["Client Application\n(e.g., Moderation Service)"]:::integration
    External["External Systems\n(Logging & Metrics)"]:::integration

    Vocabulary -->|"load raw files"| Adapter
    TrChat -->|"load raw files"| Adapter
    Adapter -->|"parse & build Trie/DFA"| Builder
    Builder -->|"initialized structure"| Engine
    Client -->|"submit text to filter"| Engine
    Engine -->|"return matched results or flags"| Client
    Engine -->|"log incidents"| External

    click Vocabulary "https://github.com/konsheng/sensitive-lexicon/tree/main/Vocabulary/"
    click TrChat "https://github.com/konsheng/sensitive-lexicon/blob/main/ThirdPartyCompatibleFormats/TrChat/SensitiveLexicon.json"
    click Adapter "https://github.com/konsheng/sensitive-lexicon/tree/main/ThirdPartyCompatibleFormats/"

    classDef data fill:#ade8f4,stroke:#0077B6,color:#000;
    classDef processing fill:#caffbf,stroke:#2E8B57,color:#000;
    classDef integration fill:#ffe5b4,stroke:#DD6B20,color:#000;
```
