How konsheng/sensitive-lexicon works
Project type and purpose
• This is an open-source tool (a reusable data library) called “Sensitive-lexicon.”
• Its primary function is to provide developers and content-moderation services with up-to-date Chinese sensitive-word lists for filtering or classification.
File structure
• Top level:
• LICENSE, README.md – project metadata
• ThirdPartyCompatibleFormats/… – adapters or export formats (e.g., TrChat JSON)
• Vocabulary/… – multiple plain-text lexicon files, each covering a category (pornography, violence, politics, etc.)
• There is no source code or service definitions here; this repo is purely data and format adapters.
README for additional insights
• No built-in server or client code; users clone or download the data and then integrate it into their own filtering pipeline.
• Integration is via simple file reads plus a matching algorithm chosen by the integrator (DFA, Trie, RegEx, etc.).
Main components
a. Lexicon Data Store
• All text files under Vocabulary/ and JSON files under ThirdPartyCompatibleFormats/TrChat
b. Adapter/Format Conversion Layer
• Code or scripts (in consuming projects) that parse plain text or JSON into in-memory structures
c. Data Structure Builder
• Modules that build a DFA or Trie from the raw word lists
d. Matching/Filtering Engine
• The runtime component that scans incoming text against the built structure to detect sensitive terms
e. Integration API / Client Application
• The host application (e.g., web-service, chat platform, moderation pipeline) that invokes the filtering engine
f. External Systems (optional)
• Logging/metrics, alerting, or database where flagged content is stored
Relationships and data flows
• Adapter Layer → Data Structure Builder (“parse & build Trie/DFA”)
• Data Structure Builder → Matching Engine (“initialized structure”)
• Client Application → Matching Engine (“submit text to filter”)
• Matching Engine → Client Application (“return matched results or flags”)
• (Optional) Matching Engine → External Systems (“log incidents”)
Architectural patterns and design principles
• Separation of Concerns: Data (word lists) is strictly separated from algorithmic modules (parsing, building, matching) and from the integration layer (your host app).
• Extensibility: The Adapter Layer is an extensibility point—new formats (CSV, database, remote service) can be added without touching the core engine.
• Pluggable Matching Algorithms: You could swap out a DFA for a regex engine or ML-based classifier.