How inspection-ai/japanese-toxic-dataset works
1. Repository purpose
- This is a research dataset repository, not an application or service.
- It packages a Japanese toxicity labeling schema and a sample annotated dataset used in the paper *“Proposal and Evaluation of Japanese Toxicity Schema.”*
2. Core artifacts
- schema.md defines the annotation taxonomy and labeling rules.
- data/subset.csv contains the annotated sentence-level data following that schema.
- README.md and README_ja.md serve as the entry documentation in English and Japanese.
3. Primary architectural boundary
- The repo has a clean split between:
- Specification layer: the schema document.
- Data layer: the CSV dataset.
- There is no runtime backend, model, API, or UI layer.
4. Annotation model
- Each row in data/subset.csv represents a single sentence with a unique id.
- The dataset encodes two kinds of labels:
- Toxicity level: Not Toxic, Hard to Say, Toxic, Very Toxic
- Toxicity category: category_卑語, category_差別, category_迷惑行為, category_猥褻, category_出会い・プライバシー侵害, category_違法行為, category_偏向表現
5. Data schema structure
- The CSV is a multi-label annotation table: one sentence can have values across multiple toxicity categories.
- annotation_num captures how many annotators contributed to the label set, which is important for interpreting agreement and confidence.
6. Semantics of the labels
- The toxicity level columns represent a severity scale.
- The category columns represent why a sentence is toxic, separating general severity from specific harmful content types.
- This makes the dataset suitable for both classification and schema evaluation tasks.
7. Documentation flow
- README.md is the main entry point and points users to the schema and dataset.
- schema.md is the authoritative source for how annotations were defined.
- README_ja.md and schema_ja.md mirror the documentation for Japanese readers.
8. Likely usage pattern
- An engineer or researcher would first read the schema, then load data/subset.csv for analysis, model training, or evaluation.
- The repo is designed to support benchmarking toxicity detection in Japanese rather than operational deployment.
9. Technology footprint
- The only concrete data format is CSV, making the dataset easy to ingest from any language or ML stack.
- There are no repository-specific build tools, dependencies, or infrastructure components.
10. What an architecture diagram should show
- A documentation/specification node for the schema.
- A dataset node for the CSV annotations.
- A linkage from the schema to the dataset, indicating the schema governs the labels in the data.
- Optional bilingual documentation nodes for English and Japanese access.
Main entry points: schema.md schema.md · subset.csv data/subset.csv · README.md README.md · README_ja.md README_ja.md
Components 7
Documentation
Entry points for English and Japanese readers.
- README.md · Main English entry point linking the schema and dataset.
README.md - README_ja.md · Japanese entry point mirroring the main documentation.
README_ja.md
Specification
Annotation taxonomy and labeling rules.
- schema.md · Authoritative definition of the toxicity taxonomy and annotation rules.
schema.md - schema_ja.md · Japanese version of the annotation schema documentation.
schema_ja.md - Label system · Two-part labeling model combining toxicity severity with multi-label toxicity categories.
Dataset
Annotated sentence-level sample data.
- data/subset.csv · Sentence-level annotated sample dataset with ids, annotator count, severity, and category labels.
data/subset.csv - Sentence rows · One row per sentence, carrying a unique id and multi-label annotation values.
Connections 10
- README.md → schema.md: points to · English documentation directs readers to the canonical schema.
- README.md → data/subset.csv: points to · English documentation introduces the sample dataset.
- README_ja.md → schema_ja.md: points to · Japanese documentation directs readers to the Japanese schema.
- README_ja.md → data/subset.csv: points to · Japanese documentation introduces the sample dataset.
- schema_ja.md → schema.md: mirrors · Japanese schema mirrors the English authoritative specification.
- schema.md → Label system: defines · The schema defines the severity scale and toxicity categories.
- Label system → Sentence rows: governs · The label model determines how each sentence row is annotated.
- data/subset.csv → Sentence rows: implements · The CSV stores the sentence-level annotation table.
- Sentence rows → schema.md: conforms to · Each row follows the annotation rules and label definitions in the schema.
- data/subset.csv → schema.md: validated by · Dataset labels are interpreted against the schema.