# sanjanabhat846/code-mixed-emotion-intent architecture

> Architecture diagram of the GitHub repository sanjanabhat846/code-mixed-emotion-intent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanjanabhat846/code-mixed-emotion-intent
- Repository: https://github.com/sanjanabhat846/code-mixed-emotion-intent

## Overview

This repository is a data-engineering foundation for Indian code-mixed emotion, intent, and token-language research—not a trained classifier or inference service. Its main workflow ingests benchmark or volunteer records, builds canonical data, preprocesses utterances, validates and audits datasets, and prepares human-annotation batches. The README documents raw, interim, annotated, and processed tiers; the tree also contains components for annotation agreement and data-quality analysis. Some subsystem links below are documented only by module references in the SOURCE INDEX; unsampled internals are not assumed.

Main entry points: [`preprocessing/pipeline.py`](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/pipeline.py) (Preprocessing pipeline), [`data_validation/quality_report.py`](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/quality_report.py) (Quality reports), [`data_collection/benchmark_parsers.py`](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/benchmark_parsers.py) (Benchmark parsers), [`data_collection/dataset_loader.py`](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/dataset_loader.py) (Dataset loading)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data collection"]
  node_benchmark_parser["Benchmark parsers"]
  node_registry["Dataset registry"]
  node_source_check["Source validation"]
  node_privacy["PII filtering<br/>[privacy_filter.py]"]
  node_corpus["Canonical corpus<br/>[corpus_builder.py]"]
  node_loader["Dataset loading<br/>[dataset_loader.py]"]
  node_tiers[("Dataset tiers<br/>[dataset_loader.py]")]
  node_ingest_cli["Ingestion workflow"]
end

subgraph group_preprocess["Text preprocessing"]
  node_pipeline["Preprocessing pipeline<br/>[pipeline.py]"]
  node_normalize["Text normalization<br/>[normalization.py]"]
  node_spelling["Spelling variants"]
  node_emoji["Emoji handling<br/>[emoji_handler.py]"]
  node_tokenizer["Tokenization<br/>[tokenization.py]"]
  node_script_detect["Script detection<br/>[script_detector.py]"]
  node_language_id["Token language ID"]
  node_transliteration["Transliteration<br/>[transliteration.py]"]
  node_preprocess_cli["Preprocessing workflow"]
end

subgraph group_quality["Validation and quality"]
  node_schema["Schema validation"]
  node_duplicates["Duplicate detection"]
  node_labels["Label validation<br/>[label_validator.py]"]
  node_pair_validation["Language-pair checks"]
  node_cmi["Code-mixing metrics<br/>[cmi_calculator.py]"]
  node_quality_report["Quality reports<br/>[quality_report.py]"]
  node_validate_cli["Validation workflow"]
end

subgraph group_annotation["Human annotation"]
  node_batch["Annotation batches<br/>[batch_generator.py]"]
  node_agreement["Annotator agreement"]
  node_annot_models["Annotation models"]
  node_batch_cli["Batch generation"]
end

node_researcher(("Researcher / curator"))

node_researcher -->|"initiates ingestion"| node_ingest_cli
node_ingest_cli -->|"parses sources"| node_benchmark_parser
node_benchmark_parser -->|"builds records"| node_corpus
node_benchmark_parser -->|"filters submissions"| node_privacy
node_benchmark_parser -->|"calculates CMI"| node_cmi
node_ingest_cli -->|"loads and saves"| node_loader
node_loader -->|"reads and writes"| node_tiers
node_registry -.-> node_source_check
node_researcher -->|"starts preprocessing"| node_preprocess_cli
node_preprocess_cli -->|"runs pipeline"| node_pipeline
node_pipeline -->|"normalizes text"| node_normalize
node_pipeline -->|"standardizes spelling"| node_spelling
node_pipeline -->|"isolates emoji"| node_emoji
node_pipeline -->|"tokenizes text"| node_tokenizer
node_pipeline -->|"detects scripts"| node_script_detect
node_pipeline -->|"tags tokens"| node_language_id
node_pipeline -.->|"optionally transliterates"| node_transliteration
node_researcher -->|"requests validation"| node_validate_cli
node_validate_cli -->|"loads dataset"| node_loader
node_validate_cli -->|"generates audit"| node_quality_report
node_quality_report -->|"validates records"| node_schema
node_quality_report -->|"checks duplicates"| node_duplicates
node_quality_report -->|"checks labels"| node_labels
node_quality_report -->|"checks language pairs"| node_pair_validation
node_quality_report -->|"summarizes mixing"| node_cmi
node_researcher -->|"requests batches"| node_batch_cli
node_batch_cli -->|"loads records"| node_loader
node_batch_cli -->|"creates batches"| node_batch
node_batch -->|"groups near-duplicates"| node_duplicates
node_batch -.-> node_annot_models
node_agreement -.-> node_annot_models

click node_benchmark_parser "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/benchmark_parsers.py"
click node_registry "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/dataset_registry.py"
click node_source_check "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/source_validator.py"
click node_privacy "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/privacy_filter.py"
click node_corpus "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/corpus_builder.py"
click node_loader "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/dataset_loader.py"
click node_tiers "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/dataset_loader.py"
click node_ingest_cli "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/ingest_real_datasets.py"
click node_pipeline "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/pipeline.py"
click node_normalize "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/normalization.py"
click node_spelling "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/spelling_variants.py"
click node_emoji "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/emoji_handler.py"
click node_tokenizer "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/tokenization.py"
click node_script_detect "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/script_detector.py"
click node_language_id "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/language_identifier.py"
click node_transliteration "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/transliteration.py"
click node_preprocess_cli "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/run_preprocessing.py"
click node_schema "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/schema_validator.py"
click node_duplicates "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/duplicate_detector.py"
click node_labels "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/label_validator.py"
click node_pair_validation "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/language_pair_validator.py"
click node_cmi "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/cmi_calculator.py"
click node_quality_report "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/quality_report.py"
click node_validate_cli "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/validate_dataset.py"
click node_batch "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/annotation/batch_generator.py"
click node_agreement "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/annotation/agreement_calculator.py"
click node_annot_models "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/annotation/annotation_models.py"
click node_batch_cli "https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/generate_actual_batches.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_benchmark_parser,node_registry,node_source_check,node_privacy,node_corpus,node_loader,node_tiers,node_ingest_cli toneBlue
class node_pipeline,node_normalize,node_spelling,node_emoji,node_tokenizer,node_script_detect,node_language_id,node_transliteration,node_preprocess_cli toneAmber
class node_schema,node_duplicates,node_labels,node_pair_validation,node_cmi,node_quality_report,node_validate_cli toneMint
class node_batch,node_agreement,node_annot_models,node_batch_cli toneRose
class node_researcher toneIndigo
```

## Components

### Data collection

- [Benchmark parsers](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/benchmark_parsers.py): `data_collection/benchmark_parsers.py`
- [Dataset registry](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/dataset_registry.py): `data_collection/dataset_registry.py`
- [Source validation](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/source_validator.py): `data_collection/source_validator.py`
- [PII filtering](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/privacy_filter.py): `data_collection/privacy_filter.py`
- [Canonical corpus](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/corpus_builder.py): `data_collection/corpus_builder.py`
- [Dataset loading](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/dataset_loader.py): `data_collection/dataset_loader.py`
- [Dataset tiers](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_collection/dataset_loader.py): `data_collection/dataset_loader.py`
- [Ingestion workflow](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/ingest_real_datasets.py): `scripts/ingest_real_datasets.py`

### Text preprocessing

- [Preprocessing pipeline](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/pipeline.py): `preprocessing/pipeline.py`
- [Text normalization](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/normalization.py): `preprocessing/normalization.py`
- [Spelling variants](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/spelling_variants.py): `preprocessing/spelling_variants.py`
- [Emoji handling](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/emoji_handler.py): `preprocessing/emoji_handler.py`
- [Tokenization](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/tokenization.py): `preprocessing/tokenization.py`
- [Script detection](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/script_detector.py): `preprocessing/script_detector.py`
- [Token language ID](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/language_identifier.py): `preprocessing/language_identifier.py`
- [Transliteration](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/preprocessing/transliteration.py): `preprocessing/transliteration.py`
- [Preprocessing workflow](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/run_preprocessing.py): `scripts/run_preprocessing.py`

### Validation and quality

- [Schema validation](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/schema_validator.py): `data_validation/schema_validator.py`
- [Duplicate detection](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/duplicate_detector.py): `data_validation/duplicate_detector.py`
- [Label validation](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/label_validator.py): `data_validation/label_validator.py`
- [Language-pair checks](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/language_pair_validator.py): `data_validation/language_pair_validator.py`
- [Code-mixing metrics](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/cmi_calculator.py): `data_validation/cmi_calculator.py`
- [Quality reports](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/data_validation/quality_report.py): `data_validation/quality_report.py`
- [Validation workflow](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/validate_dataset.py): `scripts/validate_dataset.py`

### Human annotation

- [Annotation batches](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/annotation/batch_generator.py): `annotation/batch_generator.py`
- [Annotator agreement](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/annotation/agreement_calculator.py): `annotation/agreement_calculator.py`
- [Annotation models](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/annotation/annotation_models.py): `annotation/annotation_models.py`
- [Batch generation](https://github.com/sanjanabhat846/code-mixed-emotion-intent/blob/main/scripts/generate_actual_batches.py): `scripts/generate_actual_batches.py`

### Other components

- **Researcher / curator**

## Connections

- Researcher / curator → Ingestion workflow: initiates ingestion (evidence: `README.md`)
- Ingestion workflow → Benchmark parsers: parses sources (evidence: `scripts/ingest_real_datasets.py`)
- Benchmark parsers → Canonical corpus: builds records (evidence: `data_collection/benchmark_parsers.py`)
- Benchmark parsers → PII filtering: filters submissions (evidence: `data_collection/benchmark_parsers.py`)
- Benchmark parsers → Code-mixing metrics: calculates CMI (evidence: `data_collection/benchmark_parsers.py`)
- Ingestion workflow → Dataset loading: loads and saves (evidence: `scripts/ingest_real_datasets.py`)
- Dataset loading → Dataset tiers: reads and writes (evidence: `data_collection/dataset_loader.py`)
- Dataset registry → Source validation
- Researcher / curator → Preprocessing workflow: starts preprocessing (evidence: `README.md`)
- Preprocessing workflow → Preprocessing pipeline: runs pipeline (evidence: `scripts/run_preprocessing.py`)
- Preprocessing pipeline → Text normalization: normalizes text (evidence: `preprocessing/pipeline.py`)
- Preprocessing pipeline → Spelling variants: standardizes spelling (evidence: `preprocessing/pipeline.py`)
- Preprocessing pipeline → Emoji handling: isolates emoji (evidence: `preprocessing/pipeline.py`)
- Preprocessing pipeline → Tokenization: tokenizes text (evidence: `preprocessing/pipeline.py`)
- Preprocessing pipeline → Script detection: detects scripts (evidence: `preprocessing/pipeline.py`)
- Preprocessing pipeline → Token language ID: tags tokens (evidence: `preprocessing/pipeline.py`)
- Preprocessing pipeline → Transliteration: optionally transliterates (evidence: `preprocessing/pipeline.py`)
- Researcher / curator → Validation workflow: requests validation (evidence: `README.md`)
- Validation workflow → Dataset loading: loads dataset (evidence: `scripts/validate_dataset.py`)
- Validation workflow → Quality reports: generates audit (evidence: `scripts/validate_dataset.py`)
- Quality reports → Schema validation: validates records (evidence: `data_validation/quality_report.py`)
- Quality reports → Duplicate detection: checks duplicates (evidence: `data_validation/quality_report.py`)
- Quality reports → Label validation: checks labels (evidence: `data_validation/quality_report.py`)
- Quality reports → Language-pair checks: checks language pairs (evidence: `data_validation/quality_report.py`)
- Quality reports → Code-mixing metrics: summarizes mixing (evidence: `data_validation/quality_report.py`)
- Researcher / curator → Batch generation: requests batches (evidence: `README.md`)
- Batch generation → Dataset loading: loads records (evidence: `scripts/generate_actual_batches.py`)
- Batch generation → Annotation batches: creates batches (evidence: `scripts/generate_actual_batches.py`)
- Annotation batches → Duplicate detection: groups near-duplicates (evidence: `annotation/batch_generator.py`)
- Annotation batches → Annotation models
- Annotator agreement → Annotation models