How sanjanabhat846/code-mixed-emotion-intent works
This repository is a data-engineering foundation for Indian code-mixed emotion, intent, and token-language research—not a trained classifier or inference service. Its main workflow ingests benchmark or volunteer records, builds canonical data, preprocesses utterances, validates and audits datasets, and prepares human-annotation batches. The README documents raw, interim, annotated, and processed tiers; the tree also contains components for annotation agreement and data-quality analysis. Some subsystem links below are documented only by module references in the SOURCE INDEX; unsampled internals are not assumed.
Main entry points: pipeline.py Preprocessing pipeline · quality_report.py Quality reports · benchmark_parsers.py Benchmark parsers · dataset_loader.py Dataset loading
Components 29
Data collection
- Benchmark parsers
data_collection/benchmark_parsers.py - Dataset registry
data_collection/dataset_registry.py - Source validation
data_collection/source_validator.py - PII filtering
data_collection/privacy_filter.py - Canonical corpus
data_collection/corpus_builder.py - Dataset loading
data_collection/dataset_loader.py - Dataset tiers
data_collection/dataset_loader.py - Ingestion workflow
scripts/ingest_real_datasets.py
Text preprocessing
- Preprocessing pipeline
preprocessing/pipeline.py - Text normalization
preprocessing/normalization.py - Spelling variants
preprocessing/spelling_variants.py - Emoji handling
preprocessing/emoji_handler.py - Tokenization
preprocessing/tokenization.py - Script detection
preprocessing/script_detector.py - Token language ID
preprocessing/language_identifier.py - Transliteration
preprocessing/transliteration.py - Preprocessing workflow
scripts/run_preprocessing.py
Validation and quality
- Schema validation
data_validation/schema_validator.py - Duplicate detection
data_validation/duplicate_detector.py - Label validation
data_validation/label_validator.py - Language-pair checks
data_validation/language_pair_validator.py - Code-mixing metrics
data_validation/cmi_calculator.py - Quality reports
data_validation/quality_report.py - Validation workflow
scripts/validate_dataset.py
Human annotation
- Annotation batches
annotation/batch_generator.py - Annotator agreement
annotation/agreement_calculator.py - Annotation models
annotation/annotation_models.py - Batch generation
scripts/generate_actual_batches.py
Other components
- Researcher / curator
Connections 31
- Researcher / curator → Ingestion workflow: initiates ingestion
README.md - Ingestion workflow → Benchmark parsers: parses sources
scripts/ingest_real_datasets.py - Benchmark parsers → Canonical corpus: builds records
data_collection/benchmark_parsers.py - Benchmark parsers → PII filtering: filters submissions
data_collection/benchmark_parsers.py - Benchmark parsers → Code-mixing metrics: calculates CMI
data_collection/benchmark_parsers.py - Ingestion workflow → Dataset loading: loads and saves
scripts/ingest_real_datasets.py - Dataset loading → Dataset tiers: reads and writes
data_collection/dataset_loader.py - Dataset registry → Source validation
- Researcher / curator → Preprocessing workflow: starts preprocessing
README.md - Preprocessing workflow → Preprocessing pipeline: runs pipeline
scripts/run_preprocessing.py - Preprocessing pipeline → Text normalization: normalizes text
preprocessing/pipeline.py - Preprocessing pipeline → Spelling variants: standardizes spelling
preprocessing/pipeline.py - Preprocessing pipeline → Emoji handling: isolates emoji
preprocessing/pipeline.py - Preprocessing pipeline → Tokenization: tokenizes text
preprocessing/pipeline.py - Preprocessing pipeline → Script detection: detects scripts
preprocessing/pipeline.py - Preprocessing pipeline → Token language ID: tags tokens
preprocessing/pipeline.py - Preprocessing pipeline → Transliteration: optionally transliterates
preprocessing/pipeline.py - Researcher / curator → Validation workflow: requests validation
README.md - Validation workflow → Dataset loading: loads dataset
scripts/validate_dataset.py - Validation workflow → Quality reports: generates audit
scripts/validate_dataset.py - Quality reports → Schema validation: validates records
data_validation/quality_report.py - Quality reports → Duplicate detection: checks duplicates
data_validation/quality_report.py - Quality reports → Label validation: checks labels
data_validation/quality_report.py - Quality reports → Language-pair checks: checks language pairs
data_validation/quality_report.py - Quality reports → Code-mixing metrics: summarizes mixing
data_validation/quality_report.py - Researcher / curator → Batch generation: requests batches
README.md - Batch generation → Dataset loading: loads records
scripts/generate_actual_batches.py - Batch generation → Annotation batches: creates batches
scripts/generate_actual_batches.py - Annotation batches → Duplicate detection: groups near-duplicates
annotation/batch_generator.py - Annotation batches → Annotation models
- Annotator agreement → Annotation models