How yassinekaibi/traitement-donnees works
This repository builds a harmonized Arabic emotion corpus from poetry and two tweet datasets, then produces exploratory statistics and figures from the saved corpus. Its main workflow is source-specific loading and cleaning, label harmonization, Arabic-text normalization and feature extraction, corpus export, and EDA. The graph separates those domain stages and includes the shared download/deduplication layer. README is empty; relationships are based on the shown source excerpts and SOURCE INDEX, so unsampled behavior is not inferred.
Main entry points: integrate.py Label harmonization · config.py Dataset configuration · eda.py Exploratory analysis · poems.py Poetry loader
Components 15
Source ingestion
- Poetry loader
src/sources/poems.py - ETDA loader
src/sources/etda.py - SemEval loader
src/sources/semeval.py - Source deduplication
src/sources/common.py
Corpus preparation
- Label harmonization
src/integrate.py - Text normalization
src/integrate.py - Poem share cap
src/integrate.py - Text features
src/integrate.py - Corpus exports
src/integrate.py
Corpus analysis
- Exploratory analysis
src/eda.py - Stats and figures
src/eda.py
Shared runtime
- Dataset configuration
src/config.py - Arabic text tools
src/arabic.py
Other components
- Researcher
- Dataset hosts
Connections 26
- Researcher → Label harmonization: runs pipeline
- Poetry loader → Dataset configuration: reads settings
src/sources/poems.py - ETDA loader → Dataset configuration: reads settings
src/sources/etda.py - SemEval loader → Dataset configuration: reads settings
src/sources/semeval.py - Poetry loader → Source deduplication: deduplicates with
src/sources/poems.py - ETDA loader → Source deduplication: deduplicates with
src/sources/etda.py - SemEval loader → Source deduplication: deduplicates with
src/sources/semeval.py - Poetry loader → Dataset hosts: downloads from
src/sources/poems.py - ETDA loader → Dataset hosts: downloads from
src/sources/etda.py - SemEval loader → Dataset hosts: downloads from
src/sources/semeval.py - Dataset configuration → Dataset hosts: defines URLs
src/config.py - Label harmonization → Poetry loader: loads
src/integrate.py - Label harmonization → ETDA loader: loads
src/integrate.py - Label harmonization → SemEval loader: loads
src/integrate.py - Label harmonization → Dataset configuration: maps labels
src/integrate.py - Label harmonization → Text normalization: passes rows
src/integrate.py - Text normalization → Arabic text tools: normalizes with
src/integrate.py - Label harmonization → Poem share cap: caps poems
src/integrate.py - Poem share cap → Text features: passes rows
src/integrate.py - Text features → Arabic text tools: removes stopwords
src/integrate.py - Text features → Dataset configuration: uses settings
src/integrate.py - Text features → Corpus exports: writes CSVs
src/integrate.py - Researcher → Exploratory analysis: runs analysis
- Exploratory analysis → Corpus exports: reads corpus
src/eda.py - Exploratory analysis → Dataset configuration: reads paths
src/eda.py - Exploratory analysis → Stats and figures: writes stats and figures
src/eda.py