# yassinekaibi/traitement-donnees architecture

> Architecture diagram of the GitHub repository yassinekaibi/traitement-donnees, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yassinekaibi/traitement-donnees
- Repository: https://github.com/yassinekaibi/traitement-donnees

## Overview

This repository builds a harmonized Arabic emotion corpus from poetry and two tweet datasets, then produces exploratory statistics and figures from the saved corpus. Its main workflow is source-specific loading and cleaning, label harmonization, Arabic-text normalization and feature extraction, corpus export, and EDA. The graph separates those domain stages and includes the shared download/deduplication layer. README is empty; relationships are based on the shown source excerpts and SOURCE INDEX, so unsampled behavior is not inferred.

Main entry points: [`src/integrate.py`](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py) (Label harmonization), [`src/config.py`](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/config.py) (Dataset configuration), [`src/eda.py`](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/eda.py) (Exploratory analysis), [`src/sources/poems.py`](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/poems.py) (Poetry loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source ingestion"]
  node_poems["Poetry loader<br/>[poems.py]"]
  node_etda["ETDA loader<br/>[etda.py]"]
  node_semeval["SemEval loader<br/>[semeval.py]"]
  node_source_clean["Source deduplication<br/>[common.py]"]
end

subgraph group_prepare["Corpus preparation"]
  node_harmonize["Label harmonization<br/>[integrate.py]"]
  node_normalize["Text normalization<br/>[integrate.py]"]
  node_poem_cap["Poem share cap<br/>[integrate.py]"]
  node_features["Text features<br/>[integrate.py]"]
  node_corpus[("Corpus exports<br/>[integrate.py]")]
end

subgraph group_analysis["Corpus analysis"]
  node_eda["Exploratory analysis<br/>[eda.py]"]
  node_reports["Stats and figures<br/>[eda.py]"]
end

subgraph group_shared["Shared runtime"]
  node_config["Dataset configuration<br/>[config.py]"]
  node_arabic["Arabic text tools<br/>[arabic.py]"]
end

node_user(("Researcher"))
node_datasets["Dataset hosts"]

node_user -.->|"runs pipeline"| node_harmonize
node_poems -->|"reads settings"| node_config
node_etda -->|"reads settings"| node_config
node_semeval -->|"reads settings"| node_config
node_poems -->|"deduplicates with"| node_source_clean
node_etda -->|"deduplicates with"| node_source_clean
node_semeval -->|"deduplicates with"| node_source_clean
node_poems -->|"downloads from"| node_datasets
node_etda -->|"downloads from"| node_datasets
node_semeval -->|"downloads from"| node_datasets
node_config -->|"defines URLs"| node_datasets
node_harmonize -->|"loads"| node_poems
node_harmonize -->|"loads"| node_etda
node_harmonize -->|"loads"| node_semeval
node_harmonize -->|"maps labels"| node_config
node_harmonize -->|"passes rows"| node_normalize
node_normalize -->|"normalizes with"| node_arabic
node_harmonize -->|"caps poems"| node_poem_cap
node_poem_cap -->|"passes rows"| node_features
node_features -->|"removes stopwords"| node_arabic
node_features -->|"uses settings"| node_config
node_features -->|"writes CSVs"| node_corpus
node_user -.->|"runs analysis"| node_eda
node_eda -->|"reads corpus"| node_corpus
node_eda -->|"reads paths"| node_config
node_eda -->|"writes stats and figures"| node_reports

click node_poems "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/poems.py"
click node_etda "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/etda.py"
click node_semeval "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/semeval.py"
click node_source_clean "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/common.py"
click node_config "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/config.py"
click node_harmonize "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py"
click node_normalize "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py"
click node_arabic "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/arabic.py"
click node_poem_cap "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py"
click node_features "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py"
click node_corpus "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py"
click node_eda "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/eda.py"
click node_reports "https://github.com/yassinekaibi/traitement-donnees/blob/main/src/eda.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_poems,node_etda,node_semeval,node_source_clean toneBlue
class node_harmonize,node_normalize,node_poem_cap,node_features,node_corpus toneAmber
class node_eda,node_reports toneMint
class node_config,node_arabic toneRose
class node_user,node_datasets toneIndigo
```

## Components

### Source ingestion

- [Poetry loader](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/poems.py): `src/sources/poems.py`
- [ETDA loader](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/etda.py): `src/sources/etda.py`
- [SemEval loader](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/semeval.py): `src/sources/semeval.py`
- [Source deduplication](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/sources/common.py): `src/sources/common.py`

### Corpus preparation

- [Label harmonization](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py): `src/integrate.py`
- [Text normalization](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py): `src/integrate.py`
- [Poem share cap](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py): `src/integrate.py`
- [Text features](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py): `src/integrate.py`
- [Corpus exports](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/integrate.py): `src/integrate.py`

### Corpus analysis

- [Exploratory analysis](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/eda.py): `src/eda.py`
- [Stats and figures](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/eda.py): `src/eda.py`

### Shared runtime

- [Dataset configuration](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/config.py): `src/config.py`
- [Arabic text tools](https://github.com/yassinekaibi/traitement-donnees/blob/main/src/arabic.py): `src/arabic.py`

### Other components

- **Researcher**
- **Dataset hosts**

## Connections

- Researcher → Label harmonization: runs pipeline
- Poetry loader → Dataset configuration: reads settings (evidence: `src/sources/poems.py`)
- ETDA loader → Dataset configuration: reads settings (evidence: `src/sources/etda.py`)
- SemEval loader → Dataset configuration: reads settings (evidence: `src/sources/semeval.py`)
- Poetry loader → Source deduplication: deduplicates with (evidence: `src/sources/poems.py`)
- ETDA loader → Source deduplication: deduplicates with (evidence: `src/sources/etda.py`)
- SemEval loader → Source deduplication: deduplicates with (evidence: `src/sources/semeval.py`)
- Poetry loader → Dataset hosts: downloads from (evidence: `src/sources/poems.py`)
- ETDA loader → Dataset hosts: downloads from (evidence: `src/sources/etda.py`)
- SemEval loader → Dataset hosts: downloads from (evidence: `src/sources/semeval.py`)
- Dataset configuration → Dataset hosts: defines URLs (evidence: `src/config.py`)
- Label harmonization → Poetry loader: loads (evidence: `src/integrate.py`)
- Label harmonization → ETDA loader: loads (evidence: `src/integrate.py`)
- Label harmonization → SemEval loader: loads (evidence: `src/integrate.py`)
- Label harmonization → Dataset configuration: maps labels (evidence: `src/integrate.py`)
- Label harmonization → Text normalization: passes rows (evidence: `src/integrate.py`)
- Text normalization → Arabic text tools: normalizes with (evidence: `src/integrate.py`)
- Label harmonization → Poem share cap: caps poems (evidence: `src/integrate.py`)
- Poem share cap → Text features: passes rows (evidence: `src/integrate.py`)
- Text features → Arabic text tools: removes stopwords (evidence: `src/integrate.py`)
- Text features → Dataset configuration: uses settings (evidence: `src/integrate.py`)
- Text features → Corpus exports: writes CSVs (evidence: `src/integrate.py`)
- Researcher → Exploratory analysis: runs analysis
- Exploratory analysis → Corpus exports: reads corpus (evidence: `src/eda.py`)
- Exploratory analysis → Dataset configuration: reads paths (evidence: `src/eda.py`)
- Exploratory analysis → Stats and figures: writes stats and figures (evidence: `src/eda.py`)
