# vera-bernhard/swisstextsdg architecture

> Architecture diagram of the GitHub repository vera-bernhard/swisstextsdg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vera-bernhard/swisstextsdg
- Repository: https://github.com/vera-bernhard/swisstextsdg

## Overview

##### Main Components
- Data Input and Storage:
- “data/raw”, “data/preprocessed”, and other data sources.
- Configuration files (e.g., sdg_label_mapping.json, sgd_label_subtargets.json, settings_dist.py) that are used to define mappings/settings.
- Data Processing and Preprocessing:
- Notebooks like preprocessing.ipynb and scripts in “scripts/” and “src/data/” (e.g., text_preprocessing.py, tokenizer.py, dataset.py) that handle cleaning, tokenizing, and data configuration.
- Model Training and Evaluation:
- “src/models/” splits into subfolders: Llama2, Mistral, RAG, ensemble, mbert, qlora.
- Each subdirectory contains scripts and notebooks for training, testing, and evaluation.
- Specific notebooks (e.g., eval_classification_with_Llama.ipynb, run_ensemble_predict.py) signify separate training/testing branches.
- Ensemble and Voting Mechanisms:
- Look at the ensemble directories under “results/” and “src/models/ensemble/” showing majority/soft voting implementations.
- Helpers and Utilities:
- “src/helpers” containing logging, model, path helpers, etc.
- Utility scripts in “data/” (e.g., extract_fulltext.py, helper.py) that support the data layer.
- Results and Analysis:
- “results/” directory with structured output such as CSV prediction logs.
- Notebooks in “scripts/” for further analysis (e.g., analyze_data.ipynb, confusion_matrices.ipynb).

##### Relationships and Interactions
- Data flows from the raw sources through the preprocessing modules:
- Raw data → Preprocessing (data cleaning, tokenizing) → Preprocessed files.
- Preprocessed data then feeds into various model-specific pipelines:
- Each model (Llama2, Mistral, etc.) reads preprocessed data, trains or performs predictions.
- The outputs from individual model runs are logged into the “results/” directory.
- Ensemble modules combine results from different models:
- Ensemble scripts in “src/models/ensemble” merge predictions (using majority or soft voting) and produce final outputs.
- Analysis notebooks consume both raw predictions and processed results for evaluation.
- Helper modules support each of these stages with shared utilities and configurations.

##### Architectural Patterns and Design Principles
- Modular Pipeline: A clear separation of concerns where data ingestion, preprocessing, model training/evaluation, and ensemble analysis are isolated, making it easier to update one section without impacting others.
- Experimentation Focus: Multiple directories for different model architectures and integration of ensemble methods indicate an extensible research pipeline.
- Reusability & Utility: The presence of helper modules for logging, paths, and seed management shows an emphasis on reproducibility.
- Integration with External Tools: The README shows guidelines for working in Colab/GitHub, suggesting that the design must support remote and collaborative development.

##### Relevant Technologies/Frameworks
- Python, Jupyter Notebooks, and Google Colab integration.
- Machine Learning frameworks (likely PyTorch for transformer models, as suggested by files named pytorch_model.py and run_transformer_training.py).
- Data format handling with JSONL, CSV, and Pickle.
- Possibly dependency management via pyproject.toml and requirements.txt.

- Data Inputs:
- Raw Data Files (from “data/raw”)
- Configuration Files (json and settings scripts)
- Preprocessing Layer:
- Preprocessor scripts and notebooks (“src/data/”, “preprocessing.ipynb”)
- Model Layers:
- Individual Model Pipelines: Llama2, Mistral, RAG, MBERT, QLORA, etc. (with scripts and notebooks indicating training and prediction)
- Ensemble Methods (both with and without QLORA, majority and soft voting approaches)
- Post-Processing and Evaluation:
- Results Folder: CSV logs and evaluation metrics
- Analysis Notebooks (from “scripts/”)
- Support Utilities:
- Helper Modules (“src/helpers”)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Input & Storage
    subgraph "Data Input & Storage"
        rawData["Raw Data"]
        preprocData["Preprocessed Data"]
        config1["sdg_label_mapping.json"]
        config2["sgd_label_subtargets.json"]
        config3["settings_dist.py"]
    end

    %% Data Processing & Preprocessing
    subgraph "Data Processing & Preprocessing"
        procNotebook["preprocessing.ipynb"]
        textPre["text_preprocessing.py"]
        tokenizer["tokenizer.py"]
        dataset["dataset.py"]
        preprocessor["preprocessor.py"]
    end

    %% Model Training & Evaluation
    subgraph "Model Training & Evaluation"
        M_Llama2["Llama2 Pipeline"]
        M_Mistral["Mistral Pipeline"]
        M_RAG["RAG Pipeline"]
        M_mBERT["mBERT Pipeline"]
        M_QLORA["QLORA Pipeline"]
    end

    %% Ensemble Methods
    subgraph "Ensemble Methods"
        E_Script["Ensemble Script"]
        E_Results["Ensemble Results"]
    end

    %% Helpers & Utilities
    subgraph "Helpers & Utilities"
        H_helpers["Helper Modules"]
        H_extract["extract_fulltext.py"]
        H_helper["helper.py"]
    end

    %% Results & Analysis
    subgraph "Results & Analysis"
        R_results["Results Directory"]
        A_analyze_data["analyze_data.ipynb"]
        A_analyze_correlations["analyze_correlations.ipynb"]
        A_confusion_matrices["confusion_matrices.ipynb"]
        A_analyze_prediction["analyze_prediction_log.py"]
        A_eval_transformer["evaluate_transformer_model.py"]
    end

    %% Data flow arrows
    rawData --> procNotebook
    config1 --> procNotebook
    config2 --> procNotebook
    config3 --> procNotebook

    procNotebook --> M_Llama2
    procNotebook --> M_Mistral
    procNotebook --> M_RAG
    procNotebook --> M_mBERT
    procNotebook --> M_QLORA

    M_Llama2 --> E_Script
    M_Mistral --> E_Script
    M_RAG --> E_Script
    M_mBERT --> E_Script
    M_QLORA --> E_Script

    E_Script --> E_Results
    E_Results --> R_results

    M_Llama2 --> R_results
    M_Mistral --> R_results
    M_RAG --> R_results
    M_mBERT --> R_results
    M_QLORA --> R_results

    R_results --> A_analyze_data
    R_results --> A_analyze_correlations
    R_results --> A_confusion_matrices
    R_results --> A_analyze_prediction
    R_results --> A_eval_transformer

    %% Helpers support (dashed arrows)
    H_helpers -.-> procNotebook
    H_helpers -.-> M_Llama2
    H_helpers -.-> M_Mistral
    H_helpers -.-> M_RAG
    H_helpers -.-> M_mBERT
    H_helpers -.-> M_QLORA

    H_extract -.-> procNotebook
    H_helper -.-> procNotebook

    %% Click Events for Data Input and Storage
    click rawData "https://github.com/vera-bernhard/swisstextsdg/tree/main/data/raw"
    click preprocData "https://github.com/vera-bernhard/swisstextsdg/tree/main/data/preprocessed"
    click config1 "https://github.com/vera-bernhard/swisstextsdg/blob/main/data/sdg_label_mapping.json"
    click config2 "https://github.com/vera-bernhard/swisstextsdg/blob/main/data/sgd_label_subtargets.json"
    click config3 "https://github.com/vera-bernhard/swisstextsdg/blob/main/data/settings_dist.py"

    %% Click Events for Data Processing and Preprocessing
    click procNotebook "https://github.com/vera-bernhard/swisstextsdg/blob/main/preprocessing.ipynb"
    click textPre "https://github.com/vera-bernhard/swisstextsdg/blob/main/src/data/text_preprocessing.py"
    click tokenizer "https://github.com/vera-bernhard/swisstextsdg/blob/main/src/data/tokenizer.py"
    click dataset "https://github.com/vera-bernhard/swisstextsdg/blob/main/src/data/dataset.py"
    click preprocessor "https://github.com/vera-bernhard/swisstextsdg/blob/main/src/data/preprocessor.py"

    %% Click Events for Model Training and Evaluation
    click M_Llama2 "https://github.com/vera-bernhard/swisstextsdg/tree/main/src/models/Llama2"
    click M_Mistral "https://github.com/vera-bernhard/swisstextsdg/tree/main/src/models/Mistral"
    click M_RAG "https://github.com/vera-bernhard/swisstextsdg/tree/main/src/models/RAG"
    click M_mBERT "https://github.com/vera-bernhard/swisstextsdg/tree/main/src/models/mbert"
    click M_QLORA "https://github.com/vera-bernhard/swisstextsdg/tree/main/src/models/qlora"

    %% Click Events for Ensemble Methods
    click E_Script "https://github.com/vera-bernhard/swisstextsdg/blob/main/src/models/ensemble/run_ensemble_predict.py"
    click E_Results "https://github.com/vera-bernhard/swisstextsdg/tree/main/results/ensemble_majority_voting"

    %% Click Events for Helpers & Utilities
    click H_helpers "https://github.com/vera-bernhard/swisstextsdg/tree/main/src/helpers"
    click H_extract "https://github.com/vera-bernhard/swisstextsdg/blob/main/data/extract_fulltext.py"
    click H_helper "https://github.com/vera-bernhard/swisstextsdg/blob/main/data/helper.py"

    %% Click Events for Results & Analysis
    click R_results "https://github.com/vera-bernhard/swisstextsdg/tree/main/results"
    click A_analyze_data "https://github.com/vera-bernhard/swisstextsdg/blob/main/scripts/analyze_data.ipynb"
    click A_analyze_correlations "https://github.com/vera-bernhard/swisstextsdg/blob/main/scripts/analyze_correlations.ipynb"
    click A_confusion_matrices "https://github.com/vera-bernhard/swisstextsdg/blob/main/scripts/confusion_matrices.ipynb"
    click A_analyze_prediction "https://github.com/vera-bernhard/swisstextsdg/blob/main/scripts/analyze_prediction_log.py"
    click A_eval_transformer "https://github.com/vera-bernhard/swisstextsdg/blob/main/scripts/evaluate_transformer_model.py"

    %% Style classes
    class rawData,preprocData,config1,config2,config3 data
    class procNotebook,textPre,tokenizer,dataset,preprocessor processing
    class M_Llama2,M_Mistral,M_RAG,M_mBERT,M_QLORA model
    class E_Script,E_Results ensemble
    class H_helpers,H_extract,H_helper helpers
    class R_results,A_analyze_data,A_analyze_correlations,A_confusion_matrices,A_analyze_prediction,A_eval_transformer analysis

    %% Class definitions with colors
    classDef data fill:#FDF6E3,stroke:#333,stroke-width:2px;
    classDef processing fill:#E8F6FF,stroke:#333,stroke-width:2px;
    classDef model fill:#E2F0CB,stroke:#333,stroke-width:2px;
    classDef ensemble fill:#FFEDCC,stroke:#333,stroke-width:2px;
    classDef helpers fill:#FFF2CC,stroke:#333,stroke-width:2px;
    classDef analysis fill:#CCF5FF,stroke:#333,stroke-width:2px;
```
