# ici-bci/eeg2text-cet-mae architecture

> Architecture diagram of the GitHub repository ici-bci/eeg2text-cet-mae, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ici-bci/eeg2text-cet-mae
- Repository: https://github.com/ici-bci/eeg2text-cet-mae

## Overview

##### Project type and purpose
- This is a research‐oriented Machine Learning pipeline for EEG‐to‐Text generation, centered on a Contrastive EEG‐Text Masked Autoencoder (CET-MAE).
- It provides end-to-end data ingestion (ZuCo EEG datasets), contrastive pretraining, fine-tuning for text decoding from EEG signals, and evaluation.

##### Main components
a. Data Layer
- Raw EEG “.mat” files (ZuCo v1 and v2) stored under `zuco_dataset/.../Matlab_files`
- Preprocessing scripts (`dataset_preprocess.sh`, `data_factory/data2pickle_v*.py`)
- Serialized pickled datasets via `dataset.py`
b. Model Layer
- `model_mae.py` – defines the masked autoencoder architecture
- `Multi_Stream_TransformerEncoder.py` – multi‐stream transformer encoder for EEG and text streams
- `modules.zip` – external modules (unzip per README)
c. Training/Pretraining Layer
- Pretraining scripts (`pre_train_eval_cet_mae_later_project_*.py`) driven by YAML configs in `config/`
- Optimizer definitions in `optim_new.py`
- Bash wrappers (`cet_mae_eeg2text*.sh`) to launch on specific GPU setups
d. Fine-Tuning/Evaluation Layer
- Fine-tuning scripts (`train_decoding_eeg_2_text_cet_mae.py`, `train_decoding_eeg_2_text_vanilla.py`)
- Evaluation scripts (`eval_decoding_eeg_2_text_cet_mae.py`, `eval_decoding_eeg_2_text_vanilla.py`, plus specialized eval for BART, vanilla)
- Metric wrappers (`hf_metrics.py`, `hf_metrics_vanilla.py`) leveraging Hugging Face metrics
e. Configuration & Environment
- `requirements.txt` – Python dependencies (PyTorch, Transformers, etc.)
- Conda environment instructions in README
- YAML config files under `config/` for experiment parameters

##### Relationships & Interactions
- Raw EEG data → Preprocessing scripts → Pickled datasets
- Pickled datasets + YAML config → Pretraining script → CET-MAE model → Checkpoints
- Checkpoints + fine-tuning script → EEG-to-Text decoder → Fine-tuned model
- Fine-tuned model + eval script → Metrics module → Performance reports
- Bash scripts orchestrate end-to-end runs on targeted GPU resources

##### Key Architectural Patterns & Principles
- Modular separation: Data ingestion, preprocessing, model definition, training, evaluation
- Config-driven experiments: YAML files decouple hyperparameters from code
- Reusable components: shared dataset and metrics wrappers across multiple model variants (vanilla, BART, T5, XLM-RoBERTa)
- Shell orchestration for reproducibility and multi-GPU scaling

##### Technology & Framework Highlights
- Python 3.8, PyTorch, Hugging Face Transformers & metrics
- MATLAB `.mat` file handling via custom preprocessing pipelines
- YAML for experiment configuration
- Bash for environment setup and job submission

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawData["Raw EEG .mat Files<br/>(ZuCo v1 & v2)"]:::data
        Preprocess["dataset_preprocess.sh"]:::preprocess
        DatasetCode["dataset.py"]:::preprocess
        Serialized["Serialized Pickle Dataset"]:::data
    end

    RawData -->|".mat → pickle"| Preprocess
    Preprocess -->|"calls data2pickle"| DatasetCode
    DatasetCode -->|"outputs .pkl"| Serialized

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        Modules["modules.zip"]:::external
        Autoencoder["model_mae.py"]:::model
        Transformer["Multi_Stream_TransformerEncoder.py"]:::model
    end

    Modules -->|“unzip required”| Autoencoder
    Modules --> Transformer

    %% Training/Pretraining Layer
    subgraph "Pretraining Layer"
        direction TB
        Optimizer["optim_new.py"]:::training
        subgraph "Pretraining Scripts"
            direction TB
            PT1["pre_train_eval_cet_mae_later_project_2525.py"]:::training
            PT2["pre_train_eval_cet_mae_later_project_5025.py"]:::training
            PT3["pre_train_eval_cet_mae_later_project_5050.py"]:::training
            PT4["pre_train_eval_cet_mae_later_project_7550.py"]:::training
            PT5["pre_train_eval_cet_mae_later_project_7575.py"]:::training
            PTB["pre_train_eval_cet_mae_later_project_bart.py"]:::training
            PTT5a["pre_train_eval_cet_mae_later_project_flan_t5.py"]:::training
            PTT5b["pre_train_eval_cet_mae_later_project_t5.py"]:::training
            PTX["pre_train_eval_cet_mae_later_project_xlmroberta.py"]:::training
            PTCS1["pre_train_eval_cscl_cet_mae_later_project.py"]:::training
            PTCS2["pre_train_eval_cscl_cet_mae_later_project_scaling.py"]:::training
            PTCS3["pre_train_eval_cscl_cet_mae_pre_project.py"]:::training
        end
        Checkpoints["Model Checkpoints"]:::data
    end

    Serialized --> PT1
    Serialized --> PT2
    Serialized --> PT3
    Serialized --> PT4
    Serialized --> PT5
    Serialized --> PTB
    Serialized --> PTT5a
    Serialized --> PTT5b
    Serialized --> PTX
    Serialized --> PTCS1
    Serialized --> PTCS2
    Serialized --> PTCS3

    Autoencoder --> PT1
    Transformer --> PT1
    Optimizer --> PT1

    PT1 --> Checkpoints
    PT2 --> Checkpoints
    PT3 --> Checkpoints
    PT4 --> Checkpoints
    PT5 --> Checkpoints
    PTB --> Checkpoints
    PTT5a --> Checkpoints
    PTT5b --> Checkpoints
    PTX --> Checkpoints
    PTCS1 --> Checkpoints
    PTCS2 --> Checkpoints
    PTCS3 --> Checkpoints

    %% Fine-Tuning Layer
    subgraph "Fine-Tuning & Evaluation"
        direction TB
        subgraph "Fine-Tuning"
            direction TB
            FT1["train_decoding_eeg_2_text_cet_mae.py"]:::training
            FT2["train_decoding_eeg_2_text_vanilla.py"]:::training
            FineTuned["Fine-Tuned Model"]:::model
        end
        subgraph "Evaluation"
            direction TB
            Eval1["eval_decoding_eeg_2_text_cet_mae.py"]:::training
            Eval2["eval_decoding_eeg_2_text_cet_mae_bart.py"]:::training
            Eval3["eval_decoding_eeg_2_text_vanilla.py"]:::training
            EvalGen["eval.py"]:::training
            subgraph "Metrics Wrappers"
                direction TB
                HF1["hf_metrics.py"]:::training
                HF2["hf_metrics_vanilla.py"]:::training
            end
            Report["Performance Report"]:::data
        end
    end

    Checkpoints --> FT1
    Checkpoints --> FT2
    FT1 --> FineTuned
    FT2 --> FineTuned

    FineTuned --> Eval1
    FineTuned --> Eval2
    FineTuned --> Eval3
    FineTuned --> EvalGen

    Eval1 --> HF1
    Eval2 --> HF1
    Eval3 --> HF2
    EvalGen --> HF1
    HF1 --> Report
    HF2 --> Report

    %% Configuration & Orchestration
    subgraph "Config & Orchestration"
        direction TB
        ConfigYAML["config/*.yaml"]:::config
        Req["requirements.txt"]:::config
        Readme["README.md"]:::config
        subgraph "Bash Orchestrator"
            direction TB
            SH1["cet_mae_eeg2text.sh"]:::orchestrator
            SH2["cet_mae_eeg2text_gpu2.sh"]:::orchestrator
            SH3["cet_mae_eeg2text_gpu2_2525.sh"]:::orchestrator
            SH4["cet_mae_eeg2text_gpu2_5025.sh"]:::orchestrator
            SH5["cet_mae_eeg2text_gpu2_7550.sh"]:::orchestrator
            SH6["cet_mae_eeg2text_gpu2_7575.sh"]:::orchestrator
            SH7["cet_mae_eeg2text_gpu3.sh"]:::orchestrator
            SH8["cet_mae_eeg2text_gpu4.sh"]:::orchestrator
        end
    end

    ConfigYAML --> PT1
    ConfigYAML --> FT1
    ConfigYAML --> Eval1

    Readme -->|setup env| Req

    SH1 -.-> PT1
    SH1 -.-> FT1
    SH1 -.-> Eval1
    SH2 -.-> PT2
    SH3 -.-> PT1
    SH4 -.-> PT2
    SH5 -.-> PT4
    SH6 -.-> PT5
    SH7 -.-> PT1
    SH8 -.-> PT1

    %% Styles & Classes
    classDef data fill:#ADD8E6,stroke:#000;
    classDef preprocess fill:#90EE90,stroke:#000;
    classDef model fill:#FFA500,stroke:#000;
    classDef training fill:#DDA0DD,stroke:#000;
    classDef orchestrator fill:#D3D3D3,stroke:#000;
    classDef config fill:#F0E68C,stroke:#000;
    classDef external fill:#FFE4E1,stroke:#000;

    %% Click Events
    click Preprocess "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/dataset_preprocess.sh"
    click DatasetCode "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/dataset.py"
    click Autoencoder "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/model_mae.py"
    click Transformer "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/Multi_Stream_TransformerEncoder.py"
    click Modules "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/modules.zip"
    click Optimizer "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/optim_new.py"
    click PT1 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_2525.py"
    click PT2 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_5025.py"
    click PT3 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_5050.py"
    click PT4 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_7550.py"
    click PT5 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_7575.py"
    click PTB "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_bart.py"
    click PTT5a "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_flan_t5.py"
    click PTT5b "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_t5.py"
    click PTX "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cet_mae_later_project_xlmroberta.py"
    click PTCS1 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cscl_cet_mae_later_project.py"
    click PTCS2 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cscl_cet_mae_later_project_scaling.py"
    click PTCS3 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/pre_train_eval_cscl_cet_mae_pre_project.py"
    click SH1 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text.sh"
    click SH2 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text_gpu2.sh"
    click SH3 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text_gpu2_2525.sh"
    click SH4 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text_gpu2_5025.sh"
    click SH5 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text_gpu2_7550.sh"
    click SH6 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text_gpu2_7575.sh"
    click SH7 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text_gpu3.sh"
    click SH8 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/cet_mae_eeg2text_gpu4.sh"
    click FT1 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/train_decoding_eeg_2_text_cet_mae.py"
    click FT2 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/train_decoding_eeg_2_text_vanilla.py"
    click Eval1 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/eval_decoding_eeg_2_text_cet_mae.py"
    click Eval2 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/eval_decoding_eeg_2_text_cet_mae_bart.py"
    click Eval3 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/eval_decoding_eeg_2_text_vanilla.py"
    click EvalGen "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/eval.py"
    click HF1 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/hf_metrics.py"
    click HF2 "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/hf_metrics_vanilla.py"
    click Req "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/requirements.txt"
    click Readme "https://github.com/ici-bci/eeg2text-cet-mae/blob/main/README.md"
```
