How neuspeech/eeg-to-text works
Project type and purpose
• This is a research-oriented machine-learning experiment pipeline for evaluating and correcting “EEG-to-Text” and zero-shot sentiment classification models. It implements data preparation, model training (decoding and sentiment), evaluation (using corrected generation), and result storage for academic analysis.
File structure
• Top-level scripts:
• config.py – global settings (paths, hyperparameters)
• data.py – high-level dataset loading interfaces
• train_decoding.py / train_sentiment_*.py – Python entry points to kick off training
• model_decoding.py / model_sentiment.py – model definitions and wrappers (using Transformers)
• eval_decoding.py / eval_sentiment.py – evaluation code (with corrected model.generate calls)
• util/ – helper modules for constructing and loading datasets (from .mat to pickle), sentiment label extraction, data-loading utilities
• scripts/ – shell scripts orchestrating multi-step pipelines (dataset preparation, training splits, evaluation runs)
• results/ – plain-text outputs of each experiment
• environment.yml – Conda environment specification for reproducibility
README for additional insights
• Purpose: correct earlier inadvertent use of teacher forcing, switch to proper generation-based evaluation
• Architecture: single-stage seq2seq (T5-based) EEG-to-Text model and a sentiment classifier
• Workflow overview:
a. raw EEG + text data → preprocessing (util, scripts/prepare_dataset.sh)
b. train decoder (scripts/train_decoding*.sh → train_decoding.py/model_decoding.py)
c. train sentiment baseline/classifier (scripts/train_text_sentiment_classifier.sh, train_sentiment_*.py)
d. evaluate both (scripts/eval_decoding_*.sh → eval_decoding.py; scripts/eval_sentiment_zeroshot_pipeline.sh → eval_sentiment.py)
e. collect results in results/
1) Data Preparation
– util/construct_dataset_*.py
– scripts/prepare_dataset.sh
– Output: preprocessed EEG embeddings, text targets, sentiment labels
2) Model Training
– EEG-to-Text Decoder ([model_decoding.py](#), train_decoding.py, scripts/train_decoding*.sh)
– Sentiment Classifier ([model_sentiment.py](#), train_sentiment_*.py, scripts/train_text_sentiment_classifier.sh)
3) Model Evaluation
– Decoding evaluation (eval_decoding.py, scripts/eval_decoding_*.sh)
– Sentiment evaluation (eval_sentiment.py, scripts/eval_sentiment_zeroshot_pipeline.sh)
4) Results Management
– results/ directory with .txt outcome files
5) Environment & Dependencies
– environment.yml
– Python libraries: PyTorch, HuggingFace Transformers, NumPy, SciPy
• Raw data → Data Preparation → Preprocessed dataset → Training components
• Trained models → Model Evaluation → Prediction outputs → Results Management
• config.py & environment.yml → feed into all components for consistent parameters and dependencies
c. Architectural patterns / design principles:
– Modular scripts for separation of concerns (prepare/train/evaluate)
– Reusable utility functions in util/ for dataset operations
– Shell-script orchestration for reproducible experiment runs and variant sweeps
– Clear separation between decoding and sentiment tasks
d. Key technologies / frameworks:
– Python 3.x, Conda environment (environment.yml)
– PyTorch + HuggingFace Transformers (T5-based model.generate)
– MATLAB-to-Python dataset conversion (util/construct_dataset_mat_to_pickle_v?.py)
Detailed yet concise capture
• Component is clearly named and all critical interactions are visible
• External references: original Wang code (for historical context), arXiv/IJCAI papers