How abhijitmishra/thought2text works
Thought2Text trains an EEG-to-text system in three documented stages: an EEG encoder learns visual-feature alignment, an LLM is fine-tuned with image/text data, then the model is adapted to generate text from EEG embeddings. Inference loads the trained model and EEG dataset, generates captions, and writes results for evaluation. The graph preserves the README’s evaluation and alternative inference settings; source-level links are limited to relationships shown in excerpts or the SOURCE INDEX.
Main entry points: model.py EEG-LLM model · inference.py EEG caption inference · train_eeg_classifier.py Encoder training · finetune_llm.py LLM fine-tuning
Components 21
EEG data
- EEG dataset
- Dataset splits
- Dataset utilities
datautils.py - Label mappings
constants.py
EEG encoder
- Encoder training
train_eeg_classifier.py - ChannelNet
channelnet/model.py - EEG network layers
channelnet/layers.py - Encoder configuration
channelnet/config.py - Training loss
loss.py
Model training
- Model configuration
config.py - EEG-LLM model
model.py - LLM fine-tuning
finetune_llm.py - Run arguments
args.py
Generation and evaluation
- EEG caption inference
inference.py - Object-only inference
inference_chance.py - EEG-only baseline
inference_chance2.py - EEG-only inference
inference_only_eeg.py - Generated results
- NLG evaluation
eval/metrics_based_evaluation_notebook.ipynb - GPT-4 evaluation
eval/GPT4_evaluation_of_EEG_results.ipynb
Other components
- Researcher
Connections 22
- Researcher → Encoder training: trains encoder
README.md - Researcher → LLM fine-tuning: fine-tunes model
README.md - Researcher → EEG caption inference: runs inference
README.md - Encoder training → Dataset utilities: loads data
train_eeg_classifier.py - Encoder training → ChannelNet: trains
train_eeg_classifier.py - Encoder training → Training loss: uses loss
train_eeg_classifier.py - ChannelNet → EEG network layers: uses layers
channelnet/model.py - ChannelNet → Encoder configuration: uses config
channelnet/model.py - Dataset utilities → Label mappings: uses mappings
datautils.py - LLM fine-tuning → Dataset utilities: loads data
finetune_llm.py - LLM fine-tuning → EEG-LLM model: trains model
finetune_llm.py - LLM fine-tuning → Run arguments: reads arguments
finetune_llm.py - EEG-LLM model → ChannelNet: embeds EEG
model.py - EEG caption inference → Dataset utilities: loads data
inference.py - EEG caption inference → EEG-LLM model: generates with
inference.py - EEG caption inference → Run arguments: reads arguments
inference.py - EEG caption inference → Generated results: writes captions
README.md - Object-only inference → EEG-LLM model: generates with
inference_chance.py - EEG-only baseline → EEG-LLM model: generates with
inference_chance2.py - EEG-only inference → EEG-LLM model: generates with
inference_only_eeg.py - Generated results → NLG evaluation: evaluates metrics
README.md - Generated results → GPT-4 evaluation: evaluates quality
README.md