# abhijitmishra/thought2text architecture

> Architecture diagram of the GitHub repository abhijitmishra/thought2text, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhijitmishra/thought2text
- Repository: https://github.com/abhijitmishra/thought2text

## Overview

Thought2Text trains an EEG-to-text system in three documented stages: an EEG encoder learns visual-feature alignment, an LLM is fine-tuned with image/text data, then the model is adapted to generate text from EEG embeddings. Inference loads the trained model and EEG dataset, generates captions, and writes results for evaluation. The graph preserves the README’s evaluation and alternative inference settings; source-level links are limited to relationships shown in excerpts or the SOURCE INDEX.

Main entry points: [`model.py`](https://github.com/abhijitmishra/thought2text/blob/main/model.py) (EEG-LLM model), [`inference.py`](https://github.com/abhijitmishra/thought2text/blob/main/inference.py) (EEG caption inference), [`train_eeg_classifier.py`](https://github.com/abhijitmishra/thought2text/blob/main/train_eeg_classifier.py) (Encoder training), [`finetune_llm.py`](https://github.com/abhijitmishra/thought2text/blob/main/finetune_llm.py) (LLM fine-tuning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["EEG data"]
  node_eeg_data[("EEG dataset")]
  node_splits[("Dataset splits")]
  node_datautils["Dataset utilities<br/>[datautils.py]"]
  node_constants["Label mappings<br/>[constants.py]"]
end

subgraph group_encoder["EEG encoder"]
  node_encoder_training["Encoder training"]
  node_channelnet["ChannelNet<br/>[model.py]"]
  node_channel_layers["EEG network layers<br/>[layers.py]"]
  node_channel_config["Encoder configuration<br/>[config.py]"]
  node_loss["Training loss<br/>[loss.py]"]
end

subgraph group_training["Model training"]
  node_model_config["Model configuration<br/>[config.py]"]
  node_multimodal_model["EEG-LLM model<br/>[model.py]"]
  node_finetuning["LLM fine-tuning<br/>[finetune_llm.py]"]
  node_arguments["Run arguments<br/>[args.py]"]
end

subgraph group_generation["Generation and evaluation"]
  node_inference["EEG caption inference<br/>[inference.py]"]
  node_chance_inference["Object-only inference"]
  node_chance2_inference["EEG-only baseline"]
  node_eeg_only_inference["EEG-only inference"]
  node_results["Generated results"]
  node_nlg_evaluation["NLG evaluation"]
  node_expert_evaluation["GPT-4 evaluation"]
end

node_researcher(("Researcher"))

node_researcher -->|"trains encoder"| node_encoder_training
node_researcher -->|"fine-tunes model"| node_finetuning
node_researcher -->|"runs inference"| node_inference
node_encoder_training -->|"loads data"| node_datautils
node_encoder_training -->|"trains"| node_channelnet
node_encoder_training -->|"uses loss"| node_loss
node_channelnet -->|"uses layers"| node_channel_layers
node_channelnet -->|"uses config"| node_channel_config
node_datautils -->|"uses mappings"| node_constants
node_finetuning -->|"loads data"| node_datautils
node_finetuning -->|"trains model"| node_multimodal_model
node_finetuning -->|"reads arguments"| node_arguments
node_multimodal_model -->|"embeds EEG"| node_channelnet
node_inference -->|"loads data"| node_datautils
node_inference -->|"generates with"| node_multimodal_model
node_inference -->|"reads arguments"| node_arguments
node_inference -->|"writes captions"| node_results
node_chance_inference -->|"generates with"| node_multimodal_model
node_chance2_inference -->|"generates with"| node_multimodal_model
node_eeg_only_inference -->|"generates with"| node_multimodal_model
node_results -->|"evaluates metrics"| node_nlg_evaluation
node_results -->|"evaluates quality"| node_expert_evaluation

click node_datautils "https://github.com/abhijitmishra/thought2text/blob/main/datautils.py"
click node_constants "https://github.com/abhijitmishra/thought2text/blob/main/constants.py"
click node_encoder_training "https://github.com/abhijitmishra/thought2text/blob/main/train_eeg_classifier.py"
click node_channelnet "https://github.com/abhijitmishra/thought2text/blob/main/channelnet/model.py"
click node_channel_layers "https://github.com/abhijitmishra/thought2text/blob/main/channelnet/layers.py"
click node_channel_config "https://github.com/abhijitmishra/thought2text/blob/main/channelnet/config.py"
click node_loss "https://github.com/abhijitmishra/thought2text/blob/main/loss.py"
click node_model_config "https://github.com/abhijitmishra/thought2text/blob/main/config.py"
click node_multimodal_model "https://github.com/abhijitmishra/thought2text/blob/main/model.py"
click node_finetuning "https://github.com/abhijitmishra/thought2text/blob/main/finetune_llm.py"
click node_arguments "https://github.com/abhijitmishra/thought2text/blob/main/args.py"
click node_inference "https://github.com/abhijitmishra/thought2text/blob/main/inference.py"
click node_chance_inference "https://github.com/abhijitmishra/thought2text/blob/main/inference_chance.py"
click node_chance2_inference "https://github.com/abhijitmishra/thought2text/blob/main/inference_chance2.py"
click node_eeg_only_inference "https://github.com/abhijitmishra/thought2text/blob/main/inference_only_eeg.py"
click node_nlg_evaluation "https://github.com/abhijitmishra/thought2text/blob/main/eval/metrics_based_evaluation_notebook.ipynb"
click node_expert_evaluation "https://github.com/abhijitmishra/thought2text/blob/main/eval/GPT4_evaluation_of_EEG_results.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_eeg_data,node_splits,node_datautils,node_constants toneBlue
class node_encoder_training,node_channelnet,node_channel_layers,node_channel_config,node_loss toneAmber
class node_model_config,node_multimodal_model,node_finetuning,node_arguments toneMint
class node_inference,node_chance_inference,node_chance2_inference,node_eeg_only_inference,node_results,node_nlg_evaluation,node_expert_evaluation toneRose
class node_researcher toneIndigo
```

## Components

### EEG data

- **EEG dataset**
- **Dataset splits**
- [Dataset utilities](https://github.com/abhijitmishra/thought2text/blob/main/datautils.py): `datautils.py`
- [Label mappings](https://github.com/abhijitmishra/thought2text/blob/main/constants.py): `constants.py`

### EEG encoder

- [Encoder training](https://github.com/abhijitmishra/thought2text/blob/main/train_eeg_classifier.py): `train_eeg_classifier.py`
- [ChannelNet](https://github.com/abhijitmishra/thought2text/blob/main/channelnet/model.py): `channelnet/model.py`
- [EEG network layers](https://github.com/abhijitmishra/thought2text/blob/main/channelnet/layers.py): `channelnet/layers.py`
- [Encoder configuration](https://github.com/abhijitmishra/thought2text/blob/main/channelnet/config.py): `channelnet/config.py`
- [Training loss](https://github.com/abhijitmishra/thought2text/blob/main/loss.py): `loss.py`

### Model training

- [Model configuration](https://github.com/abhijitmishra/thought2text/blob/main/config.py): `config.py`
- [EEG-LLM model](https://github.com/abhijitmishra/thought2text/blob/main/model.py): `model.py`
- [LLM fine-tuning](https://github.com/abhijitmishra/thought2text/blob/main/finetune_llm.py): `finetune_llm.py`
- [Run arguments](https://github.com/abhijitmishra/thought2text/blob/main/args.py): `args.py`

### Generation and evaluation

- [EEG caption inference](https://github.com/abhijitmishra/thought2text/blob/main/inference.py): `inference.py`
- [Object-only inference](https://github.com/abhijitmishra/thought2text/blob/main/inference_chance.py): `inference_chance.py`
- [EEG-only baseline](https://github.com/abhijitmishra/thought2text/blob/main/inference_chance2.py): `inference_chance2.py`
- [EEG-only inference](https://github.com/abhijitmishra/thought2text/blob/main/inference_only_eeg.py): `inference_only_eeg.py`
- **Generated results**
- [NLG evaluation](https://github.com/abhijitmishra/thought2text/blob/main/eval/metrics_based_evaluation_notebook.ipynb): `eval/metrics_based_evaluation_notebook.ipynb`
- [GPT-4 evaluation](https://github.com/abhijitmishra/thought2text/blob/main/eval/GPT4_evaluation_of_EEG_results.ipynb): `eval/GPT4_evaluation_of_EEG_results.ipynb`

### Other components

- **Researcher**

## Connections

- Researcher → Encoder training: trains encoder (evidence: `README.md`)
- Researcher → LLM fine-tuning: fine-tunes model (evidence: `README.md`)
- Researcher → EEG caption inference: runs inference (evidence: `README.md`)
- Encoder training → Dataset utilities: loads data (evidence: `train_eeg_classifier.py`)
- Encoder training → ChannelNet: trains (evidence: `train_eeg_classifier.py`)
- Encoder training → Training loss: uses loss (evidence: `train_eeg_classifier.py`)
- ChannelNet → EEG network layers: uses layers (evidence: `channelnet/model.py`)
- ChannelNet → Encoder configuration: uses config (evidence: `channelnet/model.py`)
- Dataset utilities → Label mappings: uses mappings (evidence: `datautils.py`)
- LLM fine-tuning → Dataset utilities: loads data (evidence: `finetune_llm.py`)
- LLM fine-tuning → EEG-LLM model: trains model (evidence: `finetune_llm.py`)
- LLM fine-tuning → Run arguments: reads arguments (evidence: `finetune_llm.py`)
- EEG-LLM model → ChannelNet: embeds EEG (evidence: `model.py`)
- EEG caption inference → Dataset utilities: loads data (evidence: `inference.py`)
- EEG caption inference → EEG-LLM model: generates with (evidence: `inference.py`)
- EEG caption inference → Run arguments: reads arguments (evidence: `inference.py`)
- EEG caption inference → Generated results: writes captions (evidence: `README.md`)
- Object-only inference → EEG-LLM model: generates with (evidence: `inference_chance.py`)
- EEG-only baseline → EEG-LLM model: generates with (evidence: `inference_chance2.py`)
- EEG-only inference → EEG-LLM model: generates with (evidence: `inference_only_eeg.py`)
- Generated results → NLG evaluation: evaluates metrics (evidence: `README.md`)
- Generated results → GPT-4 evaluation: evaluates quality (evidence: `README.md`)
