# ze33m/word2vec architecture

> Architecture diagram of the GitHub repository ze33m/word2vec, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ze33m/word2vec
- Repository: https://github.com/ze33m/word2vec

## Overview

This repository builds and trains a Word2Vec model with negative sampling from the Taiga text corpus. The documented workflow cleans text, maps tokens to IDs, creates target-context pairs, prepares negative-sampling probabilities, streams pair shards into training, and saves model checkpoints; training also uploads checkpoints to S3. The model implements both the training loss and nearest-neighbor search. Connections follow the sampled source and README; the README documents stages whose implementation details were not sampled.

Main entry points: [`train.py`](https://github.com/ze33m/word2vec/blob/main/train.py) (Training loop), [`convert_to_int.py`](https://github.com/ze33m/word2vec/blob/main/convert_to_int.py) (Integer dataset), [`dataset.py`](https://github.com/ze33m/word2vec/blob/main/dataset.py) (Pair streaming), [`prep.py`](https://github.com/ze33m/word2vec/blob/main/prep.py) (Text preprocessing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Corpus preparation"]
  node_preprocess["Text preprocessing<br/>[prep.py]"]
  node_text_dataset[("Token dataset<br/>[prep.py]")]
  node_encode["Token encoding<br/>[convert_to_int.py]"]
  node_int_dataset[("Integer dataset<br/>[convert_to_int.py]")]
end

subgraph group_data["Training data"]
  node_pair_builder["Pair generation<br/>[pairs_dataset.py]"]
  node_pair_shards[("Pair shards")]
  node_prob_builder["Negative probabilities<br/>[unigram_table.py]"]
  node_stream["Pair streaming<br/>[dataset.py]"]
end

subgraph group_train["Model training"]
  node_model["Word2Vec model<br/>[word2vec.py]"]
  node_trainer["Training loop<br/>[train.py]"]
end

subgraph group_storage["Artifacts and storage"]
  node_checkpoint[("Model checkpoints")]
  node_s3[("S3 storage<br/>[s3con.py]")]
end

node_taiga[("Taiga corpus")]

node_taiga -->|"loads corpus"| node_preprocess
node_preprocess -->|"saves tokens"| node_text_dataset
node_text_dataset -->|"maps tokens"| node_encode
node_encode -->|"saves IDs"| node_int_dataset
node_int_dataset -->|"reads tokens"| node_pair_builder
node_pair_builder -->|"writes pairs"| node_pair_shards
node_int_dataset -->|"counts tokens"| node_prob_builder
node_prob_builder -->|"provides probabilities"| node_stream
node_pair_shards -->|"reads batches"| node_stream
node_stream -->|"yields batches"| node_trainer
node_trainer -->|"computes loss"| node_model
node_trainer -->|"saves weights"| node_checkpoint
node_trainer -->|"uploads checkpoint"| node_s3

click node_preprocess "https://github.com/ze33m/word2vec/blob/main/prep.py"
click node_text_dataset "https://github.com/ze33m/word2vec/blob/main/prep.py"
click node_encode "https://github.com/ze33m/word2vec/blob/main/convert_to_int.py"
click node_int_dataset "https://github.com/ze33m/word2vec/blob/main/convert_to_int.py"
click node_pair_builder "https://github.com/ze33m/word2vec/blob/main/pairs_dataset.py"
click node_prob_builder "https://github.com/ze33m/word2vec/blob/main/unigram_table.py"
click node_stream "https://github.com/ze33m/word2vec/blob/main/dataset.py"
click node_model "https://github.com/ze33m/word2vec/blob/main/word2vec.py"
click node_trainer "https://github.com/ze33m/word2vec/blob/main/train.py"
click node_s3 "https://github.com/ze33m/word2vec/blob/main/s3con.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_preprocess,node_text_dataset,node_encode,node_int_dataset toneBlue
class node_pair_builder,node_pair_shards,node_prob_builder,node_stream,node_taiga toneAmber
class node_model,node_trainer toneMint
class node_checkpoint,node_s3 toneRose
```

## Components

### Corpus preparation

- [Text preprocessing](https://github.com/ze33m/word2vec/blob/main/prep.py): `prep.py`
- [Token dataset](https://github.com/ze33m/word2vec/blob/main/prep.py): `prep.py`
- [Token encoding](https://github.com/ze33m/word2vec/blob/main/convert_to_int.py): `convert_to_int.py`
- [Integer dataset](https://github.com/ze33m/word2vec/blob/main/convert_to_int.py): `convert_to_int.py`

### Training data

- [Pair generation](https://github.com/ze33m/word2vec/blob/main/pairs_dataset.py): `pairs_dataset.py`
- **Pair shards**
- [Negative probabilities](https://github.com/ze33m/word2vec/blob/main/unigram_table.py): `unigram_table.py`
- [Pair streaming](https://github.com/ze33m/word2vec/blob/main/dataset.py): `dataset.py`

### Model training

- [Word2Vec model](https://github.com/ze33m/word2vec/blob/main/word2vec.py): `word2vec.py`
- [Training loop](https://github.com/ze33m/word2vec/blob/main/train.py): `train.py`

### Artifacts and storage

- **Model checkpoints**
- [S3 storage](https://github.com/ze33m/word2vec/blob/main/s3con.py): `s3con.py`

### Other components

- **Taiga corpus**

## Connections

- Taiga corpus → Text preprocessing: loads corpus (evidence: `prep.py`)
- Text preprocessing → Token dataset: saves tokens (evidence: `prep.py`)
- Token dataset → Token encoding: maps tokens (evidence: `README.md`)
- Token encoding → Integer dataset: saves IDs (evidence: `convert_to_int.py`)
- Integer dataset → Pair generation: reads tokens (evidence: `README.md`)
- Pair generation → Pair shards: writes pairs (evidence: `pairs_dataset.py`)
- Integer dataset → Negative probabilities: counts tokens (evidence: `unigram_table.py`)
- Negative probabilities → Pair streaming: provides probabilities (evidence: `README.md`)
- Pair shards → Pair streaming: reads batches (evidence: `dataset.py`)
- Pair streaming → Training loop: yields batches (evidence: `train.py`)
- Training loop → Word2Vec model: computes loss (evidence: `train.py`)
- Training loop → Model checkpoints: saves weights (evidence: `train.py`)
- Training loop → S3 storage: uploads checkpoint (evidence: `train.py`)
