How ze33m/word2vec works
This repository builds and trains a Word2Vec model with negative sampling from the Taiga text corpus. The documented workflow cleans text, maps tokens to IDs, creates target-context pairs, prepares negative-sampling probabilities, streams pair shards into training, and saves model checkpoints; training also uploads checkpoints to S3. The model implements both the training loss and nearest-neighbor search. Connections follow the sampled source and README; the README documents stages whose implementation details were not sampled.
Main entry points: train.py Training loop · convert_to_int.py Integer dataset · dataset.py Pair streaming · prep.py Text preprocessing
Components 13
Corpus preparation
- Text preprocessing
prep.py - Token dataset
prep.py - Token encoding
convert_to_int.py - Integer dataset
convert_to_int.py
Training data
- Pair generation
pairs_dataset.py - Pair shards
- Negative probabilities
unigram_table.py - Pair streaming
dataset.py
Model training
- Word2Vec model
word2vec.py - Training loop
train.py
Artifacts and storage
- Model checkpoints
- S3 storage
s3con.py
Other components
- Taiga corpus
Connections 13
- Taiga corpus → Text preprocessing: loads corpus
prep.py - Text preprocessing → Token dataset: saves tokens
prep.py - Token dataset → Token encoding: maps tokens
README.md - Token encoding → Integer dataset: saves IDs
convert_to_int.py - Integer dataset → Pair generation: reads tokens
README.md - Pair generation → Pair shards: writes pairs
pairs_dataset.py - Integer dataset → Negative probabilities: counts tokens
unigram_table.py - Negative probabilities → Pair streaming: provides probabilities
README.md - Pair shards → Pair streaming: reads batches
dataset.py - Pair streaming → Training loop: yields batches
train.py - Training loop → Word2Vec model: computes loss
train.py - Training loop → Model checkpoints: saves weights
train.py - Training loop → S3 storage: uploads checkpoint
train.py