# kyubyong/wordvectors architecture

> Architecture diagram of the GitHub repository kyubyong/wordvectors, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kyubyong/wordvectors
- Repository: https://github.com/kyubyong/wordvectors

## Overview

This repository provides a workflow for building multilingual word-vector models from Wikipedia dumps. Users obtain and extract a language’s dump, run corpus preparation, then train Word2Vec or fastText vectors; the README also links pre-trained models. The sampled Python code confirms corpus cleaning, sentence and word segmentation, and Word2Vec input/output behavior. The README documents the fastText branch, but its implementation was not sampled, so that branch is shown only at the documented level.

Main entry points: [`build_corpus.py`](https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py) (Build corpus), [`fasttext.sh`](https://github.com/kyubyong/wordvectors/blob/master/fasttext.sh) (fastText runner), [`make_wordvectors.py`](https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.py) (Word2Vec training), [`make_wordvectors.sh`](https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.sh) (Word2Vec runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_corpus["Corpus preparation"]
  node_extract["Extract dump texts"]
  node_prepare["Build corpus<br/>[build_corpus.py]"]
  node_clean["Clean markup<br/>[build_corpus.py]"]
  node_sentences["Segment sentences<br/>[build_corpus.py]"]
  node_words["Segment words<br/>[build_corpus.py]"]
  node_corpusfile["Language corpus<br/>[build_corpus.py]"]
end

subgraph group_training["Vector training"]
  node_word2vec["Word2Vec training"]
  node_wordscript["Word2Vec runner"]
  node_fastscript["fastText runner<br/>[fasttext.sh]"]
end

subgraph group_models["Vector artifacts"]
  node_vectors["Word2Vec vectors"]
  node_fastvectors["fastText vectors<br/>[fasttext.sh]"]
  node_premodels["Pre-trained models<br/>[README.md]"]
end

node_user(("User"))
node_wiki["Wikipedia dumps"]

node_user -->|"downloads"| node_wiki
node_user -->|"extracts"| node_extract
node_wiki -->|"provides dump"| node_extract
node_extract -->|"runs"| node_prepare
node_prepare -->|"cleans text"| node_clean
node_prepare -->|"segments"| node_sentences
node_prepare -->|"tokenizes"| node_words
node_prepare -->|"writes"| node_corpusfile
node_user -->|"runs"| node_wordscript
node_wordscript -.->|"runs"| node_word2vec
node_corpusfile -->|"reads"| node_word2vec
node_word2vec -->|"writes"| node_vectors
node_user -->|"runs"| node_fastscript
node_fastscript -.->|"uses corpus"| node_corpusfile
node_fastscript -->|"produces"| node_fastvectors
node_user -->|"downloads"| node_premodels

click node_prepare "https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py"
click node_clean "https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py"
click node_sentences "https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py"
click node_words "https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py"
click node_corpusfile "https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py"
click node_word2vec "https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.py"
click node_wordscript "https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.sh"
click node_fastscript "https://github.com/kyubyong/wordvectors/blob/master/fasttext.sh"
click node_vectors "https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.py"
click node_fastvectors "https://github.com/kyubyong/wordvectors/blob/master/fasttext.sh"
click node_premodels "https://github.com/kyubyong/wordvectors/blob/master/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extract,node_prepare,node_clean,node_sentences,node_words,node_corpusfile,node_user toneBlue
class node_word2vec,node_wordscript,node_fastscript toneAmber
class node_vectors,node_fastvectors,node_premodels toneMint
class node_wiki toneIndigo
```

## Components

### Corpus preparation

- **Extract dump texts**
- [Build corpus](https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py): `build_corpus.py`
- [Clean markup](https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py): `build_corpus.py`
- [Segment sentences](https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py): `build_corpus.py`
- [Segment words](https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py): `build_corpus.py`
- [Language corpus](https://github.com/kyubyong/wordvectors/blob/master/build_corpus.py): `build_corpus.py`

### Vector training

- [Word2Vec training](https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.py): `make_wordvectors.py`
- [Word2Vec runner](https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.sh): `make_wordvectors.sh`
- [fastText runner](https://github.com/kyubyong/wordvectors/blob/master/fasttext.sh): `fasttext.sh`

### Vector artifacts

- [Word2Vec vectors](https://github.com/kyubyong/wordvectors/blob/master/make_wordvectors.py): `make_wordvectors.py`
- [fastText vectors](https://github.com/kyubyong/wordvectors/blob/master/fasttext.sh): `fasttext.sh`
- [Pre-trained models](https://github.com/kyubyong/wordvectors/blob/master/README.md): `README.md`

### Other components

- **User**
- **Wikipedia dumps**

## Connections

- User → Wikipedia dumps: downloads (evidence: `README.md`)
- User → Extract dump texts: extracts (evidence: `README.md`)
- Wikipedia dumps → Extract dump texts: provides dump (evidence: `README.md`)
- Extract dump texts → Build corpus: runs (evidence: `README.md`)
- Build corpus → Clean markup: cleans text (evidence: `build_corpus.py`)
- Build corpus → Segment sentences: segments (evidence: `build_corpus.py`)
- Build corpus → Segment words: tokenizes (evidence: `build_corpus.py`)
- Build corpus → Language corpus: writes (evidence: `build_corpus.py`)
- User → Word2Vec runner: runs (evidence: `README.md`)
- Word2Vec runner → Word2Vec training: runs
- Language corpus → Word2Vec training: reads (evidence: `make_wordvectors.py`)
- Word2Vec training → Word2Vec vectors: writes (evidence: `make_wordvectors.py`)
- User → fastText runner: runs (evidence: `README.md`)
- fastText runner → Language corpus: uses corpus
- fastText runner → fastText vectors: produces (evidence: `README.md`)
- User → Pre-trained models: downloads (evidence: `README.md`)