How bikramroyutsa/spl-1 works
This C++ project implements a file-to-token workflow: main.cpp reads input.txt, parses text into words, trains byte-pair merge rules and a vocabulary, encodes the words, then calls decoding. A separate occurrence module builds a token co-occurrence matrix and applies PMI scoring; the excerpts do not show it connected to the main workflow. tokenizer.cpp is another standalone input-reading program, and its word-processing function is empty. Relationships shown below come from the excerpts; the PMI and standalone tokenizer branches have no demonstrated connection to the main program.
Main entry points: main.cpp Workflow driver · parser.cpp BPE training · tokenizer.cpp Input reader · occurence.cpp Co-occurrence matrix
Components 12
BPE workflow
- Workflow driver
main.cpp - Word parsing
parser.cpp - BPE training
parser.cpp - Pair counts
parser.cpp - Pair merging
parser.cpp - Token encoding
parser.cpp - Token decoding
parser.cpp
Occurrence analysis
- Co-occurrence matrix
occurence.cpp - PMI scoring
occurence.cpp
Standalone tokenizer
- Input reader
tokenizer.cpp - UTF-8 length
tokenizer.cpp
Other components
- Input file
Connections 10
- Input file → Workflow driver: read by
main.cpp - Workflow driver → Word parsing: parses lines
main.cpp - Workflow driver → BPE training: trains rules
main.cpp - BPE training → Pair counts: counts pairs
parser.cpp - BPE training → Pair merging: merges pairs
parser.cpp - Workflow driver → Token encoding: encodes words
main.cpp - Workflow driver → Token decoding: decodes tokens
main.cpp - Co-occurrence matrix → PMI scoring: scores matrix
occurence.cpp - Input file → Input reader: read by
tokenizer.cpp - Input reader → UTF-8 length: no call shown
tokenizer.cpp