How mauricebarksdale/assignment1-basics works
This is Stanford CS336 Assignment 1’s basics package. Its visible runtime centers on a Python tokenizer API backed by a Rust BPE tokenizer; the repository also includes a chunk-boundary pretokenization example and a BPE-training entry point that is currently unimplemented. The README points to the assignment handout for broader requirements, but no model or neural-network implementation files appear in the supplied tree. The graph reflects only relationships shown in excerpts; it does not infer missing assignment subsystems.
Main entry points: lib.rs Rust tokenizer · tokenizer.py Python API
Components 7
Tokenizer API
- Python API
cs336_basics/tokenizer/tokenizer.py
Rust Tokenizer
- Rust tokenizer
cs336_basics/tokenizer/rust/src/lib.rs - Pretokenization
cs336_basics/tokenizer/rust/src/lib.rs - BPE merging
cs336_basics/tokenizer/rust/src/lib.rs
BPE Utilities
- Chunk boundaries
cs336_basics/pretokenization_example.py - BPE training
cs336_basics/tokenizer/train_bpe.py
Other components
- Text caller
Connections 6
- Text caller → Python API: calls API
- Python API → Rust tokenizer: delegates encoding
cs336_basics/tokenizer/tokenizer.py - Python API → Rust tokenizer: delegates decoding
cs336_basics/tokenizer/tokenizer.py - Rust tokenizer → Pretokenization: pretokenizes text
cs336_basics/tokenizer/rust/src/lib.rs - Rust tokenizer → BPE merging: tokenizes pretokens
cs336_basics/tokenizer/rust/src/lib.rs - BPE merging → Rust tokenizer: returns token IDs
cs336_basics/tokenizer/rust/src/lib.rs