# mauricebarksdale/assignment1-basics architecture

> Architecture diagram of the GitHub repository mauricebarksdale/assignment1-basics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mauricebarksdale/assignment1-basics
- Repository: https://github.com/mauricebarksdale/assignment1-basics

## Overview

This is Stanford CS336 Assignment 1’s basics package. Its visible runtime centers on a Python tokenizer API backed by a Rust BPE tokenizer; the repository also includes a chunk-boundary pretokenization example and a BPE-training entry point that is currently unimplemented. The README points to the assignment handout for broader requirements, but no model or neural-network implementation files appear in the supplied tree. The graph reflects only relationships shown in excerpts; it does not infer missing assignment subsystems.

Main entry points: [`cs336_basics/tokenizer/rust/src/lib.rs`](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/rust/src/lib.rs) (Rust tokenizer), [`cs336_basics/tokenizer/tokenizer.py`](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/tokenizer.py) (Python API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Tokenizer API"]
  node_wrapper["Python API<br/>[tokenizer.py]"]
end

subgraph group_engine["Rust Tokenizer"]
  node_native["Rust tokenizer<br/>[lib.rs]"]
  node_pretokenize["Pretokenization<br/>[lib.rs]"]
  node_merge["BPE merging<br/>[lib.rs]"]
end

subgraph group_examples["BPE Utilities"]
  node_chunking["Chunk boundaries"]
  node_training["BPE training<br/>[train_bpe.py]"]
end

node_user(("Text caller"))

node_user -->|"calls API"| node_wrapper
node_wrapper -->|"delegates encoding"| node_native
node_wrapper -->|"delegates decoding"| node_native
node_native -->|"pretokenizes text"| node_pretokenize
node_native -->|"tokenizes pretokens"| node_merge
node_merge -->|"returns token IDs"| node_native

click node_wrapper "https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/tokenizer.py"
click node_native "https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/rust/src/lib.rs"
click node_pretokenize "https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/rust/src/lib.rs"
click node_merge "https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/rust/src/lib.rs"
click node_chunking "https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/pretokenization_example.py"
click node_training "https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/train_bpe.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_wrapper toneBlue
class node_native,node_pretokenize,node_merge toneAmber
class node_chunking,node_training toneMint
class node_user toneIndigo
```

## Components

### Tokenizer API

- [Python API](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/tokenizer.py): `cs336_basics/tokenizer/tokenizer.py`

### Rust Tokenizer

- [Rust tokenizer](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/rust/src/lib.rs): `cs336_basics/tokenizer/rust/src/lib.rs`
- [Pretokenization](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/rust/src/lib.rs): `cs336_basics/tokenizer/rust/src/lib.rs`
- [BPE merging](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/rust/src/lib.rs): `cs336_basics/tokenizer/rust/src/lib.rs`

### BPE Utilities

- [Chunk boundaries](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/pretokenization_example.py): `cs336_basics/pretokenization_example.py`
- [BPE training](https://github.com/mauricebarksdale/assignment1-basics/blob/main/cs336_basics/tokenizer/train_bpe.py): `cs336_basics/tokenizer/train_bpe.py`

### Other components

- **Text caller**

## Connections

- Text caller → Python API: calls API
- Python API → Rust tokenizer: delegates encoding (evidence: `cs336_basics/tokenizer/tokenizer.py`)
- Python API → Rust tokenizer: delegates decoding (evidence: `cs336_basics/tokenizer/tokenizer.py`)
- Rust tokenizer → Pretokenization: pretokenizes text (evidence: `cs336_basics/tokenizer/rust/src/lib.rs`)
- Rust tokenizer → BPE merging: tokenizes pretokens (evidence: `cs336_basics/tokenizer/rust/src/lib.rs`)
- BPE merging → Rust tokenizer: returns token IDs (evidence: `cs336_basics/tokenizer/rust/src/lib.rs`)
