# bikramroyutsa/spl-1 architecture

> Architecture diagram of the GitHub repository bikramroyutsa/spl-1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bikramroyutsa/spl-1
- Repository: https://github.com/bikramroyutsa/spl-1

## Overview

This C++ project implements a file-to-token workflow: `main.cpp` reads `input.txt`, parses text into words, trains byte-pair merge rules and a vocabulary, encodes the words, then calls decoding. A separate occurrence module builds a token co-occurrence matrix and applies PMI scoring; the excerpts do not show it connected to the main workflow. `tokenizer.cpp` is another standalone input-reading program, and its word-processing function is empty. Relationships shown below come from the excerpts; the PMI and standalone tokenizer branches have no demonstrated connection to the main program.

Main entry points: [`main.cpp`](https://github.com/bikramroyutsa/spl-1/blob/main/main.cpp) (Workflow driver), [`parser.cpp`](https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp) (BPE training), [`tokenizer.cpp`](https://github.com/bikramroyutsa/spl-1/blob/main/tokenizer.cpp) (Input reader), [`occurence.cpp`](https://github.com/bikramroyutsa/spl-1/blob/main/occurence.cpp) (Co-occurrence matrix)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["BPE workflow"]
  node_main["Workflow driver<br/>[main.cpp]"]
  node_parser["Word parsing<br/>[parser.cpp]"]
  node_bpe["BPE training<br/>[parser.cpp]"]
  node_pair_counts["Pair counts<br/>[parser.cpp]"]
  node_merge["Pair merging<br/>[parser.cpp]"]
  node_encoder["Token encoding<br/>[parser.cpp]"]
  node_decoder["Token decoding<br/>[parser.cpp]"]
end

subgraph group_analysis["Occurrence analysis"]
  node_matrix["Co-occurrence matrix<br/>[occurence.cpp]"]
  node_pmi["PMI scoring<br/>[occurence.cpp]"]
end

subgraph group_standalone["Standalone tokenizer"]
  node_standalone_reader["Input reader<br/>[tokenizer.cpp]"]
  node_utf8_length["UTF-8 length<br/>[tokenizer.cpp]"]
end

node_input["Input file"]

node_input -->|"read by"| node_main
node_main -->|"parses lines"| node_parser
node_main -->|"trains rules"| node_bpe
node_bpe -->|"counts pairs"| node_pair_counts
node_bpe -->|"merges pairs"| node_merge
node_main -->|"encodes words"| node_encoder
node_main -->|"decodes tokens"| node_decoder
node_matrix -->|"scores matrix"| node_pmi
node_input -->|"read by"| node_standalone_reader
node_standalone_reader -.->|"no call shown"| node_utf8_length

click node_main "https://github.com/bikramroyutsa/spl-1/blob/main/main.cpp"
click node_parser "https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp"
click node_bpe "https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp"
click node_pair_counts "https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp"
click node_merge "https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp"
click node_encoder "https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp"
click node_decoder "https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp"
click node_matrix "https://github.com/bikramroyutsa/spl-1/blob/main/occurence.cpp"
click node_pmi "https://github.com/bikramroyutsa/spl-1/blob/main/occurence.cpp"
click node_standalone_reader "https://github.com/bikramroyutsa/spl-1/blob/main/tokenizer.cpp"
click node_utf8_length "https://github.com/bikramroyutsa/spl-1/blob/main/tokenizer.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_parser,node_bpe,node_pair_counts,node_merge,node_encoder,node_decoder toneBlue
class node_matrix,node_pmi toneAmber
class node_standalone_reader,node_utf8_length toneMint
class node_input toneIndigo
```

## Components

### BPE workflow

- [Workflow driver](https://github.com/bikramroyutsa/spl-1/blob/main/main.cpp): `main.cpp`
- [Word parsing](https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp): `parser.cpp`
- [BPE training](https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp): `parser.cpp`
- [Pair counts](https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp): `parser.cpp`
- [Pair merging](https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp): `parser.cpp`
- [Token encoding](https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp): `parser.cpp`
- [Token decoding](https://github.com/bikramroyutsa/spl-1/blob/main/parser.cpp): `parser.cpp`

### Occurrence analysis

- [Co-occurrence matrix](https://github.com/bikramroyutsa/spl-1/blob/main/occurence.cpp): `occurence.cpp`
- [PMI scoring](https://github.com/bikramroyutsa/spl-1/blob/main/occurence.cpp): `occurence.cpp`

### Standalone tokenizer

- [Input reader](https://github.com/bikramroyutsa/spl-1/blob/main/tokenizer.cpp): `tokenizer.cpp`
- [UTF-8 length](https://github.com/bikramroyutsa/spl-1/blob/main/tokenizer.cpp): `tokenizer.cpp`

### Other components

- **Input file**

## Connections

- Input file → Workflow driver: read by (evidence: `main.cpp`)
- Workflow driver → Word parsing: parses lines (evidence: `main.cpp`)
- Workflow driver → BPE training: trains rules (evidence: `main.cpp`)
- BPE training → Pair counts: counts pairs (evidence: `parser.cpp`)
- BPE training → Pair merging: merges pairs (evidence: `parser.cpp`)
- Workflow driver → Token encoding: encodes words (evidence: `main.cpp`)
- Workflow driver → Token decoding: decodes tokens (evidence: `main.cpp`)
- Co-occurrence matrix → PMI scoring: scores matrix (evidence: `occurence.cpp`)
- Input file → Input reader: read by (evidence: `tokenizer.cpp`)
- Input reader → UTF-8 length: no call shown (evidence: `tokenizer.cpp`)
