# tsoding/bpe architecture

> Architecture diagram of the GitHub repository tsoding/bpe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tsoding/bpe
- Repository: https://github.com/tsoding/bpe

## Overview

##### Main Components
- Command Line Interface (CLI) Entry Point:
- File: nob.c acts as the main entry. It handles command-line arguments and dispatches operation commands.
- Processing Modules (BPE Engine):
- Files in the src/ directory (bpe.c, bpe.h, bpe2bpe.c, bpe2dot.c, bpe_gen.c, bpe_inspect.c, tkn_inspect.c, txt2bpe.c) implement specific functionalities:
▫ bpe.c and bpe.h: Core implementation of the byte-pair encoding functions.
▫ txt2bpe.c: Likely converts plain text to a BPE format.
▫ bpe2dot.c: Presumably converts BPE outputs into DOT language for graph visualization.
▫ bpe_gen.c: Involved in generating BPE rules or sequences.
▫ bpe_inspect.c and tkn_inspect.c: For detailed inspection or debugging of BPE and tokenization outputs.
- Data Sources:
- The data/ directory (e.g. data/bpe-wiki-def.txt) probably contains default BPE definitions or reference text.
- Third-Party Dependencies:
- The thirdparty/ directory includes utilities such as flag.h for flag parsing, nob.h for potential shared definitions, and stb_ds.h for data structures, which are likely used by the BPE engine to simplify common tasks like command-line argument parsing and efficient data management.

##### Relationships & Interactions
- CLI (nob.c) collects input from the terminal, then parses command-line options (perhaps using thirdparty/flag.h) and routes control to the appropriate src module.
- The processing modules in src function as the BPE engine:
- They interact with the input (either directly from the user or via reading data from files in the data/ folder) and produce outputs such as transformed text, DOT graphs, or inspection details.
- They might share common functionality defined in bpe.h.
- The third-party libraries provide support for common tasks – for example, the flag parsing library and advanced data structure support (stb_ds.h).
- Data flows typically root from the terminal input, move through the CLI, into the specific transformation/inspection modules, and finally result in output (e.g., printed results, dot files for visualization).

##### Architectural Patterns & Design Principles
- Modularity: The project cleanly separates the CLI (nob.c) from the core processing logic (in src/). Each src module handles a specific responsibility (conversion, generation, inspection).
- Separation of Concerns: File separation indicates that input processing (CLI) is decoupled from the transformation logic.
- Reusability & Extensibility: With dedicated source files for each operation and a clear segregation in directories, it’s straightforward to add new functionalities or update individual modules.
- Dependency Management: Use of third-party header files indicates an emphasis on integrating well-known libraries to simplify tasks rather than building everything from scratch.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI Layer"
        CLI["Terminal/Command Line"]:::cliStyle
        ENTRY["nob Executable (nob.c)"]:::cliStyle
    end

    subgraph "BPE Engine"
        CORE1["BPE Core (bpe.c)"]:::bpeStyle
        CORE2["BPE Interface (bpe.h)"]:::bpeStyle
        TXTBPE["Text to BPE (txt2bpe.c)"]:::bpeStyle
        BPE2BPE["BPE to BPE Transformation (bpe2bpe.c)"]:::bpeStyle
        BPE2DOT["BPE Visualization (bpe2dot.c)"]:::bpeStyle
        BPEGEN["BPE Generation (bpe_gen.c)"]:::bpeStyle
        BPEINSPECT["BPE Inspection (bpe_inspect.c)"]:::bpeStyle
        TKNINSPECT["Token Inspection (tkn_inspect.c)"]:::bpeStyle
    end

    subgraph "External Resources"
        DATA["Data Source (bpe-wiki-def.txt)"]:::externalStyle
        THIRD_PARTY["Third-Party Libraries (flag.h, nob.h, stb_ds.h)"]:::externalStyle
    end

    CLI --> ENTRY
    ENTRY -->|"commandRouting"| CORE1
    ENTRY -->|"commandRouting"| CORE2
    ENTRY -->|"commandRouting"| TXTBPE
    ENTRY -->|"commandRouting"| BPE2BPE
    ENTRY -->|"commandRouting"| BPE2DOT
    ENTRY -->|"commandRouting"| BPEGEN
    ENTRY -->|"commandRouting"| BPEINSPECT
    ENTRY -->|"commandRouting"| TKNINSPECT

    CORE1 -->|"uses"| DATA
    ENTRY -->|"flagParsing"| THIRD_PARTY
    CORE1 -->|"dataHandling"| THIRD_PARTY

    click ENTRY "https://github.com/tsoding/bpe/blob/main/nob.c"
    click CORE1 "https://github.com/tsoding/bpe/blob/main/src/bpe.c"
    click CORE2 "https://github.com/tsoding/bpe/blob/main/src/bpe.h"
    click TXTBPE "https://github.com/tsoding/bpe/blob/main/src/txt2bpe.c"
    click BPE2BPE "https://github.com/tsoding/bpe/blob/main/src/bpe2bpe.c"
    click BPE2DOT "https://github.com/tsoding/bpe/blob/main/src/bpe2dot.c"
    click BPEGEN "https://github.com/tsoding/bpe/blob/main/src/bpe_gen.c"
    click BPEINSPECT "https://github.com/tsoding/bpe/blob/main/src/bpe_inspect.c"
    click TKNINSPECT "https://github.com/tsoding/bpe/blob/main/src/tkn_inspect.c"
    click DATA "https://github.com/tsoding/bpe/blob/main/data/bpe-wiki-def.txt"
    click THIRD_PARTY "https://github.com/tsoding/bpe/blob/main/thirdparty/flag.h"
    click THIRD_PARTY "https://github.com/tsoding/bpe/blob/main/thirdparty/nob.h"
    click THIRD_PARTY "https://github.com/tsoding/bpe/blob/main/thirdparty/stb_ds.h"

    classDef cliStyle fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef bpeStyle fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef externalStyle fill:#D8BFD8,stroke:#000,stroke-width:2px;
```
