# debopriya-ai/bpe-forge architecture

> Architecture diagram of the GitHub repository debopriya-ai/bpe-forge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/debopriya-ai/bpe-forge
- Repository: https://github.com/debopriya-ai/bpe-forge

## Overview

##### Project Type and Purpose
- This is an open-source NLP infrastructure tool: a command-line Python project to train and export a custom Byte-Pair Encoding (BPE) tokenizer.
- Primary goal: ingest the WikiText-103 corpus, build subword vocab via Hugging Face’s Tokenizers, tune and validate vocabulary, export JSON artifacts, and generate usage/length statistics for integration with transformer models.

##### Key Components
a. CLI Interface
- Entry point (e.g., tokenizer.py invoked as a script).
- Accepts arguments: input data path, output directories, hyperparameters (vocab size, special tokens).
b. Data Ingestion & Preprocessing
- WikiText-103 dataset loader.
- Text normalization, whitespace/token splitting prior to BPE training.
c. BPE Trainer
- Core algorithm implemented via Hugging Face Tokenizers library.
- Subword segmentation logic, merge operations.
d. Vocabulary Tuner & Special Tokens
- Adjust final vocab size.
- Insert [PAD], [CLS], [SEP], etc.
e. JSON Exporter
- Serialize tokenizer model to vocab.json and merges.json.
- Save config metadata (token-to-ID map, special token IDs).
f. Validator
- Post-export checks: round-trip tokenization, special token handling tests.
g. Stats Generator
- Compute token frequency distributions.
- Length distribution analysis.
- Produce plots under stats/ (e.g., histograms, bar charts in PNG or notebook).
h. Outputs
- vocab.json (core artifact).
- JSON config, validation logs.
- Visual artifacts in stats/ directory.
i. Downstream Integration
- How the exported tokenizer hooks into transformer training pipelines.

##### Interactions & Data Flow
- Parallel branch from Preprocessing → Stats Generator (to compute baseline stats).
- After exporting: flow into Validator; if passed, artifacts move to downstream models.

##### Architectural Patterns & Principles
- Pipeline pattern: modular stages, clear hand-offs between components.
- Separation of concerns: training logic vs. validation vs. statistics.
- Extensibility: ability to swap in new tokenization algorithms or special token schemes.
- Reproducibility: CLI flags, config files, versioned outputs.

##### Relevant Technologies & Libraries
- Python 3.x runtime.
- Hugging Face Tokenizers library (Rust-backed, Python bindings).
- NumPy (data arrays, frequency counts).
- JSON standard library for serialization.
- Matplotlib or similar (for plots) if referenced in stats.
- Jupyter Notebook (Midterm_NLP.ipynb) for exploratory analysis.

##### Detail & Focus
- Every stage: ingestion, BPE merges, tuning, special token injection, export, validation, visualization.
- Do not overcrowd: group related sub-steps under a single module if needed (e.g., “Vocabulary Module” containing tuning + special tokens).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Source
    DataSource["WikiText-103 Dataset"]:::dataset

    %% Core Pipeline
    CLI["CLI Interface"]:::processing
    Ingest["Data Ingestion & Preprocessing"]:::processing
    Trainer["BPE Trainer"]:::processing

    subgraph "Vocabulary Module"
        Tuner["Vocabulary Tuner"]:::processing
        Special["Special Token Handler"]:::processing
    end

    Exporter["JSON Exporter"]:::export
    Validator["Validator"]:::validation
    OutputsArtifacts["Exported Artifacts (vocab.json, merges.json)"]:::export
    Downstream["Downstream Integration"]:::external

    %% Parallel Stats Branch
    Stats["Stats Generator"]:::stats

    %% Documentation
    Docs["Project Documentation (README.md)"]:::doc
    Lic["License Information (LICENSE)"]:::doc

    %% Connections
    DataSource --> CLI
    CLI --> Ingest
    Ingest --> Trainer
    Trainer --> Tuner
    Tuner --> Special
    Special --> Exporter
    Exporter --> Validator
    Validator -->|passed| OutputsArtifacts
    OutputsArtifacts --> Downstream

    Ingest -->|baseline stats| Stats

    %% Click Events
    click Stats "https://github.com/debopriya-ai/bpe-forge/blob/main/Midterm_NLP (2).ipynb"
    click Docs "https://github.com/debopriya-ai/bpe-forge/blob/main/README.md"
    click Lic "https://github.com/debopriya-ai/bpe-forge/tree/main/LICENSE"

    %% Styles
    classDef dataset fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1;
    classDef processing fill:#C8E6C9,stroke:#388E3C,color:#1B5E20;
    classDef export fill:#FFE0B2,stroke:#F57C00,color:#E65100;
    classDef validation fill:#FFF9C4,stroke:#FBC02D,color:#F57F17;
    classDef stats fill:#D1C4E9,stroke:#7B1FA2,color:#4A148C;
    classDef external fill:#CFD8DC,stroke:#607D8B,color:#263238;
    classDef doc fill:#ECEFF1,stroke:#90A4AE,color:#546E7A;
```
