# saisensei69/tokenizer-foliente architecture

> Architecture diagram of the GitHub repository saisensei69/tokenizer-foliente, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saisensei69/tokenizer-foliente
- Repository: https://github.com/saisensei69/tokenizer-foliente

## Overview

This repository contains two standalone JavaScript tokenizer examples, not a larger application. Each file defines its own `tokenize` function and logs tokens for hard-coded sample expressions. The examples have no imports or demonstrated interaction, so they are shown as separate components; no runtime connections are inferred.

## Diagram (Mermaid)

```mermaid
flowchart TD

node_index_tokenizer["Regex tokenizer<br/>[index.js]"]
node_tokenize_tokenizer["Table tokenizer<br/>[tokenize.js]"]

click node_index_tokenizer "https://github.com/saisensei69/tokenizer-foliente/blob/main/index.js"
click node_tokenize_tokenizer "https://github.com/saisensei69/tokenizer-foliente/blob/main/tokenize.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_index_tokenizer,node_tokenize_tokenizer toneTeal
```

## Components

### Other components

- [Regex tokenizer](https://github.com/saisensei69/tokenizer-foliente/blob/main/index.js): `index.js`
- [Table tokenizer](https://github.com/saisensei69/tokenizer-foliente/blob/main/tokenize.js): `tokenize.js`