How saisensei69/tokenizer-foliente works
This repository contains two standalone JavaScript tokenizer examples, not a larger application. Each file defines its own tokenize function and logs tokens for hard-coded sample expressions. The examples have no imports or demonstrated interaction, so they are shown as separate components; no runtime connections are inferred.
Components 2
Other components
- Regex tokenizer
index.js - Table tokenizer
tokenize.js