How bhartirajesh99/custom-tokenizer works
Project Type and Purpose
• This is a single-page full-stack (front-end heavy) Next.js web application written in TypeScript and styled with Tailwind CSS.
• Purpose: Provide a text tokenizer UI in the browser that splits input into tokens (words, numbers, punctuation, whitespace), assigns persistent numeric IDs via localStorage, and allows encoding/decoding of token sequences.
File Structure Analysis
• next.config.ts, tsconfig.json, postcss.config.mjs, package.json, etc. – Next.js + Tailwind CSS build/config.
• public/images – static assets (demo screenshots).
• src/app – Next.js app directory (App Router):
– layout.tsx – global layout.
– page.tsx – main page UI.
– globals.css & tokens.ts – global styles and tokenization logic/registry.
README Insights
• No external backend or database: all data lives in browser localStorage.
• Core features are implemented entirely on the client side (via Next.js React components and a tokenization module in tokens.ts).
• Dependencies: React, Next.js, TypeScript, Tailwind CSS.
Main Components
a. Browser / Client
– Next.js React UI (Page & Layout components)
– Tokenizer module (tokens.ts)
– localStorage (persistent token dictionary)
b. Hosting / Next.js Runtime
– Node.js server (dev or production) serving static JS/CSS
– build pipeline (PostCSS/Tailwind)
c. External Services (none, aside from CDN for static assets if deployed)
Component Relationships and Data Flow
1. User → Web Browser (visits http://localhost:3000)
2. Browser ←→ Next.js Server (serves JS/CSS bundles)
3. Browser UI calls Tokenizer module:
– tokenize(inputText) → returns token array
– assignOrRetrieveID(token) → reads/writes localStorage
4. UI displays tokens + IDs, or parses a CSV of IDs → calls decode(ids) → UI output
5. Token dictionary is read/written in localStorage on each operation
Architectural Patterns and Principles
• Component-based UI (React + Next.js App Router)
• Client-side state persistence (localStorage)
• Separation of concerns: UI components vs. tokenization logic (tokens.ts) vs. styling (globals.css + Tailwind)
• Static asset hosting for images under public/
Technologies and Frameworks
• Next.js (App Router, SSR/SSG)
• React (TSX components)
• TypeScript (static typing)
• Tailwind CSS + PostCSS (utility CSS)
• Browser API: localStorage