How nadchif/in-browser-llm-inference works
Project type and purpose
• This is a single-page, front-end only web application built with React and Vite.
• It hosts and runs large language models (LLMs) entirely in the browser (zero backend) using the Wllama WebAssembly inference engine.
• A Node script (download-model.cjs) pulls down model files at build or setup time into public/models, which are then served as static assets.
File structure
• Top-level files:
– index.html, vite.config.js: SPA entry point and Vite configuration.
– package.json, package-lock.json: dependency and script definitions (npm run dev, etc.).
– download-model.cjs: Node utility to fetch LLM model files.
• public/models/: static directory for downloaded model artifacts.
• src/: core application code
– main.jsx: mounts the React app.
– App.jsx: top-level component orchestrating UI and model inference.
– components/: UI building blocks (Dropdowns, Buttons, Loader, Markdown renderer, Footer, etc.).
– lib/wllama.js: wraps the Wllama WebAssembly module for model loading and inference.
– assets/, index.css: styling and static assets.
Key points from the README
• No server-side API or database—everything runs in-browser.
• Relies on Wllama (WebAssembly) to perform inference on downloaded LLM checkpoints.
• Models come from SmolLm (HuggingFace) or Llama 3.2 (Meta) and live in public/models.
Main components
a. Static Host / Dev Server
– Vite dev server in development or Vercel/Netlify for production
– Serves index.html, JavaScript bundles, CSS, and public/models/*
b. Browser Client
– UI Layer: React components (Dropdown, Chat UI, Loader, etc.)
– App Logic: App.jsx handles user input, manages state, and calls the inference engine
– Inference Layer: Wllama WebAssembly module (wrapped by src/lib/wllama.js)
– Model Loader: fetches model files from the static host at runtime
c. External Model Sources (build time)
– download-model.cjs contacts HuggingFace/Meta to download model files into public/models
d. Static Assets
– Pre-downloaded model checkpoints stored under public/models
Relationships and data flow
1. Developer runs npm run dev
→ Vite server starts, optionally download-model.cjs downloads models
2. Browser requests index.html and JS bundles
3. User enters a prompt in the UI
4. App.jsx passes text to the Wllama wrapper
5. Wllama loads the model files (via fetch from public/models) into WebAssembly
6. Inference runs in-browser, returns tokens/response
7. UI renders the response
Architectural patterns and principles
• Separation of concerns: UI components vs. inference engine vs. static model storage
• Single responsibility: each React component handles one piece of UI
• Static hosting: no dynamic server, simple build & deploy pipeline
• Modular inference wrapper in lib/wllama.js allows swapping out engines in future
Technologies and frameworks to label
• React (JSX, components)
• Vite (build & dev server)
• Wllama (WebAssembly LLM inference)
• Node.js script for model download
• Static hosting (e.g., Vercel/Netlify)