How muthukamalan/femtogpt works
1. What this repo is
- FemtoGPT is a small end-to-end LLM project split into three main planes: tokenizer/model training, inference serving, and user-facing chat demos.
- The codebase is centered around a custom femto/ Python package, with thin entrypoints for training and serving.
2. Core reusable library: femto/
- femto/ is the architectural heart of the repo.
- It contains the reusable building blocks for the LLM stack: configuration, dataset handling, model definition, evaluation, callbacks, helper utilities, and “basics”/primitive layers.
- This is the boundary between the project’s core ML logic and the higher-level apps/scripts.
3. Model subsystem
- The model is a decoder-only Transformer built from scratch in the femto package.
- The README indicates the key design choices: RMSNorm, RoPE, SwiGLU, and multi-head self-attention.
- This layer is used both during training and during inference, so the same model code is shared across the repo.
4. Tokenization subsystem
- The project includes a custom BPE tokenizer implementation rather than relying on an external tokenizer library.
- The tokenizer supports regex pretokenization, special tokens, and encode/decode flows.
- The tokenizer is part of the offline pipeline that prepares text data for the model; it is a distinct subsystem from the transformer itself.
5. Training pipeline
- train.py is the main training entrypoint.
- The README says training uses the Hugging Face Trainer API, so the training loop is largely delegated to that framework while the repo supplies the model, data, tokenizer, and callbacks/config.
- Training outputs land in trainer_output/, including checkpoints and TensorBoard event logs.
6. Data flow for training
- Data comes from Hugging Face Datasets and is streamed from several text corpora listed in the README, including smollm-corpus, Dolly, Alpaca, SQuAD, quotes, and SQL context data.
- The flow is: external dataset sources → dataset preprocessing/tokenization in femto/ → Hugging Face Trainer → checkpoints/logs in trainer_output/.
- This is the main offline ML pipeline in the repo.
7. Inference / generation path
- pred.py appears to be the direct script for local inference or generation from trained checkpoints.
- Inference uses the same model/tokenizer artifacts produced by training, which keeps the training and serving boundaries consistent.
- The README also references an inference API and chat generation, so the repo is structured to reuse the trained model for interactive use.
8. Service layer: app/
- app/backend/ is the server-side service boundary; it contains its own Dockerfile, requirements, and main.py.
- app/frontend/ is a separate static UI layer (index.html) for a browser-based chat experience.
- app/gradio/ is an alternative lightweight UI/service wrapper using Gradio (main.py, reqs.txt), likely for rapid demoing of model chat.
9. Deployment and composition
- docker-compose.yml ties the runtime pieces together for local deployment.
- Each app surface has its own Dockerfile, so backend, frontend, and Gradio demo are independently containerized.
- The README mentions an HF_TOKEN env var for the backend, indicating the serving layer depends on Hugging Face credentials or model access at runtime.
10. Experimentation and analysis artifacts
- notebooks/ contains exploratory work on BPE, generation, and tokenization behavior.
- misc/ holds additional helper/demo scripts, likely for prototyping outside the main training/inference path.
- These are support artifacts, not primary runtime components.
11. Operational outputs
- trainer_output/ stores the concrete training artifacts: model checkpoints and TensorBoard logs.
- assets/ contains documentation images used in the README, showing logs and the web UI.
- These directories are useful for understanding the runtime state, but they are not core application logic.
12. Overall architecture boundary map
- External data sources feed the tokenizer and training pipeline.
- The core femto library defines the tokenizer, transformer, and training helpers.
- Training produces checkpoints and logs.
- Inference scripts and app services consume those trained artifacts.
- The UI surfaces (frontend and gradio) sit on top of the backend/inference layer and do not implement model logic themselves.
Main entry points: femto femto · train.py Train · fmodel.py Transformer · pred.py Generate
Components 18
Core library
Reusable LLM building blocks and shared model/tokenization logic.
- femto · Core reusable LLM library.
femto - Config · Model and training configuration definitions.
femto/fconfig.py - Dataset · Dataset loading and preprocessing for training.
femto/fdataset.py - Transformer · Decoder-only transformer with attention, RMSNorm, RoPE, and SwiGLU.
femto/fmodel.py - Callbacks · Training-time callbacks and integration helpers.
femto/fcallback.py - Eval · Evaluation and scoring utilities.
femto/feval.py - Helpers · Shared helper functions used across the stack.
femto/fhelper.py - Basics · Low-level building blocks and primitive layers.
femto/basics.py
Offline pipeline
Training and generation scripts that operate on data and checkpoints.
Serving apps
Backend and UI surfaces for interactive inference.
Artifacts
Generated outputs and documentation assets.
- Trainer output · Training checkpoints and TensorBoard logs.
trainer_output - Assets · Images and documentation artifacts used in the README.
assets - Notebooks · Exploratory notebooks for BPE, generation, and tokenization.
notebooks - Misc · Additional helper and demo scripts outside the main runtime path.
misc
Connections 24
- femto → Config: contains · Core package includes configuration.
- femto → Dataset: contains · Core package includes dataset tooling.
- femto → Transformer: contains · Core package includes the shared transformer implementation.
- femto → Callbacks: contains · Core package includes training callbacks.
- femto → Eval: contains · Core package includes evaluation utilities.
- femto → Helpers: contains · Core package includes shared helpers.
- femto → Basics: contains · Core package includes primitive layers and building blocks.
- Train → Dataset: uses · Training consumes prepared datasets.
- Train → Transformer: uses · Training instantiates the shared transformer.
- Train → Callbacks: uses · Training plugs in callbacks.
- Train → Eval: uses · Training/evaluation flows reuse scoring utilities.
- Train → Trainer output: writes · Training produces checkpoints and logs.
- Generate → Transformer: loads · Inference loads the trained model architecture.
- Generate → Helpers: uses · Generation reuses shared helpers.
- Backend → Transformer: serves · Backend exposes the model for inference.
- Backend → Generate: aligns with · Backend and local generation share inference behavior.
- Frontend → Backend: calls · Browser UI talks to backend API.
- Gradio → Transformer: serves · Gradio wrapper exposes the model for chat.
- Compose → Backend: orchestrates · Compose wires the backend into local deployment.
- Compose → Frontend: orchestrates · Compose includes the browser UI.
- Compose → Gradio: orchestrates · Compose can also include the Gradio demo surface.
- Assets → Frontend: documents · Assets include UI screenshots and docs images.
- Notebooks → Dataset: explores · Notebooks analyze tokenization and data behavior.
- Misc → Generate: supports · Misc scripts support experimentation and demos.