# muthukamalan/femtogpt architecture

> Architecture diagram of the GitHub repository muthukamalan/femtogpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/muthukamalan/femtogpt
- Repository: https://github.com/muthukamalan/femtogpt

## Overview

1. **What this repo is**
   - FemtoGPT is a small end-to-end LLM project split into three main planes: **tokenizer/model training**, **inference serving**, and **user-facing chat demos**.
   - The codebase is centered around a custom `femto/` Python package, with thin entrypoints for training and serving.

2. **Core reusable library: `femto/`**
   - `femto/` is the architectural heart of the repo.
   - It contains the reusable building blocks for the LLM stack: configuration, dataset handling, model definition, evaluation, callbacks, helper utilities, and “basics”/primitive layers.
   - This is the boundary between the project’s core ML logic and the higher-level apps/scripts.

3. **Model subsystem**
   - The model is a **decoder-only Transformer** built from scratch in the `femto` package.
   - The README indicates the key design choices: **RMSNorm**, **RoPE**, **SwiGLU**, and **multi-head self-attention**.
   - This layer is used both during training and during inference, so the same model code is shared across the repo.

4. **Tokenization subsystem**
   - The project includes a **custom BPE tokenizer implementation** rather than relying on an external tokenizer library.
   - The tokenizer supports **regex pretokenization**, **special tokens**, and encode/decode flows.
   - The tokenizer is part of the offline pipeline that prepares text data for the model; it is a distinct subsystem from the transformer itself.

5. **Training pipeline**
   - `train.py` is the main training entrypoint.
   - The README says training uses the **Hugging Face Trainer** API, so the training loop is largely delegated to that framework while the repo supplies the model, data, tokenizer, and callbacks/config.
   - Training outputs land in `trainer_output/`, including checkpoints and TensorBoard event logs.

6. **Data flow for training**
   - Data comes from **Hugging Face Datasets** and is streamed from several text corpora listed in the README, including `smollm-corpus`, Dolly, Alpaca, SQuAD, quotes, and SQL context data.
   - The flow is: external dataset sources → dataset preprocessing/tokenization in `femto/` → Hugging Face Trainer → checkpoints/logs in `trainer_output/`.
   - This is the main offline ML pipeline in the repo.

7. **Inference / generation path**
   - `pred.py` appears to be the direct script for local inference or generation from trained checkpoints.
   - Inference uses the same model/tokenizer artifacts produced by training, which keeps the training and serving boundaries consistent.
   - The README also references an inference API and chat generation, so the repo is structured to reuse the trained model for interactive use.

8. **Service layer: `app/`**
   - `app/backend/` is the server-side service boundary; it contains its own Dockerfile, requirements, and `main.py`.
   - `app/frontend/` is a separate static UI layer (`index.html`) for a browser-based chat experience.
   - `app/gradio/` is an alternative lightweight UI/service wrapper using Gradio (`main.py`, `reqs.txt`), likely for rapid demoing of model chat.

9. **Deployment and composition**
   - `docker-compose.yml` ties the runtime pieces together for local deployment.
   - Each app surface has its own Dockerfile, so backend, frontend, and Gradio demo are independently containerized.
   - The README mentions an `HF_TOKEN` env var for the backend, indicating the serving layer depends on Hugging Face credentials or model access at runtime.

10. **Experimentation and analysis artifacts**
   - `notebooks/` contains exploratory work on BPE, generation, and tokenization behavior.
   - `misc/` holds additional helper/demo scripts, likely for prototyping outside the main training/inference path.
   - These are support artifacts, not primary runtime components.

11. **Operational outputs**
   - `trainer_output/` stores the concrete training artifacts: model checkpoints and TensorBoard logs.
   - `assets/` contains documentation images used in the README, showing logs and the web UI.
   - These directories are useful for understanding the runtime state, but they are not core application logic.

12. **Overall architecture boundary map**
   - External data sources feed the tokenizer and training pipeline.
   - The core `femto` library defines the tokenizer, transformer, and training helpers.
   - Training produces checkpoints and logs.
   - Inference scripts and app services consume those trained artifacts.
   - The UI surfaces (`frontend` and `gradio`) sit on top of the backend/inference layer and do not implement model logic themselves.

Main entry points: [`femto`](https://github.com/muthukamalan/femtogpt/tree/main/femto) (femto), [`train.py`](https://github.com/muthukamalan/femtogpt/blob/main/train.py) (Train), [`femto/fmodel.py`](https://github.com/muthukamalan/femtogpt/blob/main/femto/fmodel.py) (Transformer), [`pred.py`](https://github.com/muthukamalan/femtogpt/blob/main/pred.py) (Generate)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core library"]
  node_femto["femto<br/>python package"]
  node_fconfig["Config<br/>[fconfig.py]"]
  node_fdataset["Dataset<br/>data module<br/>[fdataset.py]"]
  node_fmodel{{"Transformer<br/>model module<br/>[fmodel.py]"}}
  node_fcallback["Callbacks<br/>trainer hooks<br/>[fcallback.py]"]
  node_feval["Eval<br/>[feval.py]"]
  node_fhelper["Helpers<br/>utility module<br/>[fhelper.py]"]
  node_basics["Basics<br/>primitives module<br/>[basics.py]"]
end

subgraph group_pipeline["Offline pipeline"]
  node_train["Train<br/>[train.py]"]
  node_pred["Generate<br/>inference script<br/>[pred.py]"]
end

subgraph group_apps["Serving apps"]
  node_backend["Backend<br/>serving app<br/>[main.py]"]
  node_frontend["Frontend<br/>static ui<br/>[index.html]"]
  node_gradio["Gradio<br/>demo app<br/>[main.py]"]
  node_compose["Compose<br/>deployment spec<br/>[docker-compose.yml]"]
end

subgraph group_artifacts["Artifacts"]
  node_trainer_output[("Trainer output<br/>artifacts dir")]
  node_assets["Assets"]
  node_notebooks["Notebooks<br/>analysis workspace"]
  node_misc["Misc<br/>prototype scripts"]
end

node_femto -->|"contains"| node_fconfig
node_femto -->|"contains"| node_fdataset
node_femto -->|"contains"| node_fmodel
node_femto -->|"contains"| node_fcallback
node_femto -->|"contains"| node_feval
node_femto -->|"contains"| node_fhelper
node_femto -->|"contains"| node_basics
node_train -->|"uses"| node_fdataset
node_train -->|"uses"| node_fmodel
node_train -->|"uses"| node_fcallback
node_train -->|"uses"| node_feval
node_train -->|"writes"| node_trainer_output
node_pred -->|"loads"| node_fmodel
node_pred -->|"uses"| node_fhelper
node_backend -->|"serves"| node_fmodel
node_backend -.->|"aligns with"| node_pred
node_frontend -->|"calls"| node_backend
node_gradio -->|"serves"| node_fmodel
node_compose -->|"orchestrates"| node_backend
node_compose -->|"orchestrates"| node_frontend
node_compose -->|"orchestrates"| node_gradio
node_assets -.->|"documents"| node_frontend
node_notebooks -.->|"explores"| node_fdataset
node_misc -.->|"supports"| node_pred

click node_femto "https://github.com/muthukamalan/femtogpt/tree/main/femto"
click node_fconfig "https://github.com/muthukamalan/femtogpt/blob/main/femto/fconfig.py"
click node_fdataset "https://github.com/muthukamalan/femtogpt/blob/main/femto/fdataset.py"
click node_fmodel "https://github.com/muthukamalan/femtogpt/blob/main/femto/fmodel.py"
click node_fcallback "https://github.com/muthukamalan/femtogpt/blob/main/femto/fcallback.py"
click node_feval "https://github.com/muthukamalan/femtogpt/blob/main/femto/feval.py"
click node_fhelper "https://github.com/muthukamalan/femtogpt/blob/main/femto/fhelper.py"
click node_basics "https://github.com/muthukamalan/femtogpt/blob/main/femto/basics.py"
click node_train "https://github.com/muthukamalan/femtogpt/blob/main/train.py"
click node_pred "https://github.com/muthukamalan/femtogpt/blob/main/pred.py"
click node_backend "https://github.com/muthukamalan/femtogpt/blob/main/app/backend/main.py"
click node_frontend "https://github.com/muthukamalan/femtogpt/blob/main/app/frontend/index.html"
click node_gradio "https://github.com/muthukamalan/femtogpt/blob/main/app/gradio/main.py"
click node_compose "https://github.com/muthukamalan/femtogpt/blob/main/docker-compose.yml"
click node_trainer_output "https://github.com/muthukamalan/femtogpt/tree/main/trainer_output"
click node_assets "https://github.com/muthukamalan/femtogpt/tree/main/assets"
click node_notebooks "https://github.com/muthukamalan/femtogpt/tree/main/notebooks"
click node_misc "https://github.com/muthukamalan/femtogpt/tree/main/misc"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_femto,node_fconfig,node_fdataset,node_fmodel,node_fcallback,node_feval,node_fhelper,node_basics toneBlue
class node_train,node_pred toneAmber
class node_backend,node_frontend,node_gradio,node_compose toneMint
class node_trainer_output,node_assets,node_notebooks,node_misc toneRose
```

## Components

### Core library

Reusable LLM building blocks and shared model/tokenization logic.

- [femto](https://github.com/muthukamalan/femtogpt/tree/main/femto): python package · Core reusable LLM library. · `femto`
- [Config](https://github.com/muthukamalan/femtogpt/blob/main/femto/fconfig.py): config module · Model and training configuration definitions. · `femto/fconfig.py`
- [Dataset](https://github.com/muthukamalan/femtogpt/blob/main/femto/fdataset.py): data module · Dataset loading and preprocessing for training. · `femto/fdataset.py`
- [Transformer](https://github.com/muthukamalan/femtogpt/blob/main/femto/fmodel.py): model module · Decoder-only transformer with attention, RMSNorm, RoPE, and SwiGLU. · `femto/fmodel.py`
- [Callbacks](https://github.com/muthukamalan/femtogpt/blob/main/femto/fcallback.py): trainer hooks · Training-time callbacks and integration helpers. · `femto/fcallback.py`
- [Eval](https://github.com/muthukamalan/femtogpt/blob/main/femto/feval.py): evaluation module · Evaluation and scoring utilities. · `femto/feval.py`
- [Helpers](https://github.com/muthukamalan/femtogpt/blob/main/femto/fhelper.py): utility module · Shared helper functions used across the stack. · `femto/fhelper.py`
- [Basics](https://github.com/muthukamalan/femtogpt/blob/main/femto/basics.py): primitives module · Low-level building blocks and primitive layers. · `femto/basics.py`

### Offline pipeline

Training and generation scripts that operate on data and checkpoints.

- [Train](https://github.com/muthukamalan/femtogpt/blob/main/train.py): training entrypoint · Main Hugging Face Trainer-based training script. · `train.py`
- [Generate](https://github.com/muthukamalan/femtogpt/blob/main/pred.py): inference script · Local checkpoint loading and text generation entrypoint. · `pred.py`

### Serving apps

Backend and UI surfaces for interactive inference.

- [Backend](https://github.com/muthukamalan/femtogpt/blob/main/app/backend/main.py): serving app · Server-side inference API and runtime container boundary. · `app/backend/main.py`
- [Frontend](https://github.com/muthukamalan/femtogpt/blob/main/app/frontend/index.html): static ui · Browser chat interface. · `app/frontend/index.html`
- [Gradio](https://github.com/muthukamalan/femtogpt/blob/main/app/gradio/main.py): demo app · Lightweight Gradio-based chat wrapper. · `app/gradio/main.py`
- [Compose](https://github.com/muthukamalan/femtogpt/blob/main/docker-compose.yml): deployment spec · Local multi-service composition for app runtime. · `docker-compose.yml`

### Artifacts

Generated outputs and documentation assets.

- [Trainer output](https://github.com/muthukamalan/femtogpt/tree/main/trainer_output): artifacts dir · Training checkpoints and TensorBoard logs. · `trainer_output`
- [Assets](https://github.com/muthukamalan/femtogpt/tree/main/assets): docs assets · Images and documentation artifacts used in the README. · `assets`
- [Notebooks](https://github.com/muthukamalan/femtogpt/tree/main/notebooks): analysis workspace · Exploratory notebooks for BPE, generation, and tokenization. · `notebooks`
- [Misc](https://github.com/muthukamalan/femtogpt/tree/main/misc): prototype scripts · Additional helper and demo scripts outside the main runtime path. · `misc`

## Connections

- femto → Config: contains · Core package includes configuration.
- femto → Dataset: contains · Core package includes dataset tooling.
- femto → Transformer: contains · Core package includes the shared transformer implementation.
- femto → Callbacks: contains · Core package includes training callbacks.
- femto → Eval: contains · Core package includes evaluation utilities.
- femto → Helpers: contains · Core package includes shared helpers.
- femto → Basics: contains · Core package includes primitive layers and building blocks.
- Train → Dataset: uses · Training consumes prepared datasets.
- Train → Transformer: uses · Training instantiates the shared transformer.
- Train → Callbacks: uses · Training plugs in callbacks.
- Train → Eval: uses · Training/evaluation flows reuse scoring utilities.
- Train → Trainer output: writes · Training produces checkpoints and logs.
- Generate → Transformer: loads · Inference loads the trained model architecture.
- Generate → Helpers: uses · Generation reuses shared helpers.
- Backend → Transformer: serves · Backend exposes the model for inference.
- Backend → Generate: aligns with · Backend and local generation share inference behavior.
- Frontend → Backend: calls · Browser UI talks to backend API.
- Gradio → Transformer: serves · Gradio wrapper exposes the model for chat.
- Compose → Backend: orchestrates · Compose wires the backend into local deployment.
- Compose → Frontend: orchestrates · Compose includes the browser UI.
- Compose → Gradio: orchestrates · Compose can also include the Gradio demo surface.
- Assets → Frontend: documents · Assets include UI screenshots and docs images.
- Notebooks → Dataset: explores · Notebooks analyze tokenization and data behavior.
- Misc → Generate: supports · Misc scripts support experimentation and demos.
