# kunjcr2/how-llms-are-made architecture

> Architecture diagram of the GitHub repository kunjcr2/how-llms-are-made, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kunjcr2/how-llms-are-made
- Repository: https://github.com/kunjcr2/how-llms-are-made

## Overview

##### Project type and purpose
- This repository is a monorepo of research code, notebooks, models, and documentation for large-language-model (LLM) experiments, including:
- DeepSeek: custom MoE-inspired attention modules and research papers
- GPT: end-to-end notebooks covering tokenization, attention, architecture, training, post-training, and fine-tuning
- Kimi: Mixture‐of‐D experts research modules
- MLandDL & Reasoning Models: classical ML/DL lecture notes and code
- My Models: stored fine-tuned model artifacts plus a minimal serving app (app.py)
- Purpose: Explore, implement, and compare LLM techniques (attention variants, MoE, fine-tuning) and serve a selected fine-tuned model.

##### File structure to infer architecture
- Top-level directories each represent a component or “logical service”:
- DeepSeek & Kimi: Python libraries/modules containing core algorithm implementations (.py) and PDFs/markdown for theory
- GPT: Jupyter notebooks driving data flow from raw text → tokenization → model building/training → fine-tuning
- My Models: model storage (safetensors, config, tokenizer) + app.py as a minimal model-serving layer
- MLandDL & Reasoning Models: supplemental educational material, not part of the runtime flow
- Configuration/build: No Dockerfile or CI scripts present; version control only.
- Deployment: Only My Models/app.py suggests a runtime service (likely Flask/FastAPI) that loads saved model.

##### Insights from README
- It merely points to the main topic folders; no high-level architecture specified.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        DS1[(alice_in_wonderland.txt)]:::dataStorage
        DS2[(hairy_potter.txt)]:::dataStorage
        DS3[(the_verdict.txt)]:::dataStorage
    end

    %% Preprocessing & Tokenization
    T1["LLM_tokenizer.ipynb"]:::computation

    %% Core Libraries
    subgraph "Core Libraries"
        subgraph "DeepSeek Library"
            Ex["Expert.py"]:::computation
            MTP["MTPModule.py"]:::computation
            Moe["MoeBase.py"]:::computation
            Noisy["NoisyTopK.py"]:::computation
            RMS["RMSNorm.py"]:::computation
            Rope["Rope.py"]:::computation
            RA["RopeAttention.py"]:::computation
            RLA["RopelessAttention.py"]:::computation
            DeepSeekMD["DeepSeek.md"]:::computation
        end
        subgraph "Kimi Library"
            MoDpdf["MoD.pdf"]:::computation
            howMD["how_mod_work.md"]:::computation
            whatMD["what_is_mod.md"]:::computation
        end
    end

    %% Training Pipeline
    subgraph "Training Pipeline Notebooks"
        N2["LLM_attention.ipynb"]:::computation
        N3a["LLM_GPT_arch.ipynb"]:::computation
        N3b["LLM_arch_pynb.ipynb"]:::computation
        N4["LLM_training.ipynb"]:::computation
        N5["LLM_post_training.ipynb"]:::computation
        N6a["LLM_LoRA_finetune.ipynb"]:::computation
        N6b["LLM_full_finetune.ipynb"]:::computation
    end

    %% Model Artifacts Storage
    subgraph "Model Artifacts Storage"
        Fin1[(Finetuned_flan_t5/)]:::dataStorage
        G2[(gpt2-med-finetuned/)]:::dataStorage
        M155_nb[(LLM_155M.ipynb)]:::dataStorage
        M155_cfg[(LLM_CONFIG_155M.py)]:::dataStorage
        M211_nb[(LLM_211M.ipynb)]:::dataStorage
        M211_cfg[(LLM_CONFIG_211M.py)]:::dataStorage
    end

    %% Serving Layer
    API["app.py"]:::serving

    %% Research & Educational Materials
    subgraph "Research & Educational Materials"
        Edu1["MLandDL/"]:::edu
        Edu2["Reasoning Models/"]:::edu
    end

    %% Connections
    DS1 --> T1
    DS2 --> T1
    DS3 --> T1
    T1 --> N2
    T1 --> N3a
    T1 --> N3b
    T1 --> N4
    T1 --> N5
    T1 --> N6a
    T1 --> N6b

    N2 --> Fin1
    N3a --> Fin1
    N3b --> Fin1
    N4 --> Fin1
    N5 --> Fin1
    N6a --> Fin1
    N6b --> Fin1

    Fin1 --> API
    G2 --> API
    M155_nb --> API
    M155_cfg --> API
    M211_nb --> API
    M211_cfg --> API

    Edu1 -.-> N2
    Edu1 -.-> N3a
    Edu1 -.-> N4
    Edu2 -.-> N5
    Edu2 -.-> N6a

    %% Click Events
    click DS1 "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/1_tokenizer/alice_in_wonderland.txt"
    click DS2 "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/1_tokenizer/hairy_potter.txt"
    click DS3 "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/1_tokenizer/the_verdict.txt"
    click T1 "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/1_tokenizer/LLM_tokenizer.ipynb"
    click Ex "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/Expert.py"
    click MTP "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/MTPModule.py"
    click Moe "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/MoeBase.py"
    click Noisy "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/NoisyTopK.py"
    click RMS "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/RMSNorm.py"
    click Rope "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/Rope.py"
    click RA "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/RopeAttention.py"
    click RLA "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/Codes/RopelessAttention.py"
    click DeepSeekMD "https://github.com/kunjcr2/how-llms-are-made/blob/main/DeepSeek/DeepSeek.md"
    click MoDpdf "https://github.com/kunjcr2/how-llms-are-made/blob/main/Kimi/MoD.pdf"
    click howMD "https://github.com/kunjcr2/how-llms-are-made/blob/main/Kimi/how_mod_work.md"
    click whatMD "https://github.com/kunjcr2/how-llms-are-made/blob/main/Kimi/what_is_mod.md"
    click N2 "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/2_attention/LLM_attention.ipynb"
    click N3a "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/3_architecture/LLM_GPT_arch.ipynb"
    click N3b "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/3_architecture/LLM_arch_pynb.ipynb"
    click N4 "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/4_training/LLM_training.ipynb"
    click N5 "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/5_post_training/LLM_post_training.ipynb"
    click N6a "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/6_finetune/LLM_LoRA_finetune.ipynb"
    click N6b "https://github.com/kunjcr2/how-llms-are-made/blob/main/GPT/6_finetune/LLM_full_finetune.ipynb"
    click Fin1 "https://github.com/kunjcr2/how-llms-are-made/tree/main/My Models/Finetuned_flan_t5/"
    click G2 "https://github.com/kunjcr2/how-llms-are-made/tree/main/My Models/gpt2-med-finetuned/"
    click M155_nb "https://github.com/kunjcr2/how-llms-are-made/blob/main/My Models/GPT_155M/LLM_155M.ipynb"
    click M155_cfg "https://github.com/kunjcr2/how-llms-are-made/blob/main/My Models/GPT_155M/LLM_CONFIG_155M.py"
    click M211_nb "https://github.com/kunjcr2/how-llms-are-made/blob/main/My Models/GPT_211M/LLM_211M.ipynb"
    click M211_cfg "https://github.com/kunjcr2/how-llms-are-made/blob/main/My Models/GPT_211M/LLM_CONFIG_211M.py"
    click API "https://github.com/kunjcr2/how-llms-are-made/blob/main/My Models/Finetuned_flan_t5/app.py"

    %% Styles
    classDef dataStorage fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef computation fill:#98FB98,stroke:#333,stroke-width:1px
    classDef serving fill:#FFA500,stroke:#333,stroke-width:1px
    classDef edu fill:#D3D3D3,stroke:#333,stroke-width:1px
```
