# thviet79/asm1-development-llm-model architecture

> Architecture diagram of the GitHub repository thviet79/asm1-development-llm-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thviet79/asm1-development-llm-model
- Repository: https://github.com/thviet79/asm1-development-llm-model

## Overview

##### File structure
- Top-level Python modules:
- base_llama.py – base classes and shared utilities for the model
- llama.py – your transformer block implementation and full forward/inference pipeline
- rope.py – rotary positional embeddings
- tokenizer.py – tokenization (vocab load, encode/decode)
- classifier.py – a classification head wrapping the LLM’s outputs
- optimizer.py – simple SGD/Adam-like routines + weight updates
- config.py – hyperparameter definitions (layer count, hidden size, etc.)
- utils.py – shared helpers (logging, metrics)
- run_llama.py – CLI entry point for generation, zero-shot, and fine-tuning workflows
- sanity_check.py & tests – smoke tests for your components
- data/ – raw train/dev/test splits + label maps for two sentiment datasets

##### README insights
- You load pretrained TinyStories weights (stories42M.pt), then run three workflows:
a. Text generation
b. Zero-shot prompting for sentiment
c. Fine-tuning with the classification head
- README emphasizes gaining understanding of model internals by implementing core classes.

##### Detail
- Every Python module as its own component
- How run_llama.py orchestrates workflows by invoking tokenizer → model → classifier/LM head → optimizer

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data & Preprocessing Layer
    subgraph "Data & Preprocessing"
        direction TB
        Data["Data Ingestion & Splitting"]:::data
        Tokenizer["Tokenizer Module"]:::data
        Rope["Positional Embedding Module"]:::data
    end

    %% Model Core Layer
    subgraph "Model Core"
        direction TB
        Base["Transformer Base Classes & Utilities"]:::core
        Llama["Transformer Core & Inference Pipeline"]:::core
        Classifier["Classification Head"]:::core
    end

    %% Training & Optimization Layer
    subgraph "Training & Optimization"
        direction TB
        Utils["Loss Computation & Metrics"]:::train
        Optimizer["Optimizer Module"]:::train
    end

    %% Orchestrator & Scripts
    subgraph "Orchestrator & Config"
        direction TB
        Run["CLI Orchestrator (run_llama.py)"]:::script
        Config["Configuration / Hyperparameters"]:::script
        Setup["Environment Setup Script"]:::script
    end

    %% Tests & Sanity Checks
    subgraph "Tests & Sanity Checks"
        direction TB
        Sanity["Sanity Checks"]:::test
        RopeTest["Positional Embedding Tests"]:::test
        OptTest["Optimizer Tests"]:::test
    end

    %% Documentation
    subgraph "Documentation & License"
        direction TB
        Readme["README.md"]:::doc
        License["LICENSE"]:::doc
    end

    %% Data Flow
    Data -->|"raw text"| Tokenizer
    Tokenizer -->|"token IDs"| Rope
    Rope -->|"positional embeddings"| Base
    Base -->|"hidden reps"| Llama
    Llama -->|"hidden reps"| Classifier
    Llama -->|"generated tokens"| Run
    Classifier -->|"logits"| Utils
    Utils -->|"loss/metrics"| Optimizer
    Optimizer -->|"update weights"| Base
    Optimizer -->|"update weights"| Llama

    %% Orchestration
    Run -->|invokes| Tokenizer
    Run -->|invokes| Llama
    Run -->|invokes| Classifier
    Run -->|invokes| Optimizer
    Run -->|loads config| Config
    Run -->|setup env| Setup

    %% Tests Hooks
    RopeTest --> Rope
    OptTest --> Optimizer
    Sanity --> Tokenizer
    Sanity --> Base
    Sanity --> Llama
    Sanity --> Classifier

    %% Click Events
    click Data "https://github.com/thviet79/asm1-development-llm-model/tree/main/data/"
    click Tokenizer "https://github.com/thviet79/asm1-development-llm-model/blob/main/tokenizer.py"
    click Rope "https://github.com/thviet79/asm1-development-llm-model/blob/main/rope.py"
    click RopeTest "https://github.com/thviet79/asm1-development-llm-model/blob/main/rope_test.py"
    click Base "https://github.com/thviet79/asm1-development-llm-model/blob/main/base_llama.py"
    click Llama "https://github.com/thviet79/asm1-development-llm-model/blob/main/llama.py"
    click Classifier "https://github.com/thviet79/asm1-development-llm-model/blob/main/classifier.py"
    click Utils "https://github.com/thviet79/asm1-development-llm-model/blob/main/utils.py"
    click Optimizer "https://github.com/thviet79/asm1-development-llm-model/blob/main/optimizer.py"
    click OptTest "https://github.com/thviet79/asm1-development-llm-model/blob/main/optimizer_test.py"
    click Sanity "https://github.com/thviet79/asm1-development-llm-model/blob/main/sanity_check.py"
    click Config "https://github.com/thviet79/asm1-development-llm-model/blob/main/config.py"
    click Setup "https://github.com/thviet79/asm1-development-llm-model/blob/main/setup.sh"
    click Readme "https://github.com/thviet79/asm1-development-llm-model/blob/main/README.md"
    click License "https://github.com/thviet79/asm1-development-llm-model/tree/main/LICENSE"

    %% Styles
    classDef data fill:#9f6,stroke:#393,stroke-width:1px
    classDef core fill:#6cf,stroke:#039,stroke-width:1px
    classDef train fill:#f90,stroke:#f60,stroke-width:1px
    classDef script fill:#ccc,stroke:#888,stroke-width:1px
    classDef test fill:#ddd,stroke:#666,stroke-width:1px
    classDef doc fill:#eee,stroke:#aaa,stroke-width:1px
```
