# mahrkeenerh/llama-games architecture

> Architecture diagram of the GitHub repository mahrkeenerh/llama-games, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mahrkeenerh/llama-games
- Repository: https://github.com/mahrkeenerh/llama-games

## Overview

##### Main Components

A. Data Pipeline Layer
- Input data flows (images and text)
- Preprocessing components (modeling/preprocess_vit.py)
- Dataset handlers (data/datasets.py)

B. Model Architecture Layer
- Vision Transformer (ViT) component with two variants (224 and 448)
- Projection Layer
- LLaMA 3.1 8B Language Model with LoRA
- Model integration component (modeling/vit_gpt2.py)

C. Training Layer
- LoRA fine-tuning workflow
- Training utilities (training/train_utils.py)
- Callbacks system (training/callbacks.py)

D. Evaluation Layer
- Testing components (testing/test_utils.py)
- Multiple evaluation metrics
- Visualization tools

**[Input Layer]**

**[Preprocessing Layer]**

**[Model Architecture]**

**[Training/Inference Layer]**

**[Evaluation Layer]**

##### Specific Elements

a. Input Layer:
- Image input node
- Text input node (for training)
- Dataset connectors (COCO, Games dataset)

b. Preprocessing Layer:
- ViT preprocessor (224x224 and 448x448)
- Tokenization pipeline
- Data augmentation

c. Model Architecture:
- ViT Encoder (show both 224 and 448 variants)
- Projection Layer (highlight dimensionality mapping)
- LLaMA 3.1 8B with LoRA adaptation
- Merge strategies for multiple images

d. Training/Inference Pipeline:
- LoRA fine-tuning flow
- Full-precision vs 8-bit quantized paths
- Callback system integration

e. Evaluation Components:
- Automated metrics (BLEU, METEOR, ROUGE)
- Semantic analysis tools
- Visualization components

##### Technical Details to Highlight

- Model dimensions and sizes
- Key configuration parameters
- Integration points between components
- Data flow directions
- Memory optimization techniques
- Training/inference paths

##### Implementation Notes

- Maintain consistent spacing and alignment
- Brief annotations for complex components
- Creating both high-level and detailed views
- Version control integration points

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        IMG[Image Input]
        TXT[Text Input]
        DS["Dataset Handler"]:::data
    end

    subgraph Preprocessing
        VP["Vision Transformer Preprocessor"]:::modeling
        TK["Tokenizer"]:::models
        DA[Data Augmentation]
    end

    subgraph "Model Architecture"
        subgraph Vision
            V224["ViT-224"]:::models
            V448["ViT-448"]:::models
        end
        
        PL["Projection Layer"]
        
        subgraph Language
            LLAMA["LLaMA 3.1 8B"]
            LORA["LoRA Adaptation"]
            MI["Model Integration"]:::modeling
            LI["LLaMA Integration"]:::modeling
        end
    end

    subgraph "Training Pipeline"
        TP["Training Process"]:::training
        TU["Training Utilities"]:::training
        CB["Callbacks"]:::training
    end

    subgraph Evaluation
        TST["Testing Utils"]:::testing
        subgraph Metrics
            BLEU["BLEU Score"]
            METEOR["METEOR Score"]
            ROUGE["ROUGE Score"]
        end
        VIZ["Visualization Tools"]:::testing
        COCO["COCO Evaluation"]:::testing
        GAMES["Games Evaluation"]:::testing
    end

    IMG --> DS
    TXT --> DS
    DS --> VP
    DS --> TK
    
    VP --> V224
    VP --> V448
    V224 --> PL
    V448 --> PL
    
    PL --> MI
    MI --> LLAMA
    LLAMA --> LORA
    LI --> LORA
    
    LORA --> TP
    TU --> TP
    CB --> TP
    
    TP --> TST
    TST --> BLEU
    TST --> METEOR
    TST --> ROUGE
    TST --> COCO
    TST --> GAMES
    TST --> VIZ

    %% Style Definitions
    classDef models fill:#f9f,stroke:#333
    classDef modeling fill:#bbf,stroke:#333
    classDef data fill:#bfb,stroke:#333
    classDef training fill:#fbf,stroke:#333
    classDef testing fill:#fbb,stroke:#333

    %% Click Events
    click DS "https://github.com/Mahrkeenerh/Llama-Games/blob/master/data/datasets.py"
    click VP "https://github.com/Mahrkeenerh/Llama-Games/blob/master/modeling/preprocess_vit.py"
    click V224 "https://github.com/Mahrkeenerh/Llama-Games/tree/master/models/vit_224/"
    click V448 "https://github.com/Mahrkeenerh/Llama-Games/tree/master/models/vit_448/"
    click MI "https://github.com/Mahrkeenerh/Llama-Games/blob/master/modeling/vit_gpt2.py"
    click LI "https://github.com/Mahrkeenerh/Llama-Games/blob/master/modeling/llama_games.py"
    click TK "https://github.com/Mahrkeenerh/Llama-Games/tree/master/models/tokenizer/"
    click TP "https://github.com/Mahrkeenerh/Llama-Games/blob/master/training/train.py"
    click TU "https://github.com/Mahrkeenerh/Llama-Games/blob/master/training/train_utils.py"
    click CB "https://github.com/Mahrkeenerh/Llama-Games/blob/master/training/callbacks.py"
    click TST "https://github.com/Mahrkeenerh/Llama-Games/blob/master/testing/test_utils.py"
    click COCO "https://github.com/Mahrkeenerh/Llama-Games/tree/master/testing/captions/coco/"
    click GAMES "https://github.com/Mahrkeenerh/Llama-Games/tree/master/testing/captions/games/"
    click VIZ "https://github.com/Mahrkeenerh/Llama-Games/blob/master/testing/coco_visualization.ipynb"
```
