How mahrkeenerh/llama-games works
Main Components
A. Data Pipeline Layer
• Input data flows (images and text)
• Preprocessing components (modeling/preprocess_vit.py)
• Dataset handlers (data/datasets.py)
B. Model Architecture Layer
• Vision Transformer (ViT) component with two variants (224 and 448)
• Projection Layer
• LLaMA 3.1 8B Language Model with LoRA
• Model integration component (modeling/vit_gpt2.py)
C. Training Layer
• LoRA fine-tuning workflow
• Training utilities (training/train_utils.py)
• Callbacks system (training/callbacks.py)
D. Evaluation Layer
• Testing components (testing/test_utils.py)
• Multiple evaluation metrics
• Visualization tools
[Input Layer]
[Preprocessing Layer]
[Model Architecture]
[Training/Inference Layer]
[Evaluation Layer]
Specific Elements
a. Input Layer:
• Image input node
• Text input node (for training)
• Dataset connectors (COCO, Games dataset)
b. Preprocessing Layer:
• ViT preprocessor (224x224 and 448x448)
• Tokenization pipeline
• Data augmentation
c. Model Architecture:
• ViT Encoder (show both 224 and 448 variants)
• Projection Layer (highlight dimensionality mapping)
• LLaMA 3.1 8B with LoRA adaptation
• Merge strategies for multiple images
d. Training/Inference Pipeline:
• LoRA fine-tuning flow
• Full-precision vs 8-bit quantized paths
• Callback system integration
e. Evaluation Components:
• Automated metrics (BLEU, METEOR, ROUGE)
• Semantic analysis tools
• Visualization components
Technical Details to Highlight
• Model dimensions and sizes
• Key configuration parameters
• Integration points between components
• Data flow directions
• Memory optimization techniques
• Training/inference paths
Implementation Notes
• Maintain consistent spacing and alignment
• Brief annotations for complex components
• Creating both high-level and detailed views
• Version control integration points