# nahidalam/maya architecture

> Architecture diagram of the GitHub repository nahidalam/maya, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nahidalam/maya
- Repository: https://github.com/nahidalam/maya

## Overview

##### Main System Components

a) Core Model Components:
- Language Model Layer (Cohere's Aya-35B)
- Multimodal Encoder (CLIP/SigLIP)
- Multimodal Projector (for connecting vision and language)

b) Processing Pipeline:
- Input Processing Layer (handling 8 languages)
- Vision Processing Pipeline
- Language Processing Pipeline
- Multi-modal Fusion Layer

##### Key Architectural Elements

a) Data Flow Section:
**Input → Preprocessing → Model Pipeline → Output Generation**
- Image Input Stream
- Text Input Stream (8 languages)
- Unified Output Generation

b) Training Pipeline:
**Pretraining → Instruction Tuning → Evaluation**

##### Specific Implementation Details

**1. Input Layer**
- Image Input Handler
- Multilingual Text Input Handler (EN, CN, FR, ES, RU, JP, AR, HI)

**2. Processing Core**
- Vision Encoder (CLIP/SigLIP)
- Language Model (Aya-35B)
- Projection Layer

**3. Training Infrastructure**
- Pretraining Module
- Instruction Tuning Module
- Flash Attention Implementation

**4. Evaluation System**
- Multiple Benchmark Evaluators
- Performance Metrics Collection

##### Important Technical Details to Highlight

- Flash Attention implementation
- CUDA compatibility (v12.4)
- Multiple evaluation benchmarks integration
- Dataset handling for various sources (COCO, GQA, OCR-VQA, etc.)

##### Integration Points

Clear connections between:
- Vision and language processing streams
- Training and evaluation pipelines
- Data input and model components

##### Evaluation system comprehensiveness

Use a hierarchical layout with:
- Top level: Main system components
- Second level: Detailed processing pipelines
- Third level: Implementation specifics

This system design should effectively communicate both the architectural complexity and the practical implementation details of the Maya multimodal multilingual LLM system.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input["Input Layer"]
        IMG[Image Input]
        TXT[Text Input]
    end

    subgraph Processing["Processing Layer"]
        direction TB
        subgraph ImageProc["Image Processing"]
            CLIP["CLIP Encoder"]:::encoder
            SIG["SigLIP Encoder"]:::encoder
        end

        subgraph TextProc["Text Processing"]
            LM["Language Model (Aya-35B)"]:::lm
        end
    end

    subgraph Core["Core Model Components"]
        direction TB
        MME["Multimodal Encoder"]:::core
        MMP["Multimodal Projector"]:::core
        MA["Model Architecture"]:::core
    end

    subgraph Training["Training Pipeline"]
        direction TB
        PRE["Pretraining Module"]:::training
        FT["Fine-tuning Module"]:::training
        OPT["Training Optimizations"]:::training
        subgraph TO["Optimization Components"]
            FA["Flash Attention"]:::opt
            XF["XFormers"]:::opt
        end
    end

    subgraph Eval["Evaluation System"]
        direction TB
        subgraph Benchmarks["Benchmark Evaluation"]
            MMB["MMBench"]:::eval
            POPE["POPE"]:::eval
            VQA["VQA-v2"]:::eval
            GQA["GQA"]:::eval
        end
        ML["Multilingual Evaluation"]:::eval
        PERF["Performance Metrics"]:::eval
    end

    OUT["Output Generation"]

    %% Connections
    IMG --> CLIP & SIG
    TXT --> LM
    CLIP & SIG --> MME
    LM --> MME
    MME --> MMP
    MMP --> MA
    MA --> OUT

    MME & MMP & MA --> Training
    Training --> Eval

    %% Click Events
    click LM "https://github.com/nahidalam/maya/blob/main/llava/model/language_model/llava_cohere.py"
    click CLIP "https://github.com/nahidalam/maya/blob/main/llava/model/multimodal_encoder/clip_encoder.py"
    click SIG "https://github.com/nahidalam/maya/blob/main/llava/model/multimodal_encoder/siglip_encoder.py"
    click MMP "https://github.com/nahidalam/maya/blob/main/llava/model/multimodal_projector/builder.py"
    click MA "https://github.com/nahidalam/maya/blob/main/llava/model/llava_arch.py"
    click PRE "https://github.com/nahidalam/maya/blob/main/scripts/maya/pretrain_aya_siglip.sh"
    click FT "https://github.com/nahidalam/maya/blob/main/scripts/maya/finetune_aya_siglip.sh"
    click FA "https://github.com/nahidalam/maya/blob/main/llava/train/llama_flash_attn_monkey_patch.py"
    click XF "https://github.com/nahidalam/maya/blob/main/llava/train/llama_xformers_attn_monkey_patch.py"
    click MMB "https://github.com/nahidalam/maya/blob/main/scripts/maya/eval/mmbench.sh"
    click POPE "https://github.com/nahidalam/maya/blob/main/scripts/maya/eval/pope.sh"
    click VQA "https://github.com/nahidalam/maya/blob/main/scripts/maya/eval/vqav2.sh"
    click GQA "https://github.com/nahidalam/maya/blob/main/scripts/maya/eval/gqa.sh"
    click ML "https://github.com/nahidalam/maya/blob/main/scripts/maya/eval/palo_multilingual_llava_bench.sh"

    %% Styling
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef encoder fill:#2ecc71,stroke:#27ae60,color:white
    classDef lm fill:#e74c3c,stroke:#c0392b,color:white
    classDef training fill:#f1c40f,stroke:#f39c12,color:black
    classDef eval fill:#9b59b6,stroke:#8e44ad,color:white
    classDef opt fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Legend
    subgraph Legend
        direction TB
        L1["Core Components"]:::core
        L2["Encoders"]:::encoder
        L3["Language Model"]:::lm
        L4["Training"]:::training
        L5["Evaluation"]:::eval
        L6["Optimizations"]:::opt
    end
```
