# thearkaprava/ski-models architecture

> Architecture diagram of the GitHub repository thearkaprava/ski-models, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thearkaprava/ski-models
- Repository: https://github.com/thearkaprava/ski-models

## Overview

##### Project type and purpose
- This repository implements a research‐oriented, deep‐learning system for zero‐shot action recognition and video captioning in Activities of Daily Living (ADL). It comprises two main pipelines:
- SKI-VLM: A two‐phase training pipeline that first injects skeleton information into a CLIP‐based vision-language model (SkeletonCLIP), then fine‐tunes on video+text pairs (SKIViFiCLIP).
- SKI-LVLM: An extension to large vision‐language models for video‐to‐language inference on datasets like Charades.

##### File structure
- Top-level entry scripts:
- P1_SkelCLIP_main.py (phase-1 skeleton‐CLIP training)
- P2_SKIVLM_main.py (phase-2 video‐text fine‐tuning)
- SKI_LVLM/: a self‐contained LVLM subproject with its own data annotation, evaluation, inference, demo, model, train, and utility modules.
- configs/: YAML config files for all experiments (phase-1, phase-2, zero-shot eval).
- datasets/, labels/: data‐loading pipelines and zero‐shot splits.
- trainers/: high‐level training wrappers for different phases.
- clip/, apex/, torchlight/: third-party submodules for CLIP, mixed-precision, and training utilities.
- scripts/: convenience bash/Python scripts for training and evaluation orchestration.
- utils/: shared configuration, logging, optimizer and utility functions.

##### README for additional insights
- The README describes two sequential experiment phases, and points to SKI_LVLM/README.md for the LVLM variant. It clarifies that skeleton data is only needed during training.

a. Main components
1. Data Sources
- Raw RGB videos
- 3D skeleton sequences
- Zero-shot split labels (labels/)
2. Data Preparation & Annotation
- datasets/build_*.py (CSV creation)
- SKI_LVLM/data_annotation (QA pair, dense descriptions generation)
3. Model Components
- Skeleton encoder (SkeletonCLIP)
- Vision encoder (CLIP backbone)
- Fusion & projection heads (consolidate.py, make_delta.py)
- LVLM modules (SKI_LVLM/skilvlm/model)
4. Training Pipelines
- Phase-1: SkeletonCLIP training (trainers/P1_PoseCLIP_model.py)
- Phase-2: SKIViFiCLIP training (trainers/P2_SKIVLM_model.py)
- SKI-LVLM training (skilvlm/train/train_skilvlm.py & llava_trainer.py)
5. Evaluation & Inference
- Zero-shot video classification (scripts/eval_SKIViFiCLIP.sh → evaluation modules)
- Video caption / QA inference (SKI_LVLM/skilvlm/inference.py, video_demo.py)
6. Demo & Visualization
- Gradio‐based UI (SKI_LVLM/skilvlm/demo)
- Notebook for model visualization (visualization/SKI_models_viz.ipynb)

b. Relationships & interactions
- Data Sources feed Data Preparation, producing CSVs & QA formats.
- CSVs feed DataLoader pipelines → Trainers.
- Trainers instantiate model components; during training: skeleton encoder and CLIP share embeddings via collaborative training.
- Trained Phase-2 model weights are consumed by SKI-LVLM modules for extended LVLM fine-tuning.
- Evaluation modules ingest trained weights + test CSVs → produce metrics / captions.
- Demo/UI wraps inference APIs.

c. Architectural patterns & principles
- Modular pipeline separation: data, model, training, evaluation, demo.
- Two‐phase curriculum learning: skeleton infusion → video‐text fine‐tuning.
- Plugin-style subprojects: SKI_LVLM lives alongside core SKI-VLM code, sharing utilities.
- Reuse of third-party submodules (clip, apex, torchlight) via vendoring.

d. Key technologies/frameworks
- PyTorch for models & training
- CLIP as the vision‐language backbone
- NVIDIA Apex for mixed precision
- Hydra / YAML config pattern for experiments
- Gradio for demos
- Bash scripts for orchestration

##### Elements
- Data Sources, Data Annotation, Data Loaders
- Model components (Skeleton Encoder, CLIP Vision Encoder, Fusion Heads, LVLM)
- Training steps (Phase-1, Phase-2, LVLM)
- Evaluation & Inference modules
- Demo/UI
- Data flow from sources → preprocessing → training → evaluation
- Weight hand‐off from Phase-2 → LVLM modules
- Blue: Data components
- Green: Model components
- Orange: Training pipelines
- Purple: Evaluation & Inference
- Grey: Third‐party dependencies

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        Videos["Raw RGB videos & 3D skeleton sequences"]:::data
        Labels["Zero-shot split labels"]:::data
    end
    click Videos "https://github.com/thearkaprava/ski-models/tree/main/datasets/"
    click Labels "https://github.com/thearkaprava/ski-models/tree/main/labels/"

    %% Data Preparation & Annotation
    subgraph "Data Preparation & Annotation"
        BuildScripts["build.py / build_AS.py / build_AS_p1.py"]:::data
        PipelineTools["pipeline.py / tools.py"]:::data
        QA_Gen["generate_QA_pairs.py"]:::data
        DenseDesc["generate_dense_descriptions.py"]:::data
        DescriptiveQA["generate_descriptive_qa.py"]:::data
        PoseDesc["generate_pose_descriptions_qa.py"]:::data
        QA_NTU["generate_qa_ntu.py"]:::data
        ProcessSeq["process_video_sequences.py"]:::data
    end
    click BuildScripts "https://github.com/thearkaprava/ski-models/blob/main/datasets/build.py"
    click PipelineTools "https://github.com/thearkaprava/ski-models/blob/main/datasets/pipeline.py"
    click QA_Gen "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/data_annotation/generate_QA_pairs.py"
    click DenseDesc "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/data_annotation/generate_dense_descriptions.py"
    click DescriptiveQA "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/data_annotation/generate_descriptive_qa.py"
    click PoseDesc "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/data_annotation/generate_pose_descriptions_qa.py"
    click QA_NTU "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/data_annotation/generate_qa_ntu.py"
    click ProcessSeq "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/data_annotation/process_video_sequences.py"

    %% Data Loaders
    subgraph "Data Loaders"
        Dataloaders["datasets/ data loaders"]:::data
    end
    click Dataloaders "https://github.com/thearkaprava/ski-models/tree/main/datasets/"

    %% Configuration Management
    subgraph "Configuration Management"
        Configs["YAML configs (configs/)"]:::data
    end
    click Configs "https://github.com/thearkaprava/ski-models/tree/main/configs/"

    %% Model Components
    subgraph "Model Components"
        CLIPBackbone["CLIP backbone"]:::model
        SkeletonEnc["Skeleton encoder"]:::model
        FusionHeads["Fusion & projection heads"]:::model
        LVLM_Mod["LVLM modules"]:::model
    end
    click CLIPBackbone "https://github.com/thearkaprava/ski-models/blob/main/clip/clip.py"
    click SkeletonEnc "https://github.com/thearkaprava/ski-models/blob/main/graph/ntu_rgb_d.py"
    click FusionHeads "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/skilvlm/model/consolidate.py"
    click LVLM_Mod "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/skilvlm/model/skilvlm.py"

    %% Third-party Dependencies
    subgraph "Third-party Dependencies"
        Apex["Apex"]:::third
        Torchlight["Torchlight"]:::third
        Tokenizer["Tokenizer & vocab"]:::third
    end
    click Apex "https://github.com/thearkaprava/ski-models/tree/main/apex/"
    click Torchlight "https://github.com/thearkaprava/ski-models/tree/main/torchlight/"
    click Tokenizer "https://github.com/thearkaprava/ski-models/blob/main/clip/simple_tokenizer.py"

    %% Training Pipelines
    subgraph "Training Pipelines"
        Phase1["Phase-1 SkeletonCLIP training"]:::train
        Phase2["Phase-2 SKIViFiCLIP training"]:::train
        LVLM_Train["SKI-LVLM training"]:::train
    end
    click Phase1 "https://github.com/thearkaprava/ski-models/blob/main/P1_SkelCLIP_main.py"
    click Phase2 "https://github.com/thearkaprava/ski-models/blob/main/P2_SKIVLM_main.py"
    click LVLM_Train "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/skilvlm/train/train_skilvlm.py"

    %% Evaluation & Inference
    subgraph "Evaluation & Inference"
        ZeroShotEval["Zero-shot video classification"]:::eval
        LVLM_Infer["SKI-LVLM inference APIs"]:::eval
        LVLM_EvalUtils["LVLM evaluation utilities"]:::eval
        ADL_Bench["ADL Descriptions benchmarks"]:::eval
    end
    click ZeroShotEval "https://github.com/thearkaprava/ski-models/blob/main/scripts/eval_SKIViFiCLIP.sh"
    click LVLM_Infer "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/skilvlm/inference.py"

    %% Demo & Visualization
    subgraph "Demo & Visualization"
        GradioUI["Gradio-based UI"]:::eval
        VizNotebook["Visualization notebook"]:::eval
    end
    click GradioUI "https://github.com/thearkaprava/ski-models/blob/main/SKI_LVLM/skilvlm/demo/chat.py"
    click VizNotebook "https://github.com/thearkaprava/ski-models/blob/main/visualization/SKI_models_viz.ipynb"

    %% Connections
    Videos -->|feeds| BuildScripts
    Labels -->|feeds| BuildScripts
    BuildScripts -->|produces CSVs| Dataloaders
    PipelineTools --> Dataloaders
    Dataloaders -->|feeds| Phase1
    Dataloaders -->|feeds| Phase2
    Configs --> Phase1
    Configs --> Phase2
    Configs --> LVLM_Train
    CLIPBackbone --> Phase1
    SkeletonEnc --> Phase1
    CLIPBackbone --> Phase2
    FusionHeads --> Phase2
    Phase1 -->|weights| Phase2
    Phase2 -->|weights| LVLM_Train
    LVLM_Mod --> LVLM_Train
    Apex --> Phase1
    Apex --> Phase2
    Torchlight --> Phase1
    Torchlight --> Phase2
    Tokenizer --> CLIPBackbone
    Phase2 --> ZeroShotEval
    Phase2 --> LVLM_Infer
    LVLM_Train --> LVLM_Infer
    LVLM_Infer --> ADL_Bench
    LVLM_Infer --> GradioUI
    LVLM_Infer --> VizNotebook

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef model fill:#90EE90,stroke:#000,stroke-width:1px
    classDef train fill:#FFA500,stroke:#000,stroke-width:1px
    classDef eval fill:#DDA0DD,stroke:#000,stroke-width:1px
    classDef third fill:#D3D3D3,stroke:#000,stroke-width:1px
```
