How thearkaprava/ski-models works
Project type and purpose
• This repository implements a research‐oriented, deep‐learning system for zero‐shot action recognition and video captioning in Activities of Daily Living (ADL). It comprises two main pipelines:
• SKI-VLM: A two‐phase training pipeline that first injects skeleton information into a CLIP‐based vision-language model (SkeletonCLIP), then fine‐tunes on video+text pairs (SKIViFiCLIP).
• SKI-LVLM: An extension to large vision‐language models for video‐to‐language inference on datasets like Charades.
File structure
• Top-level entry scripts:
• P1_SkelCLIP_main.py (phase-1 skeleton‐CLIP training)
• P2_SKIVLM_main.py (phase-2 video‐text fine‐tuning)
• SKI_LVLM/: a self‐contained LVLM subproject with its own data annotation, evaluation, inference, demo, model, train, and utility modules.
• configs/: YAML config files for all experiments (phase-1, phase-2, zero-shot eval).
• datasets/, labels/: data‐loading pipelines and zero‐shot splits.
• trainers/: high‐level training wrappers for different phases.
• clip/, apex/, torchlight/: third-party submodules for CLIP, mixed-precision, and training utilities.
• scripts/: convenience bash/Python scripts for training and evaluation orchestration.
• utils/: shared configuration, logging, optimizer and utility functions.
README for additional insights
• The README describes two sequential experiment phases, and points to SKI_LVLM/README.md for the LVLM variant. It clarifies that skeleton data is only needed during training.
a. Main components
1. Data Sources
• Raw RGB videos
• 3D skeleton sequences
• Zero-shot split labels (labels/)
2. Data Preparation & Annotation
• datasets/build_*.py (CSV creation)
• SKI_LVLM/data_annotation (QA pair, dense descriptions generation)
3. Model Components
• Skeleton encoder (SkeletonCLIP)
• Vision encoder (CLIP backbone)
• Fusion & projection heads (consolidate.py, make_delta.py)
• LVLM modules (SKI_LVLM/skilvlm/model)
4. Training Pipelines
• Phase-1: SkeletonCLIP training (trainers/P1_PoseCLIP_model.py)
• Phase-2: SKIViFiCLIP training (trainers/P2_SKIVLM_model.py)
• SKI-LVLM training (skilvlm/train/train_skilvlm.py & llava_trainer.py)
5. Evaluation & Inference
• Zero-shot video classification (scripts/eval_SKIViFiCLIP.sh → evaluation modules)
• Video caption / QA inference (SKI_LVLM/skilvlm/inference.py, video_demo.py)
6. Demo & Visualization
• Gradio‐based UI (SKI_LVLM/skilvlm/demo)
• Notebook for model visualization (visualization/SKI_models_viz.ipynb)
b. Relationships & interactions
• Data Sources feed Data Preparation, producing CSVs & QA formats.
• CSVs feed DataLoader pipelines → Trainers.
• Trainers instantiate model components; during training: skeleton encoder and CLIP share embeddings via collaborative training.
• Trained Phase-2 model weights are consumed by SKI-LVLM modules for extended LVLM fine-tuning.
• Evaluation modules ingest trained weights + test CSVs → produce metrics / captions.
• Demo/UI wraps inference APIs.
c. Architectural patterns & principles
• Modular pipeline separation: data, model, training, evaluation, demo.
• Two‐phase curriculum learning: skeleton infusion → video‐text fine‐tuning.
• Plugin-style subprojects: SKI_LVLM lives alongside core SKI-VLM code, sharing utilities.
• Reuse of third-party submodules (clip, apex, torchlight) via vendoring.
d. Key technologies/frameworks
• PyTorch for models & training
• CLIP as the vision‐language backbone
• NVIDIA Apex for mixed precision
• Hydra / YAML config pattern for experiments
• Gradio for demos
• Bash scripts for orchestration
Elements
• Data Sources, Data Annotation, Data Loaders
• Model components (Skeleton Encoder, CLIP Vision Encoder, Fusion Heads, LVLM)
• Training steps (Phase-1, Phase-2, LVLM)
• Evaluation & Inference modules
• Demo/UI
• Data flow from sources → preprocessing → training → evaluation
• Weight hand‐off from Phase-2 → LVLM modules
• Blue: Data components
• Green: Model components
• Orange: Training pipelines
• Purple: Evaluation & Inference
• Grey: Third‐party dependencies