How damo-nlp-sg/video-llama works
Project type and purpose
• This is a full-stack machine-learning system for audio-visual language understanding. It provides:
• Two cross-modal pretrain & fine-tune pipelines (Vision-Language and Audio-Language branches)
• An inference/demo layer for interactive video-grounded chat
• Configurable scripts (train.py, demo_audiovideo.py, demo_video.py) driven by YAML configs
• Integration of multiple third-party models: BLIP-2, MiniGPT-4, ImageBind, LLaMA/Vicuna decoders
File structure
• Top-level directories & files:
• train_configs/, eval_configs/: YAML configs for pre-training, fine-tuning, evaluation
• video_llama/: core Python package, subdivided into common, configs, datasets, models, processors, runners, tasks
• demo_*.py, train.py, setup.py, environment.yml, requirement.txt
• examples/: sample videos/audio; prompts/: text prompts
• Patterns & architectural cues:
• “datasets/builders” & “datasets/datasets”: data ingestion and loader utilities
• “models”: wrappers for encoders (Vision Q-Former, Audio Q-Former), language decoder interfaces
• “processors”: video/audio/frame transformations and augmentations
• “runners” & “tasks”: higher-level orchestrators for training/inference loops
README for additional insights
• Two-stage training: pre-training on WebVid & CC3M, then instruction-tuning on MiniGPT-4, LLaVA, VideoChat
• Demo setup: configuration → demo script → GPU inference
Elements
• Clear labels for each component (e.g., “Vision Q-Former”, “Audio Segment Embedder”)
• Blue for data related (datasets, dataloaders)
• Green for model components (encoders, Q-Formers, decoder)
• Orange for orchestration (scripts, runners, tasks)
• Grey for utilities/config/common modules
• Rectangles for processing modules
• Trapezoids or parallelograms for scripts/CLI