How gyxxyg/trace works
Project type and purpose
• TRACE is a research codebase for a multimodal deep-learning system: a “Temporal Grounding Video LLM” that ingests videos, models them as causal events, and outputs timestamped captions or answers to queries.
• It provides pipelines for pretraining, fine-tuning (SFT), zero-shot inference, and benchmark evaluations (VTG, MVBench, VideoMME, etc.).
File structure
• Top-level directories and files:
• scripts/ ← bash wrappers and entrypoints for inference & training
– scripts/inference/inference.py
– scripts/train/pretrain-128.sh, sft-128.sh, sft-youcook2.sh
• trace/ ← core Python modules
– trace/model/ ← model construction: language_model, multimodal_encoder (clip, time, sync, score), projector
– trace/trace_trainer.py, train_mt.py, train_mt_npu.py ← training loops
– trace/conversation.py, constants.py, utils.py ← support utilities
– trace/eval/ ← evaluation suites by task (vtg, mvbench, videomme) & helper scripts
– trace/metrics/ ← structured metric implementations (dvc/SODA, tvg, vhd)
• Configuration & environment:
• requirements.txt, install_requirements-npu.sh
• zero3.json (ZeRO3 config for distributed training)
README for additional insights
• Two‐stage training: Stage-1 pretrain, Stage-2 SFT, plus downstream fine-tune scripts.
• Evaluations are organized into separate subfolders with shell and Python scripts.