How bytedance/vidi works
Project type and purpose
• This is an open-source, CLI-based evaluation toolkit (no GUI frontend/backend) for “temporal retrieval” in video understanding, built around a research paper (Vidi: Large Multimodal Models for Video Understanding and Editing).
• Its core use case is: given a set of video IDs and text queries, download or load videos, run a model (when available) or supplied predictions, compare against ground-truth annotations, and generate evaluation metrics and plots.
File structure
• Top-level scripts and metadata: LICENSE.txt, README.md, install.sh, requirements.txt
• Data artifacts: VUE-TR_ground_truth.json (gold annotations), video_id.txt (list of YouTube IDs)
• Evaluation code: qa_eval.py
• Example results: results_Vidi.json, results_*.json (predictions from multiple models), results_table.csv, results/ (output plots)
• No src/ or frontend/ dir – it’s purely a command-line evaluation pipeline.
Extract additional insights from the README
• install.sh sets up the Python environment (pip install -r requirements.txt)
• qa_eval.py consumes a prediction file and outputs metrics & plots under ./results
• Users must download videos separately (e.g., via yt-dlp) using video_id.txt
• Predictions must follow a JSON schema: list of objects with query_id, video_id, duration, query, answer, task
Main components & interactions
a. External Data Sources
• YouTube (via yt-dlp) – downloads raw video files
• video_id.txt – list of IDs to fetch
b. Local Data Stores
• VUE-TR_ground_truth.json – annotation database
• predictions JSON files (e.g., results_Vidi.json) – model outputs
• results_table.csv and results/ (IoU_radar_plot.png, overall_IoU_plot.png) – evaluation outputs
c. Processing Components
• install.sh + Python environment (dep container)
• qa_eval.py – evaluation engine that:
– reads ground truth & predictions
– computes metrics (IoU, recall, etc.)
– writes CSV & visualization files
• (Optional future) inference.py – model inference component to produce predictions from videos & text queries
d. External Tools
• yt-dlp – video downloader
• Python runtime & libraries (listed in requirements.txt)
Relationships & data flows
• video_id.txt → [yt-dlp] → raw videos
• raw videos & text queries → [inference component] → predictions JSON
• VUE-TR_ground_truth.json + predictions JSON → [qa_eval.py] → metrics & plots → results folder
• install.sh → sets up Python env → all Python scripts
Architectural patterns & design principles
• Modular CLI pipeline: separate steps for data ingestion, inference, evaluation, visualization
• Clear data-processing flow with immutable data artifacts (JSON, CSV, PNG)
• Extensibility points: swap in new prediction files or add new metrics in qa_eval.py