# bytedance/vidi architecture

> Architecture diagram of the GitHub repository bytedance/vidi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance/vidi
- Repository: https://github.com/bytedance/vidi

## Overview

##### Project type and purpose
- This is an open-source, CLI-based evaluation toolkit (no GUI frontend/backend) for “temporal retrieval” in video understanding, built around a research paper (Vidi: Large Multimodal Models for Video Understanding and Editing).
- Its core use case is: given a set of video IDs and text queries, download or load videos, run a model (when available) or supplied predictions, compare against ground-truth annotations, and generate evaluation metrics and plots.

##### File structure
- Top-level scripts and metadata: LICENSE.txt, README.md, install.sh, requirements.txt
- Data artifacts: VUE-TR_ground_truth.json (gold annotations), video_id.txt (list of YouTube IDs)
- Evaluation code: qa_eval.py
- Example results: results_Vidi.json, results_*.json (predictions from multiple models), results_table.csv, results/ (output plots)
- No src/ or frontend/ dir – it’s purely a command-line evaluation pipeline.

##### Extract additional insights from the README
- install.sh sets up the Python environment (pip install -r requirements.txt)
- qa_eval.py consumes a prediction file and outputs metrics & plots under ./results
- Users must download videos separately (e.g., via yt-dlp) using video_id.txt
- Predictions must follow a JSON schema: list of objects with query_id, video_id, duration, query, answer, task

##### Main components & interactions
a. External Data Sources
- YouTube (via yt-dlp) – downloads raw video files
- video_id.txt – list of IDs to fetch
b. Local Data Stores
- VUE-TR_ground_truth.json – annotation database
- predictions JSON files (e.g., results_Vidi.json) – model outputs
- results_table.csv and results/ (IoU_radar_plot.png, overall_IoU_plot.png) – evaluation outputs
c. Processing Components
- install.sh + Python environment (dep container)
- qa_eval.py – evaluation engine that:
– reads ground truth & predictions
– computes metrics (IoU, recall, etc.)
– writes CSV & visualization files
- (Optional future) inference.py – model inference component to produce predictions from videos & text queries
d. External Tools
- yt-dlp – video downloader
- Python runtime & libraries (listed in requirements.txt)

##### Relationships & data flows
- video_id.txt → [yt-dlp] → raw videos
- raw videos & text queries → [inference component] → predictions JSON
- VUE-TR_ground_truth.json + predictions JSON → [qa_eval.py] → metrics & plots → results folder
- install.sh → sets up Python env → all Python scripts

##### Architectural patterns & design principles
- Modular CLI pipeline: separate steps for data ingestion, inference, evaluation, visualization
- Clear data-processing flow with immutable data artifacts (JSON, CSV, PNG)
- Extensibility points: swap in new prediction files or add new metrics in qa_eval.py

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion
    subgraph "Data Ingestion"
        VideoIDs["video_id.txt"]:::datastore
        Ytdlp["YouTube (via yt-dlp)"]:::external
        RawVideos["Raw Videos (MP4)"]:::datastore
    end
    VideoIDs -->|"IDs"| Ytdlp
    Ytdlp -->|"MP4 video"| RawVideos

    %% Optional Inference
    subgraph "Inference (Optional)"
        Inference["inference.py"]:::processing_optional
        Query["Text Queries (CLI)"]:::external
        Predictions["Prediction Artifacts (results_*.json)"]:::datastore
    end
    RawVideos -->|"MP4, Query"| Inference
    Query -->|"Text Query"| Inference
    Inference -->|"Predictions JSON"| Predictions

    %% Evaluation
    subgraph "Evaluation"
        GroundTruth["VUE-TR_ground_truth.json"]:::datastore
        Eval["qa_eval.py"]:::processing
        MetricsTable["results_table.csv"]:::datastore
        Visualization["results/"]:::datastore
    end
    GroundTruth -->|"Ground Truth JSON"| Eval
    Predictions -->|"Prediction JSON"| Eval
    Eval -->|"CSV Metrics"| MetricsTable
    Eval -->|"PNG Plots"| Visualization

    %% Dependency Setup
    subgraph "Dependency Setup"
        Install["install.sh"]:::processing
        Requirements["requirements.txt"]:::datastore
    end
    Install -->|"reads"| Requirements
    Install -->|"sets up env"| Eval
    Install -->|"sets up env"| Inference

    %% Legend
    subgraph Legend
        DS["Data Store"]:::datastore
        PR["Processing"]:::processing
        EX["External/Tool"]:::external
        OP["Optional"]:::processing_optional
    end

    %% Click Events
    click VideoIDs "https://github.com/bytedance/vidi/blob/main/video_id.txt"
    click GroundTruth "https://github.com/bytedance/vidi/blob/main/VUE-TR_ground_truth.json"
    click Install "https://github.com/bytedance/vidi/blob/main/install.sh"
    click Requirements "https://github.com/bytedance/vidi/blob/main/requirements.txt"
    click Eval "https://github.com/bytedance/vidi/blob/main/qa_eval.py"
    click Predictions "https://github.com/bytedance/vidi/blob/main/results_Vidi.json"
    click MetricsTable "https://github.com/bytedance/vidi/blob/main/results_table.csv"
    click Visualization "https://github.com/bytedance/vidi/tree/main/results/"

    %% Styles
    classDef datastore fill:#ADD8E6,stroke:#000
    classDef processing fill:#90EE90,stroke:#000
    classDef external fill:#FFA500,stroke:#000
    classDef processing_optional fill:transparent,stroke:#000,stroke-dasharray:5 5
    class VideoIDs,GroundTruth,RawVideos,Predictions,MetricsTable,Visualization datastore
    class Eval,Install processing
    class Ytdlp,Query external
    class Inference processing_optional
    class DS,PR,EX,OP fill-opacity:0.3,font-style:italic
```
