# gyxxyg/trace architecture

> Architecture diagram of the GitHub repository gyxxyg/trace, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gyxxyg/trace
- Repository: https://github.com/gyxxyg/trace

## Overview

##### Project type and purpose
- TRACE is a research codebase for a multimodal deep-learning system: a “Temporal Grounding Video LLM” that ingests videos, models them as causal events, and outputs timestamped captions or answers to queries.
- It provides pipelines for pretraining, fine-tuning (SFT), zero-shot inference, and benchmark evaluations (VTG, MVBench, VideoMME, etc.).

##### File structure
- Top-level directories and files:
- scripts/            ← bash wrappers and entrypoints for inference & training
– scripts/inference/inference.py
– scripts/train/pretrain-128.sh, sft-128.sh, sft-youcook2.sh
- trace/              ← core Python modules
– trace/model/      ← model construction: language_model, multimodal_encoder (clip, time, sync, score), projector
– trace/trace_trainer.py, train_mt.py, train_mt_npu.py   ← training loops
– trace/conversation.py, constants.py, utils.py          ← support utilities
– trace/eval/        ← evaluation suites by task (vtg, mvbench, videomme) & helper scripts
– trace/metrics/     ← structured metric implementations (dvc/SODA, tvg, vhd)
- Configuration & environment:
- requirements.txt, install_requirements-npu.sh
- zero3.json (ZeRO3 config for distributed training)

##### README for additional insights
- Two‐stage training: Stage-1 pretrain, Stage-2 SFT, plus downstream fine-tune scripts.
- Evaluations are organized into separate subfolders with shell and Python scripts.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        DataAssets["Video Assets"]:::data
        Prompts["Annotations & Prompts"]:::data
    end

    %% Preprocessing
    subgraph "Preprocessing" 
        FrameSampler["Frame Sampler & Feature Extraction"]:::pre
        subgraph "Multimodal Encoder Components"
            ClipEnc["CLIP Encoder"]:::pre
            TimeEnc["Time Encoder"]:::pre
            SyncEnc["Sync Encoder"]:::pre
            ScoreEnc["Score Encoder"]:::pre
        end
    end

    %% Model Core
    subgraph "Model Core"
        MultEnc["Multimodal Encoder"]:::model
        Projector["Multimodal Projector"]:::model
        LLM["LLM Backbone"]:::model
        EventModule["Event Sequence Manager"]:::model
        Builder["Core Model Builder"]:::model
    end

    %% Execution Engines
    subgraph "Execution Engines"
        subgraph "Training Engines"
            Trainer["Trainer"]:::engine
            TrainMT["train_mt.py"]:::engine
            TrainNPU["train_mt_npu.py"]:::engine
            PretrainScript["pretrain-128.sh"]:::engine
            SFT128["sft-128.sh"]:::engine
            SFTYoucook["sft-youcook2.sh"]:::engine
        end
        Inference["Inference Engine"]:::engine
    end

    %% Evaluation Framework
    subgraph "Evaluation Framework"
        Evaluator["Task Evaluators"]:::eval
        ReformatTVG["reformat_tvg.py"]:::eval
        MVBench["MVBench"]:::eval
        VideoMME["VideoMME"]:::eval
        ReformatDVC["reformat_dvc.py"]:::eval
        ReformatVHD["reformat_vhd.py"]:::eval
        subgraph "Metric Calculators"
            MetricsDVC["DVC/SODA Metrics"]:::eval
            MetricsTVG["TVG Metrics"]:::eval
            MetricsVHD["VHD Metrics"]:::eval
        end
    end

    %% Deployment & Utilities
    subgraph "Deployment & Utilities"
        Zero3["zero3.json"]:::deploy
        Utils["Utilities & Constants"]:::util
        Constants["constants.py"]:::util
        MMUtils["mm_utils.py"]:::util
        Conversation["conversation.py"]:::util
    end

    %% Data Flow
    DataAssets --> FrameSampler
    Prompts --> FrameSampler
    FrameSampler --> ClipEnc
    FrameSampler --> TimeEnc
    FrameSampler --> SyncEnc
    FrameSampler --> ScoreEnc
    ClipEnc --> MultEnc
    TimeEnc --> MultEnc
    SyncEnc --> MultEnc
    ScoreEnc --> MultEnc
    MultEnc --> Projector
    Projector --> LLM
    LLM --> EventModule
    Builder --> MultEnc
    Builder --> Projector
    Builder --> LLM
    Builder --> EventModule

    %% Execution Flow
    Trainer --> Builder
    Trainer --> MultEnc
    Trainer --> LLM
    Trainer --> EventModule
    TrainMT --> Trainer
    TrainNPU --> Trainer
    PretrainScript --> Trainer
    SFT128 --> Trainer
    SFTYoucook --> Trainer
    Inference --> MultEnc
    Inference --> LLM
    Inference --> EventModule

    %% Evaluation Flow
    Inference --> Evaluator
    Evaluator --> ReformatTVG
    Evaluator --> MVBench
    Evaluator --> VideoMME
    Evaluator --> ReformatDVC
    Evaluator --> ReformatVHD
    ReformatTVG --> MetricsTVG
    MVBench --> MetricsDVC
    VideoMME --> MetricsTVG
    VideoMME --> MetricsVHD
    ReformatDVC --> MetricsDVC
    ReformatVHD --> MetricsVHD

    %% Deployment & Utils usage
    Zero3 --> Trainer
    Utils --> Trainer
    Constants --> Trainer
    MMUtils --> MultEnc
    Conversation --> EventModule

    %% Click Events
    click DataAssets "https://github.com/gyxxyg/trace/tree/master/assets/"
    click Prompts "https://github.com/gyxxyg/trace/tree/master/trace/prompts/"
    click FrameSampler "https://github.com/gyxxyg/trace/tree/master/trace/model/multimodal_encoder/"
    click ClipEnc "https://github.com/gyxxyg/trace/blob/master/trace/model/multimodal_encoder/clip_encoder.py"
    click TimeEnc "https://github.com/gyxxyg/trace/blob/master/trace/model/multimodal_encoder/time_encoder.py"
    click SyncEnc "https://github.com/gyxxyg/trace/blob/master/trace/model/multimodal_encoder/sync_encoder.py"
    click ScoreEnc "https://github.com/gyxxyg/trace/blob/master/trace/model/multimodal_encoder/score_encoder.py"
    click LLM "https://github.com/gyxxyg/trace/blob/master/trace/model/language_model/trace_mistral.py"
    click Projector "https://github.com/gyxxyg/trace/tree/master/trace/model/multimodal_projector/"
    click EventModule "https://github.com/gyxxyg/trace/blob/master/trace/model/trace_arch.py"
    click Builder "https://github.com/gyxxyg/trace/blob/master/trace/model/builder.py"
    click Trainer "https://github.com/gyxxyg/trace/blob/master/trace/trace_trainer.py"
    click TrainMT "https://github.com/gyxxyg/trace/blob/master/trace/train_mt.py"
    click TrainNPU "https://github.com/gyxxyg/trace/blob/master/trace/train_mt_npu.py"
    click PretrainScript "https://github.com/gyxxyg/trace/blob/master/scripts/train/pretrain-128.sh"
    click SFT128 "https://github.com/gyxxyg/trace/blob/master/scripts/train/sft-128.sh"
    click SFTYoucook "https://github.com/gyxxyg/trace/blob/master/scripts/train/sft-youcook2.sh"
    click Inference "https://github.com/gyxxyg/trace/blob/master/scripts/inference/inference.py"
    click Evaluator "https://github.com/gyxxyg/trace/tree/master/trace/eval/"
    click ReformatTVG "https://github.com/gyxxyg/trace/blob/master/trace/eval/reformat_tvg.py"
    click MVBench "https://github.com/gyxxyg/trace/tree/master/trace/eval/mvbench/"
    click VideoMME "https://github.com/gyxxyg/trace/tree/master/trace/eval/videomme/"
    click ReformatDVC "https://github.com/gyxxyg/trace/blob/master/trace/eval/reformat_dvc.py"
    click ReformatVHD "https://github.com/gyxxyg/trace/blob/master/trace/eval/reformat_vhd.py"
    click MetricsDVC "https://github.com/gyxxyg/trace/tree/master/trace/metrics/dvc/"
    click MetricsTVG "https://github.com/gyxxyg/trace/tree/master/trace/metrics/tvg/"
    click MetricsVHD "https://github.com/gyxxyg/trace/tree/master/trace/metrics/vhd/"
    click Zero3 "https://github.com/gyxxyg/trace/blob/master/scripts/zero3.json"
    click Utils "https://github.com/gyxxyg/trace/blob/master/trace/utils.py"
    click Constants "https://github.com/gyxxyg/trace/blob/master/trace/constants.py"
    click MMUtils "https://github.com/gyxxyg/trace/blob/master/trace/mm_utils.py"
    click Conversation "https://github.com/gyxxyg/trace/blob/master/trace/conversation.py"

    %% Styles
    classDef data fill:#BBDEFB,stroke:#0D47A1,color:#0D47A1;
    classDef pre fill:#C8E6C9,stroke:#1B5E20,color:#1B5E20;
    classDef model fill:#FFE0B2,stroke:#E65100,color:#E65100;
    classDef engine fill:#D1C4E9,stroke:#4A148C,color:#4A148C;
    classDef eval fill:#F8BBD0,stroke:#880E4F,color:#880E4F;
    classDef util fill:#ECEFF1,stroke:#37474F,color:#37474F;
    classDef deploy fill:#FFF9C4,stroke:#F57F17,color:#F57F17;
```
