# harryhsing/echotraffic architecture

> Architecture diagram of the GitHub repository harryhsing/echotraffic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harryhsing/echotraffic
- Repository: https://github.com/harryhsing/echotraffic

## Overview

EchoTraffic is an audio-visual traffic anomaly understanding model built on Video-LLaMA. Its documented user path is to supply a video and question, encode video/audio context, and generate a response; its training path prepares annotated datasets and optimizes the model through pretraining and supervised fine-tuning. The README also identifies AV-TAU and InternVid as distinct data capabilities. Several domain modules are indexed but not excerpted, so only documented or source-index-supported links are shown; unspecified connections are omitted.

Main entry points: [`inference.py`](https://github.com/harryhsing/echotraffic/blob/main/inference.py) (Inference entry), [`video_llama/models/video_llama.py`](https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/video_llama.py) (EchoTraffic model), [`video_llama/tasks/base_task.py`](https://github.com/harryhsing/echotraffic/blob/main/video_llama/tasks/base_task.py) (Training task), [`video_llama/processors/video_processor.py`](https://github.com/harryhsing/echotraffic/blob/main/video_llama/processors/video_processor.py) (Video and audio processing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Question answering"]
  node_inference["Inference entry<br/>[inference.py]"]
  node_chat["Video chat"]
end

subgraph group_data["Dataset preparation"]
  node_avtau["AV-TAU data"]
  node_video_instruction["Video instruction data"]
  node_internvid["InternVid data"]
  node_builders["Dataset builders"]
  node_base_dataset["Dataset base<br/>[base_dataset.py]"]
  node_data_utils["Dataset utilities<br/>[data_utils.py]"]
end

subgraph group_model["Multimodal model"]
  node_video_processor["Video and audio processing<br/>[video_processor.py]"]
  node_video_transforms["Video transforms"]
  node_imagebind_data["Audio-visual input data<br/>[data.py]"]
  node_imagebind["ImageBind encoder<br/>[imagebind_model.py]"]
  node_video_llama["EchoTraffic model<br/>[video_llama.py]"]
  node_qformer["Q-Former<br/>[Qformer.py]"]
  node_llama["Language model<br/>[modeling_llama.py]"]
end

subgraph group_training["Training runtime"]
  node_config["Config loading<br/>[config.py]"]
  node_registry["Component registry<br/>[registry.py]"]
  node_train_entry["Training entry<br/>[train.py]"]
  node_task["Training task<br/>[base_task.py]"]
  node_runner["Training runner<br/>[runner_base.py]"]
  node_video_pretrain["Video-text pretraining"]
end

node_user(("Analyst"))
node_external_video["Video input"]
node_answer["Generated answer"]
node_avtau_source["AV-TAU annotations and clips"]
node_internvid_source["InternVid annotations and clips"]
node_checkpoint["Model checkpoint"]

node_user -->|"asks about video"| node_inference
node_external_video -->|"provides video"| node_inference
node_inference -->|"loads config"| node_config
node_inference -->|"builds processor"| node_video_processor
node_inference -->|"creates chat"| node_chat
node_inference -->|"loads model"| node_video_llama
node_chat -->|"processes video"| node_video_processor
node_chat -->|"uses input data"| node_imagebind_data
node_video_llama -.->|"uses Q-Former"| node_qformer
node_video_llama -.->|"uses encoder"| node_imagebind
node_video_llama -.->|"uses language model"| node_llama
node_chat -->|"returns response"| node_answer
node_avtau_source -->|"loads annotations"| node_avtau
node_internvid_source -->|"loads annotations"| node_internvid
node_avtau -->|"extends dataset base"| node_base_dataset
node_video_instruction -->|"extends dataset base"| node_base_dataset
node_avtau -->|"uses processor"| node_video_processor
node_video_instruction -->|"uses processor"| node_video_processor
node_internvid -.->|"registered for building"| node_builders
node_builders -->|"builds datasets"| node_base_dataset
node_video_processor -->|"applies transforms"| node_video_transforms
node_imagebind_data -->|"prepares encoder input"| node_imagebind
node_train_entry -->|"loads config"| node_config
node_train_entry -->|"sets up task"| node_task
node_train_entry -->|"starts runner"| node_runner
node_task -->|"builds model"| node_video_llama
node_task -->|"builds datasets"| node_builders
node_runner -->|"runs task steps"| node_task
node_video_pretrain -->|"specializes task"| node_task
node_checkpoint -->|"loads weights"| node_inference
node_registry -->|"resolves builders"| node_builders
node_registry -->|"resolves models"| node_video_llama

click node_inference "https://github.com/harryhsing/echotraffic/blob/main/inference.py"
click node_chat "https://github.com/harryhsing/echotraffic/blob/main/video_llama/conversation/conversation_video.py"
click node_config "https://github.com/harryhsing/echotraffic/blob/main/video_llama/common/config.py"
click node_registry "https://github.com/harryhsing/echotraffic/blob/main/video_llama/common/registry.py"
click node_avtau "https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/my_dataset_with_audio.py"
click node_video_instruction "https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/video_instruct_dataset.py"
click node_internvid "https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/internvid_datasets.py"
click node_builders "https://github.com/harryhsing/echotraffic/tree/main/video_llama/datasets/builders"
click node_base_dataset "https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/base_dataset.py"
click node_data_utils "https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/data_utils.py"
click node_video_processor "https://github.com/harryhsing/echotraffic/blob/main/video_llama/processors/video_processor.py"
click node_video_transforms "https://github.com/harryhsing/echotraffic/blob/main/video_llama/processors/transforms_video.py"
click node_imagebind_data "https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/ImageBind/data.py"
click node_imagebind "https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/ImageBind/models/imagebind_model.py"
click node_video_llama "https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/video_llama.py"
click node_qformer "https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/Qformer.py"
click node_llama "https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/modeling_llama.py"
click node_train_entry "https://github.com/harryhsing/echotraffic/blob/main/train.py"
click node_task "https://github.com/harryhsing/echotraffic/blob/main/video_llama/tasks/base_task.py"
click node_runner "https://github.com/harryhsing/echotraffic/blob/main/video_llama/runners/runner_base.py"
click node_video_pretrain "https://github.com/harryhsing/echotraffic/blob/main/video_llama/tasks/video_text_pretrain.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_inference,node_chat toneBlue
class node_avtau,node_video_instruction,node_internvid,node_builders,node_base_dataset,node_data_utils toneAmber
class node_video_processor,node_video_transforms,node_imagebind_data,node_imagebind,node_video_llama,node_qformer,node_llama toneMint
class node_config,node_registry,node_train_entry,node_task,node_runner,node_video_pretrain toneRose
class node_user,node_external_video,node_answer,node_avtau_source,node_internvid_source,node_checkpoint toneIndigo
```

## Components

### Question answering

- [Inference entry](https://github.com/harryhsing/echotraffic/blob/main/inference.py): `inference.py`
- [Video chat](https://github.com/harryhsing/echotraffic/blob/main/video_llama/conversation/conversation_video.py): `video_llama/conversation/conversation_video.py`

### Dataset preparation

- [AV-TAU data](https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/my_dataset_with_audio.py): `video_llama/datasets/datasets/my_dataset_with_audio.py`
- [Video instruction data](https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/video_instruct_dataset.py): `video_llama/datasets/datasets/video_instruct_dataset.py`
- [InternVid data](https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/internvid_datasets.py): `video_llama/datasets/datasets/internvid_datasets.py`
- [Dataset builders](https://github.com/harryhsing/echotraffic/tree/main/video_llama/datasets/builders): `video_llama/datasets/builders`
- [Dataset base](https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/datasets/base_dataset.py): `video_llama/datasets/datasets/base_dataset.py`
- [Dataset utilities](https://github.com/harryhsing/echotraffic/blob/main/video_llama/datasets/data_utils.py): `video_llama/datasets/data_utils.py`

### Multimodal model

- [Video and audio processing](https://github.com/harryhsing/echotraffic/blob/main/video_llama/processors/video_processor.py): `video_llama/processors/video_processor.py`
- [Video transforms](https://github.com/harryhsing/echotraffic/blob/main/video_llama/processors/transforms_video.py): `video_llama/processors/transforms_video.py`
- [Audio-visual input data](https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/ImageBind/data.py): `video_llama/models/ImageBind/data.py`
- [ImageBind encoder](https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/ImageBind/models/imagebind_model.py): `video_llama/models/ImageBind/models/imagebind_model.py`
- [EchoTraffic model](https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/video_llama.py): `video_llama/models/video_llama.py`
- [Q-Former](https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/Qformer.py): `video_llama/models/Qformer.py`
- [Language model](https://github.com/harryhsing/echotraffic/blob/main/video_llama/models/modeling_llama.py): `video_llama/models/modeling_llama.py`

### Training runtime

- [Config loading](https://github.com/harryhsing/echotraffic/blob/main/video_llama/common/config.py): `video_llama/common/config.py`
- [Component registry](https://github.com/harryhsing/echotraffic/blob/main/video_llama/common/registry.py): `video_llama/common/registry.py`
- [Training entry](https://github.com/harryhsing/echotraffic/blob/main/train.py): `train.py`
- [Training task](https://github.com/harryhsing/echotraffic/blob/main/video_llama/tasks/base_task.py): `video_llama/tasks/base_task.py`
- [Training runner](https://github.com/harryhsing/echotraffic/blob/main/video_llama/runners/runner_base.py): `video_llama/runners/runner_base.py`
- [Video-text pretraining](https://github.com/harryhsing/echotraffic/blob/main/video_llama/tasks/video_text_pretrain.py): `video_llama/tasks/video_text_pretrain.py`

### Other components

- **Analyst**
- **Video input**
- **Generated answer**
- **AV-TAU annotations and clips**
- **InternVid annotations and clips**
- **Model checkpoint**

## Connections

- Analyst → Inference entry: asks about video
- Video input → Inference entry: provides video
- Inference entry → Config loading: loads config (evidence: `inference.py`)
- Inference entry → Video and audio processing: builds processor (evidence: `inference.py`)
- Inference entry → Video chat: creates chat (evidence: `inference.py`)
- Inference entry → EchoTraffic model: loads model
- Video chat → Video and audio processing: processes video (evidence: `video_llama/conversation/conversation_video.py`)
- Video chat → Audio-visual input data: uses input data (evidence: `video_llama/conversation/conversation_video.py`)
- EchoTraffic model → Q-Former: uses Q-Former
- EchoTraffic model → ImageBind encoder: uses encoder
- EchoTraffic model → Language model: uses language model
- Video chat → Generated answer: returns response
- AV-TAU annotations and clips → AV-TAU data: loads annotations
- InternVid annotations and clips → InternVid data: loads annotations
- AV-TAU data → Dataset base: extends dataset base (evidence: `video_llama/datasets/datasets/my_dataset_with_audio.py`)
- Video instruction data → Dataset base: extends dataset base (evidence: `video_llama/datasets/datasets/video_instruct_dataset.py`)
- AV-TAU data → Video and audio processing: uses processor (evidence: `video_llama/datasets/datasets/my_dataset_with_audio.py`)
- Video instruction data → Video and audio processing: uses processor (evidence: `video_llama/datasets/datasets/video_instruct_dataset.py`)
- InternVid data → Dataset builders: registered for building
- Dataset builders → Dataset base: builds datasets (evidence: `video_llama/datasets/builders/base_dataset_builder.py`)
- Video and audio processing → Video transforms: applies transforms (evidence: `video_llama/processors/video_processor.py`)
- Audio-visual input data → ImageBind encoder: prepares encoder input (evidence: `video_llama/models/ImageBind/data.py`)
- Training entry → Config loading: loads config (evidence: `train.py`)
- Training entry → Training task: sets up task (evidence: `train.py`)
- Training entry → Training runner: starts runner (evidence: `train.py`)
- Training task → EchoTraffic model: builds model (evidence: `video_llama/tasks/base_task.py`)
- Training task → Dataset builders: builds datasets (evidence: `video_llama/tasks/base_task.py`)
- Training runner → Training task: runs task steps (evidence: `video_llama/runners/runner_base.py`)
- Video-text pretraining → Training task: specializes task (evidence: `video_llama/tasks/video_text_pretrain.py`)
- Model checkpoint → Inference entry: loads weights
- Component registry → Dataset builders: resolves builders (evidence: `video_llama/common/registry.py`)
- Component registry → EchoTraffic model: resolves models (evidence: `video_llama/common/registry.py`)