How harryhsing/echotraffic works
EchoTraffic is an audio-visual traffic anomaly understanding model built on Video-LLaMA. Its documented user path is to supply a video and question, encode video/audio context, and generate a response; its training path prepares annotated datasets and optimizes the model through pretraining and supervised fine-tuning. The README also identifies AV-TAU and InternVid as distinct data capabilities. Several domain modules are indexed but not excerpted, so only documented or source-index-supported links are shown; unspecified connections are omitted.
Main entry points: inference.py Inference entry · video_llama.py EchoTraffic model · base_task.py Training task · video_processor.py Video and audio processing
Components 27
Question answering
- Inference entry
inference.py - Video chat
video_llama/conversation/conversation_video.py
Dataset preparation
- AV-TAU data
video_llama/datasets/datasets/my_dataset_with_audio.py - Video instruction data
video_llama/datasets/datasets/video_instruct_dataset.py - InternVid data
video_llama/datasets/datasets/internvid_datasets.py - Dataset builders
video_llama/datasets/builders - Dataset base
video_llama/datasets/datasets/base_dataset.py - Dataset utilities
video_llama/datasets/data_utils.py
Multimodal model
- Video and audio processing
video_llama/processors/video_processor.py - Video transforms
video_llama/processors/transforms_video.py - Audio-visual input data
video_llama/models/ImageBind/data.py - ImageBind encoder
video_llama/models/ImageBind/models/imagebind_model.py - EchoTraffic model
video_llama/models/video_llama.py - Q-Former
video_llama/models/Qformer.py - Language model
video_llama/models/modeling_llama.py
Training runtime
- Config loading
video_llama/common/config.py - Component registry
video_llama/common/registry.py - Training entry
train.py - Training task
video_llama/tasks/base_task.py - Training runner
video_llama/runners/runner_base.py - Video-text pretraining
video_llama/tasks/video_text_pretrain.py
Other components
- Analyst
- Video input
- Generated answer
- AV-TAU annotations and clips
- InternVid annotations and clips
- Model checkpoint
Connections 32
- Analyst → Inference entry: asks about video
- Video input → Inference entry: provides video
- Inference entry → Config loading: loads config
inference.py - Inference entry → Video and audio processing: builds processor
inference.py - Inference entry → Video chat: creates chat
inference.py - Inference entry → EchoTraffic model: loads model
- Video chat → Video and audio processing: processes video
video_llama/conversation/conversation_video.py - Video chat → Audio-visual input data: uses input data
video_llama/conversation/conversation_video.py - EchoTraffic model → Q-Former: uses Q-Former
- EchoTraffic model → ImageBind encoder: uses encoder
- EchoTraffic model → Language model: uses language model
- Video chat → Generated answer: returns response
- AV-TAU annotations and clips → AV-TAU data: loads annotations
- InternVid annotations and clips → InternVid data: loads annotations
- AV-TAU data → Dataset base: extends dataset base
video_llama/datasets/datasets/my_dataset_with_audio.py - Video instruction data → Dataset base: extends dataset base
video_llama/datasets/datasets/video_instruct_dataset.py - AV-TAU data → Video and audio processing: uses processor
video_llama/datasets/datasets/my_dataset_with_audio.py - Video instruction data → Video and audio processing: uses processor
video_llama/datasets/datasets/video_instruct_dataset.py - InternVid data → Dataset builders: registered for building
- Dataset builders → Dataset base: builds datasets
video_llama/datasets/builders/base_dataset_builder.py - Video and audio processing → Video transforms: applies transforms
video_llama/processors/video_processor.py - Audio-visual input data → ImageBind encoder: prepares encoder input
video_llama/models/ImageBind/data.py - Training entry → Config loading: loads config
train.py - Training entry → Training task: sets up task
train.py - Training entry → Training runner: starts runner
train.py - Training task → EchoTraffic model: builds model
video_llama/tasks/base_task.py - Training task → Dataset builders: builds datasets
video_llama/tasks/base_task.py - Training runner → Training task: runs task steps
video_llama/runners/runner_base.py - Video-text pretraining → Training task: specializes task
video_llama/tasks/video_text_pretrain.py - Model checkpoint → Inference entry: loads weights
- Component registry → Dataset builders: resolves builders
video_llama/common/registry.py - Component registry → EchoTraffic model: resolves models
video_llama/common/registry.py