How aistairc/finebio works
FineBio is a dataset and benchmark codebase for hierarchical video understanding in biological experiments. The user-facing workflow is to obtain videos and annotations, extract visual features, then run step segmentation or atomic-operation detection; separate image-based benchmarks detect objects and hand/object interactions. The graph retains these distinct tasks and their evaluation branches. Source excerpts are partial, so links for unsampled benchmark implementations are limited to relationships supported by the README and file tree.
Main entry points: main.py DINO entry point · extract_feature.py I3D extractor · ASFormer/inference.py ASFormer · MS-TCN2/inference.py MS-TCN2
Components 24
Video features
- I3D extractor
feature_extraction/I3D/extract_feature.py - I3D model
feature_extraction/I3D/pytorch_i3d.py - RAFT flow
feature_extraction/RAFT/get_finebio_flow.py - I3D features
Step segmentation
- ASFormer
step_segmentation/ASFormer/inference.py - ASFormer model
step_segmentation/ASFormer/model.py - MS-TCN2
step_segmentation/MS-TCN2/inference.py - MS-TCN2 model
step_segmentation/MS-TCN2/model.py - Step labels
Atomic operations
- ActionFormer training
atomic_operation_detection/actionformer2/train.py - ActionFormer model
atomic_operation_detection/actionformer2/libs/modeling/meta_archs.py - Detection inference
atomic_operation_detection/actionformer2/libs/utils/inference.py - Multi-head evaluation
atomic_operation_detection/actionformer2/eval_multihead.py - Operation detections
Object detection
- DINO entry point
manipulated_affected_object_detection/DINO_ver/main.py - FineBio image dataset
manipulated_affected_object_detection/DINO_ver/datasets/finebio.py - DINO detector
manipulated_affected_object_detection/DINO_ver/models/dino/dino.py - Train and evaluate
manipulated_affected_object_detection/DINO_ver/engine.py - COCO object detection
object_detection/dino-4scale_r50_8xb2-12e_finebio.py - Hand-object detection
manipulated_affected_object_detection/DINO_ver/demo_finebio.py - Object detections
Other components
- Researcher
- FineBio videos
- FineBio annotations
Connections 29
- Researcher → FineBio videos: accesses
- Researcher → FineBio annotations: accesses
- FineBio videos → I3D extractor: extracts features
- I3D extractor → I3D model: uses network
- I3D extractor → I3D features: writes features
- FineBio videos → RAFT flow: estimates flow
- I3D features → ASFormer: loads features
- ASFormer → ASFormer model: uses model
- ASFormer → Step labels: writes labels
- I3D features → MS-TCN2: loads features
- MS-TCN2 → MS-TCN2 model: uses model
- MS-TCN2 → Step labels: writes labels
- FineBio annotations → ActionFormer training: supervises training
- ActionFormer training → ActionFormer model: trains
- I3D features → Detection inference: provides features
- ActionFormer model → Detection inference: predicts segments
- Detection inference → Operation detections: produces detections
- FineBio annotations → Multi-head evaluation: evaluation reference
- Operation detections → Multi-head evaluation: evaluates
- Researcher → DINO entry point: runs benchmark
- DINO entry point → FineBio image dataset: builds dataset
- DINO entry point → DINO detector: builds detector
- DINO entry point → Train and evaluate: runs training
- FineBio image dataset → Train and evaluate: provides batches
- DINO detector → Train and evaluate: trained and evaluated
- FineBio image dataset → Object detections: supplies images
- COCO object detection → Object detections: configures benchmark
- Researcher → Hand-object detection: runs demo
- Hand-object detection → Object detections: detects entities