# aistairc/finebio architecture

> Architecture diagram of the GitHub repository aistairc/finebio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aistairc/finebio
- Repository: https://github.com/aistairc/finebio

## Overview

FineBio is a dataset and benchmark codebase for hierarchical video understanding in biological experiments. The user-facing workflow is to obtain videos and annotations, extract visual features, then run step segmentation or atomic-operation detection; separate image-based benchmarks detect objects and hand/object interactions. The graph retains these distinct tasks and their evaluation branches. Source excerpts are partial, so links for unsampled benchmark implementations are limited to relationships supported by the README and file tree.

Main entry points: [`manipulated_affected_object_detection/DINO_ver/main.py`](https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/main.py) (DINO entry point), [`feature_extraction/I3D/extract_feature.py`](https://github.com/aistairc/finebio/blob/main/feature_extraction/I3D/extract_feature.py) (I3D extractor), [`step_segmentation/ASFormer/inference.py`](https://github.com/aistairc/finebio/blob/main/step_segmentation/ASFormer/inference.py) (ASFormer), [`step_segmentation/MS-TCN2/inference.py`](https://github.com/aistairc/finebio/blob/main/step_segmentation/MS-TCN2/inference.py) (MS-TCN2)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_features["Video features"]
  node_i3d["I3D extractor<br/>[extract_feature.py]"]
  node_i3dmodel["I3D model<br/>[pytorch_i3d.py]"]
  node_raft["RAFT flow"]
  node_featuresdata["I3D features"]
end

subgraph group_steps["Step segmentation"]
  node_asformer["ASFormer<br/>[inference.py]"]
  node_asmodel["ASFormer model<br/>[model.py]"]
  node_mstcn["MS-TCN2<br/>[inference.py]"]
  node_msmodel["MS-TCN2 model<br/>[model.py]"]
  node_stepresults["Step labels"]
end

subgraph group_operations["Atomic operations"]
  node_actiontrain["ActionFormer training<br/>[train.py]"]
  node_actionmodel["ActionFormer model<br/>[meta_archs.py]"]
  node_actioninfer["Detection inference<br/>[inference.py]"]
  node_actioneval["Multi-head evaluation<br/>[eval_multihead.py]"]
  node_operationresults["Operation detections"]
end

subgraph group_objects["Object detection"]
  node_dinoentry["DINO entry point<br/>[main.py]"]
  node_dinodata["FineBio image dataset<br/>[finebio.py]"]
  node_dinomodel["DINO detector<br/>[dino.py]"]
  node_dinoengine["Train and evaluate<br/>[engine.py]"]
  node_coco["COCO object detection"]
  node_interactions["Hand-object detection<br/>[demo_finebio.py]"]
  node_objectresults["Object detections"]
end

node_researcher(("Researcher"))
node_videos["FineBio videos"]
node_annotations["FineBio annotations"]

node_researcher -->|"accesses"| node_videos
node_researcher -->|"accesses"| node_annotations
node_videos -->|"extracts features"| node_i3d
node_i3d -->|"uses network"| node_i3dmodel
node_i3d -->|"writes features"| node_featuresdata
node_videos -->|"estimates flow"| node_raft
node_featuresdata -->|"loads features"| node_asformer
node_asformer -->|"uses model"| node_asmodel
node_asformer -->|"writes labels"| node_stepresults
node_featuresdata -->|"loads features"| node_mstcn
node_mstcn -->|"uses model"| node_msmodel
node_mstcn -->|"writes labels"| node_stepresults
node_annotations -->|"supervises training"| node_actiontrain
node_actiontrain -->|"trains"| node_actionmodel
node_featuresdata -->|"provides features"| node_actioninfer
node_actionmodel -->|"predicts segments"| node_actioninfer
node_actioninfer -->|"produces detections"| node_operationresults
node_annotations -->|"evaluation reference"| node_actioneval
node_operationresults -->|"evaluates"| node_actioneval
node_researcher -->|"runs benchmark"| node_dinoentry
node_dinoentry -->|"builds dataset"| node_dinodata
node_dinoentry -->|"builds detector"| node_dinomodel
node_dinoentry -->|"runs training"| node_dinoengine
node_dinodata -->|"provides batches"| node_dinoengine
node_dinomodel -->|"trained and evaluated"| node_dinoengine
node_dinodata -->|"supplies images"| node_objectresults
node_coco -.->|"configures benchmark"| node_objectresults
node_researcher -->|"runs demo"| node_interactions
node_interactions -->|"detects entities"| node_objectresults

click node_i3d "https://github.com/aistairc/finebio/blob/main/feature_extraction/I3D/extract_feature.py"
click node_i3dmodel "https://github.com/aistairc/finebio/blob/main/feature_extraction/I3D/pytorch_i3d.py"
click node_raft "https://github.com/aistairc/finebio/blob/main/feature_extraction/RAFT/get_finebio_flow.py"
click node_asformer "https://github.com/aistairc/finebio/blob/main/step_segmentation/ASFormer/inference.py"
click node_asmodel "https://github.com/aistairc/finebio/blob/main/step_segmentation/ASFormer/model.py"
click node_mstcn "https://github.com/aistairc/finebio/blob/main/step_segmentation/MS-TCN2/inference.py"
click node_msmodel "https://github.com/aistairc/finebio/blob/main/step_segmentation/MS-TCN2/model.py"
click node_actiontrain "https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/train.py"
click node_actionmodel "https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/libs/modeling/meta_archs.py"
click node_actioninfer "https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/libs/utils/inference.py"
click node_actioneval "https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/eval_multihead.py"
click node_dinoentry "https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/main.py"
click node_dinodata "https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/datasets/finebio.py"
click node_dinomodel "https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/models/dino/dino.py"
click node_dinoengine "https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/engine.py"
click node_coco "https://github.com/aistairc/finebio/blob/main/object_detection/dino-4scale_r50_8xb2-12e_finebio.py"
click node_interactions "https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/demo_finebio.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_i3d,node_i3dmodel,node_raft,node_featuresdata toneBlue
class node_asformer,node_asmodel,node_mstcn,node_msmodel,node_stepresults toneAmber
class node_actiontrain,node_actionmodel,node_actioninfer,node_actioneval,node_operationresults toneMint
class node_dinoentry,node_dinodata,node_dinomodel,node_dinoengine,node_coco,node_interactions,node_objectresults toneRose
class node_researcher,node_videos,node_annotations toneIndigo
```

## Components

### Video features

- [I3D extractor](https://github.com/aistairc/finebio/blob/main/feature_extraction/I3D/extract_feature.py): `feature_extraction/I3D/extract_feature.py`
- [I3D model](https://github.com/aistairc/finebio/blob/main/feature_extraction/I3D/pytorch_i3d.py): `feature_extraction/I3D/pytorch_i3d.py`
- [RAFT flow](https://github.com/aistairc/finebio/blob/main/feature_extraction/RAFT/get_finebio_flow.py): `feature_extraction/RAFT/get_finebio_flow.py`
- **I3D features**

### Step segmentation

- [ASFormer](https://github.com/aistairc/finebio/blob/main/step_segmentation/ASFormer/inference.py): `step_segmentation/ASFormer/inference.py`
- [ASFormer model](https://github.com/aistairc/finebio/blob/main/step_segmentation/ASFormer/model.py): `step_segmentation/ASFormer/model.py`
- [MS-TCN2](https://github.com/aistairc/finebio/blob/main/step_segmentation/MS-TCN2/inference.py): `step_segmentation/MS-TCN2/inference.py`
- [MS-TCN2 model](https://github.com/aistairc/finebio/blob/main/step_segmentation/MS-TCN2/model.py): `step_segmentation/MS-TCN2/model.py`
- **Step labels**

### Atomic operations

- [ActionFormer training](https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/train.py): `atomic_operation_detection/actionformer2/train.py`
- [ActionFormer model](https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/libs/modeling/meta_archs.py): `atomic_operation_detection/actionformer2/libs/modeling/meta_archs.py`
- [Detection inference](https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/libs/utils/inference.py): `atomic_operation_detection/actionformer2/libs/utils/inference.py`
- [Multi-head evaluation](https://github.com/aistairc/finebio/blob/main/atomic_operation_detection/actionformer2/eval_multihead.py): `atomic_operation_detection/actionformer2/eval_multihead.py`
- **Operation detections**

### Object detection

- [DINO entry point](https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/main.py): `manipulated_affected_object_detection/DINO_ver/main.py`
- [FineBio image dataset](https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/datasets/finebio.py): `manipulated_affected_object_detection/DINO_ver/datasets/finebio.py`
- [DINO detector](https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/models/dino/dino.py): `manipulated_affected_object_detection/DINO_ver/models/dino/dino.py`
- [Train and evaluate](https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/engine.py): `manipulated_affected_object_detection/DINO_ver/engine.py`
- [COCO object detection](https://github.com/aistairc/finebio/blob/main/object_detection/dino-4scale_r50_8xb2-12e_finebio.py): `object_detection/dino-4scale_r50_8xb2-12e_finebio.py`
- [Hand-object detection](https://github.com/aistairc/finebio/blob/main/manipulated_affected_object_detection/DINO_ver/demo_finebio.py): `manipulated_affected_object_detection/DINO_ver/demo_finebio.py`
- **Object detections**

### Other components

- **Researcher**
- **FineBio videos**
- **FineBio annotations**

## Connections

- Researcher → FineBio videos: accesses
- Researcher → FineBio annotations: accesses
- FineBio videos → I3D extractor: extracts features
- I3D extractor → I3D model: uses network
- I3D extractor → I3D features: writes features
- FineBio videos → RAFT flow: estimates flow
- I3D features → ASFormer: loads features
- ASFormer → ASFormer model: uses model
- ASFormer → Step labels: writes labels
- I3D features → MS-TCN2: loads features
- MS-TCN2 → MS-TCN2 model: uses model
- MS-TCN2 → Step labels: writes labels
- FineBio annotations → ActionFormer training: supervises training
- ActionFormer training → ActionFormer model: trains
- I3D features → Detection inference: provides features
- ActionFormer model → Detection inference: predicts segments
- Detection inference → Operation detections: produces detections
- FineBio annotations → Multi-head evaluation: evaluation reference
- Operation detections → Multi-head evaluation: evaluates
- Researcher → DINO entry point: runs benchmark
- DINO entry point → FineBio image dataset: builds dataset
- DINO entry point → DINO detector: builds detector
- DINO entry point → Train and evaluate: runs training
- FineBio image dataset → Train and evaluate: provides batches
- DINO detector → Train and evaluate: trained and evaluated
- FineBio image dataset → Object detections: supplies images
- COCO object detection → Object detections: configures benchmark
- Researcher → Hand-object detection: runs demo
- Hand-object detection → Object detections: detects entities