# briannlongzhao/animal-in-motion architecture

> Architecture diagram of the GitHub repository briannlongzhao/animal-in-motion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/briannlongzhao/animal-in-motion
- Repository: https://github.com/briannlongzhao/animal-in-motion

## Overview

##### Project Type and Purpose
- This is an open-source data-processing pipeline for “4D animal reconstruction” from in-the-wild video.
- It provides end-to-end stages: video ingestion (download or local), preprocessing (shot split, filtering), object tracking (segmentation, flow, pose), dataset building, feature extraction (DINO+PCA, denoising), and optional 4D reconstruction via 3D-Fauna.

##### File Structure
- Top-level folders:
- categories/ – animal category IDs
- configs/ – YAML config for all scripts
- database/ – (empty stub for future DB?)
- externals/ – Git submodules for heavy ML models (SAM2, SEA-RAFT, ViTPose, Denoising_ViT)
- models/ – wrappers and orchestrators for each model family: animal_models, depth_models, feature_models, flow_models, pose_models, trackers
- scripts/ – entrypoint CLI scripts that invoke each pipeline stage
- tools/ – miscellaneous utilities for data management (resize, rename, preview, copy, restore RGB, etc.)
- environment.yml and python path usage imply a single Python monolithic codebase orchestrating all steps.

##### Readme Insights
- Describes data sources (YouTube or user videos), multiple scripts in a prescribed order: download_video → prepare_video → preprocess_video → track_animal → build_dataset → extract_feature → reconstruct_video_fauna
- Relies heavily on external ML checkpoints and submodule code.
- Configuration-driven: configs/default.yml controls categories, paths, thresholds.

A. Components (Nodes)
1. Ingestion
- YouTube Downloader (scripts/download_video.py)
- Local-Video Loader (scripts/prepare_video.py)
– Output: raw video files organized by category
2. Preprocessing
- Shot-Change Detector & Clip Splitter (scripts/preprocess_video.py)
– Output: short video clips
3. Object Tracking & Segmentation
- GroundingDINO + Grounded-SAM2 (models/trackers/*)
- Optical Flow (SEA-RAFT)
- Pose Estimation (ViTPose)
– Output: object-centric tracks, masks, flow, pose metadata
4. Dataset Builder
- Filtering & Split (scripts/build_dataset.py)
– Output: train/test dataset directories with per-frame metadata
5. Feature Extraction
- DINO Feature Extractor + PCA (scripts/extract_feature.py)
- Optional Denoising (models/feature_models/denoiser.py)
– Output: features appended to dataset
6. 4D Reconstruction (Optional Stage)
- 3D-Fauna pipeline (scripts/reconstruct_video_fauna.py + externals/Animals)
– Output: 3D meshes / tet meshes for each video sequence
7. Utilities
- Data utilities (tools/*.py): metadata, preview, restore RGB, convert to video, etc.
B. Data Stores
- Raw video storage (on disk, per category)
- Intermediate clip storage
- Track & mask storage (JSON + PNGs)
- Final dataset directory (structured by category / train-test)
- Feature storage (binary PCA, feature arrays)
- 4D reconstruction outputs (npz, glb)

C. Config & Control
- configs/default.yml: central config
- categories/__init__.py: category list
- CLI scripts: read config, invoke modules

##### Architectural Patterns & Principles
- Pipeline pattern: sequential stages, each with clear input/output
- Modular design: each model family in its own subpackage
- Plugin/adapter pattern: externals are treated as interchangeable modules
- Configuration-first: no hardcoded paths or hyperparameters

##### Final Advice
- Call out the externals folder as a special integration layer.
- Both control flow (script invocation) and data flow (file formats, directories).
- Be detailed: annotate config file relationships (e.g., “all scripts read configs/default.yml”).
- This level of granularity will give the team a clear, maintainable overview and make onboarding or extension easier.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    YouTube["YouTube Source"]:::external
    Local["Local Videos"]:::external

    %% Ingestion
    Download["YouTube Downloader"]:::module
    Prepare["Local-Video Loader"]:::module

    %% Data Store: Raw Videos
    RawVid(["Raw Video Storage\n(.mp4)"]):::data

    %% Preprocessing
    Preprocess["Shot-Change Detector & Clip Splitter"]:::module

    %% Data Store: Clips
    Clips(["Video Clips\n(.mp4)"]):::data

    %% Object Tracking & Segmentation
    subgraph "Object Tracking & Segmentation"
        GroundedSAM["Grounded SAM2 Tracker"]:::module
        NaiveTracker["Naive Tracker"]:::module
        FlowModel["SEA-RAFT Optical Flow"]:::module
        PoseModel["ViTPose Estimator"]:::module
    end

    %% Data Store: Tracks & Masks
    Tracks(["Tracks & Masks\n(.json, .png)"]):::data

    %% Dataset Building
    Builder["Dataset Builder"]:::module

    %% Data Store: Dataset
    Dataset(["Dataset\n(train/test)"]):::data

    %% Feature Extraction
    Extractor["DINO + PCA Feature Extractor"]:::module
    Denoiser["Optional Denoiser"]:::module

    %% Data Store: Features
    Features(["Features\n(.npy, PCA)"]):::data

    %% 4D Reconstruction
    Recon["3D-Fauna Reconstruction"]:::module

    %% Data Store: Reconstruction Outputs
    ReconOut(["4D Outputs\n(.npz, .glb)"]):::data

    %% Configuration and Utilities
    Config["Configuration"]:::module
    Tools["Data Utilities"]:::module

    %% Flow Connections
    YouTube -->|download_video.py| Download
    Local -->|prepare_video.py| Prepare
    Download --> RawVid
    Prepare --> RawVid
    RawVid -->|preprocess_video.py| Preprocess
    Preprocess --> Clips

    Clips --> GroundedSAM
    Clips --> NaiveTracker
    Clips --> FlowModel
    Clips --> PoseModel

    GroundedSAM --> Tracks
    NaiveTracker --> Tracks
    FlowModel --> Tracks
    PoseModel --> Tracks

    Tracks --> Builder
    Builder --> Dataset

    Dataset --> Extractor
    Extractor --> Features
    Features --> Recon
    Recon --> ReconOut

    %% Config usage
    Config -.-> Download
    Config -.-> Prepare
    Config -.-> Preprocess
    Config -.-> Builder
    Config -.-> Extractor
    Config -.-> Recon

    %% Utilities usage
    Tools -.-> RawVid
    Tools -.-> Clips
    Tools -.-> Tracks
    Tools -.-> Dataset
    Tools -.-> Features

    %% Click Events
    click Download "https://github.com/briannlongzhao/animal-in-motion/blob/main/scripts/download_video.py"
    click Download "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/youtube_downloader.py"
    click Prepare "https://github.com/briannlongzhao/animal-in-motion/blob/main/scripts/prepare_video.py"
    click Preprocess "https://github.com/briannlongzhao/animal-in-motion/blob/main/scripts/preprocess_video.py"
    click Preprocess "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/video_preprocessor.py"
    click GroundedSAM "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/trackers/grounded_sam2_tracker.py"
    click GroundedSAM "https://github.com/briannlongzhao/animal-in-motion/tree/main/externals/Grounded_SAM2"
    click NaiveTracker "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/trackers/naive_tracker.py"
    click FlowModel "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/flow_models/sea_raft.py"
    click FlowModel "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/flow_processor.py"
    click FlowModel "https://github.com/briannlongzhao/animal-in-motion/tree/main/externals/SEA_RAFT"
    click PoseModel "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/pose_models/vitpose.py"
    click PoseModel "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/pose_processor.py"
    click PoseModel "https://github.com/briannlongzhao/animal-in-motion/tree/main/externals/ViTPose"
    click Builder "https://github.com/briannlongzhao/animal-in-motion/blob/main/scripts/build_dataset.py"
    click Builder "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/dataset_builder.py"
    click Extractor "https://github.com/briannlongzhao/animal-in-motion/blob/main/scripts/extract_feature.py"
    click Extractor "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/feature_extractor.py"
    click Extractor "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/feature_models/dino_extractor.py"
    click Denoiser "https://github.com/briannlongzhao/animal-in-motion/blob/main/models/feature_models/denoiser.py"
    click Denoiser "https://github.com/briannlongzhao/animal-in-motion/tree/main/externals/Denoising_ViT"
    click Recon "https://github.com/briannlongzhao/animal-in-motion/blob/main/scripts/reconstruct_video_fauna.py"
    click Recon "https://github.com/briannlongzhao/animal-in-motion/tree/main/externals/Animals"
    click Config "https://github.com/briannlongzhao/animal-in-motion/blob/main/configs/default.yml"
    click Config "https://github.com/briannlongzhao/animal-in-motion/blob/main/categories/__init__.py"
    click Tools "https://github.com/briannlongzhao/animal-in-motion/tree/main/tools/"

    %% Styles
    classDef module fill:#D6E4FF,stroke:#1f77b4
    classDef data fill:#DCFFDC,stroke:#23a100
    classDef external fill:#FFE4B5,stroke:#f08c00
```
