How briannlongzhao/animal-in-motion works
Project Type and Purpose
• This is an open-source data-processing pipeline for “4D animal reconstruction” from in-the-wild video.
• It provides end-to-end stages: video ingestion (download or local), preprocessing (shot split, filtering), object tracking (segmentation, flow, pose), dataset building, feature extraction (DINO+PCA, denoising), and optional 4D reconstruction via 3D-Fauna.
File Structure
• Top-level folders:
• categories/ – animal category IDs
• configs/ – YAML config for all scripts
• database/ – (empty stub for future DB?)
• externals/ – Git submodules for heavy ML models (SAM2, SEA-RAFT, ViTPose, Denoising_ViT)
• models/ – wrappers and orchestrators for each model family: animal_models, depth_models, feature_models, flow_models, pose_models, trackers
• scripts/ – entrypoint CLI scripts that invoke each pipeline stage
• tools/ – miscellaneous utilities for data management (resize, rename, preview, copy, restore RGB, etc.)
• environment.yml and python path usage imply a single Python monolithic codebase orchestrating all steps.
Readme Insights
• Describes data sources (YouTube or user videos), multiple scripts in a prescribed order: download_video → prepare_video → preprocess_video → track_animal → build_dataset → extract_feature → reconstruct_video_fauna
• Relies heavily on external ML checkpoints and submodule code.
• Configuration-driven: configs/default.yml controls categories, paths, thresholds.
A. Components (Nodes)
1. Ingestion
• YouTube Downloader (scripts/download_video.py)
• Local-Video Loader (scripts/prepare_video.py)
– Output: raw video files organized by category
2. Preprocessing
• Shot-Change Detector & Clip Splitter (scripts/preprocess_video.py)
– Output: short video clips
3. Object Tracking & Segmentation
• GroundingDINO + Grounded-SAM2 (models/trackers/*)
• Optical Flow (SEA-RAFT)
• Pose Estimation (ViTPose)
– Output: object-centric tracks, masks, flow, pose metadata
4. Dataset Builder
• Filtering & Split (scripts/build_dataset.py)
– Output: train/test dataset directories with per-frame metadata
5. Feature Extraction
• DINO Feature Extractor + PCA (scripts/extract_feature.py)
• Optional Denoising (models/feature_models/denoiser.py)
– Output: features appended to dataset
6. 4D Reconstruction (Optional Stage)
• 3D-Fauna pipeline (scripts/reconstruct_video_fauna.py + externals/Animals)
– Output: 3D meshes / tet meshes for each video sequence
7. Utilities
• Data utilities (tools/*.py): metadata, preview, restore RGB, convert to video, etc.
B. Data Stores
• Raw video storage (on disk, per category)
• Intermediate clip storage
• Track & mask storage (JSON + PNGs)
• Final dataset directory (structured by category / train-test)
• Feature storage (binary PCA, feature arrays)
• 4D reconstruction outputs (npz, glb)
C. Config & Control
• configs/default.yml: central config
• categories/__init__.py: category list
• CLI scripts: read config, invoke modules
Architectural Patterns & Principles
• Pipeline pattern: sequential stages, each with clear input/output
• Modular design: each model family in its own subpackage
• Plugin/adapter pattern: externals are treated as interchangeable modules
• Configuration-first: no hardcoded paths or hyperparameters
Final Advice
• Call out the externals folder as a special integration layer.
• Both control flow (script invocation) and data flow (file formats, directories).
• Be detailed: annotate config file relationships (e.g., “all scripts read configs/default.yml”).
• This level of granularity will give the team a clear, maintainable overview and make onboarding or extension easier.