How facebookresearch/animalavatar works
Project Type and Purpose
• Project Type: A modular research pipeline (not a traditional full‐stack app) for reconstructing animatable 3D animal avatars from casual videos.
File Structure and High‐Level Modules
• Top‐level orchestration scripts:
• main_preprocess_scene.py – Preprocessing
• main_optimize_scene.py – Scene optimization / reconstruction
• main_visualize_reconstruction.py – Rendering & visualization
• Core functional packages:
• config/ – Hydra‐based configuration
• data/ – Dataset loaders and utilities
• cse_embedding/ – DensePoseCSE embedding extraction & preprocessing
• optical_flow/ – (Optional) Optical‐flow–based refinement
• pnp/ – Perspective‐n‐Point initialization & visualization
• scene_optim/ – General scene‐optimizer framework + loss modules
• model/ – Model initialization, pose & texture sub‐models, inference, logging
• rendering/ – Mesh renderer, visualization helpers
• smal/ – SMAL model interface (initialization, utilities)
• losses/ – Core loss functions (ARAP, Chamfer, TV, etc.) used by scene_optim
• utils/ – Shared utilities (image, mesh, camera alignment)
README for Architectural Insights
• External dependencies: PyTorch, PyTorch3D, Lightplane, DensePoseCSE, vgGsfM/Segment‐Anything for missing cameras/masks.
• External data: CoP3D dataset; SMAL priors; DensePose weights; precomputed CSE embeddings and LBOs.
Main Components & Interactions
a. Inputs
• Video frames + masks + camera intrinsics/extrinsics (CoP3D or custom)
• External data: SMAL model priors, DensePose configs & weights, precomputed embeddings
b. Preprocessing Stage (main_preprocess_scene.py)
• cse_embedding module: extract per‐frame CSE maps
• pnp module: estimate root orientation & initial camera
• optical_flow (optional): refine CSE maps via flow
→ Outputs: CSE maps, root orientations, refined masks (if available)
c. Dataset Loader (data/*.py)
• CustomSingleVideo or Cop3dDataloader reads inputs, organizes frames/masks/cameras
d. Model Initialization & Inference (model/)
• Instantiate SMAL mesh & pose + texture models (model_initialization.py)
• Positional embeddings, trajectory sampling, texture model
• Inferencer orchestrates forward pass through pose & texture branches
e. Scene Optimization (main_optimize_scene.py → scene_optim/)
• scene_optimizer: iterative optimization loop
• callbacks: logging, early stopping, evaluation
• evaluators: PSNR, LPIPS, IoU
→ Outputs: optimized SMAL parameters, textures
f. Rendering & Visualization (main_visualize_reconstruction.py → rendering/)
• renderer: rasterizes mesh with textures & lighting
• visualization_renderer: overlays results on original frames
→ Outputs: video reconstructions, qualitative figures
Architectural Patterns & Principles
• Modular pipeline: Separate preprocessing / optimization / visualization phases.
• Config‐driven: Hydra centralizes all experiment parameters (config/config.yaml).
• Extensibility: Loss modules and callbacks follow plug‐in pattern under scene_optim/.
• Decoupling: Model definitions (pose vs. texture) live in separate sub‐packages, enabling independent swapping.
Final Checklist
• Annotation of key technologies/frameworks