# facebookresearch/animalavatar architecture

> Architecture diagram of the GitHub repository facebookresearch/animalavatar, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/animalavatar
- Repository: https://github.com/facebookresearch/animalavatar

## Overview

##### Project Type and Purpose
- Project Type: A modular research pipeline (not a traditional full‐stack app) for reconstructing animatable 3D animal avatars from casual videos.

##### File Structure and High‐Level Modules
- Top‐level orchestration scripts:
- main_preprocess_scene.py – Preprocessing
- main_optimize_scene.py – Scene optimization / reconstruction
- main_visualize_reconstruction.py – Rendering & visualization
- Core functional packages:
- config/ – Hydra‐based configuration
- data/ – Dataset loaders and utilities
- cse_embedding/ – DensePoseCSE embedding extraction & preprocessing
- optical_flow/ – (Optional) Optical‐flow–based refinement
- pnp/ – Perspective‐n‐Point initialization & visualization
- scene_optim/ – General scene‐optimizer framework + loss modules
- model/ – Model initialization, pose & texture sub‐models, inference, logging
- rendering/ – Mesh renderer, visualization helpers
- smal/ – SMAL model interface (initialization, utilities)
- losses/ – Core loss functions (ARAP, Chamfer, TV, etc.) used by scene_optim
- utils/ – Shared utilities (image, mesh, camera alignment)

##### README for Architectural Insights
- External dependencies: PyTorch, PyTorch3D, Lightplane, DensePoseCSE, vgGsfM/Segment‐Anything for missing cameras/masks.
- External data: CoP3D dataset; SMAL priors; DensePose weights; precomputed CSE embeddings and LBOs.

##### Main Components & Interactions
a. Inputs
- Video frames + masks + camera intrinsics/extrinsics (CoP3D or custom)
- External data: SMAL model priors, DensePose configs & weights, precomputed embeddings
b. Preprocessing Stage (main_preprocess_scene.py)
- cse_embedding module: extract per‐frame CSE maps
- pnp module: estimate root orientation & initial camera
- optical_flow (optional): refine CSE maps via flow
→ Outputs: CSE maps, root orientations, refined masks (if available)
c. Dataset Loader (data/*.py)
- CustomSingleVideo or Cop3dDataloader reads inputs, organizes frames/masks/cameras
d. Model Initialization & Inference (model/)
- Instantiate SMAL mesh & pose + texture models (model_initialization.py)
- Positional embeddings, trajectory sampling, texture model
- Inferencer orchestrates forward pass through pose & texture branches
e. Scene Optimization (main_optimize_scene.py → scene_optim/)
- scene_optimizer: iterative optimization loop
- callbacks: logging, early stopping, evaluation
- evaluators: PSNR, LPIPS, IoU
→ Outputs: optimized SMAL parameters, textures
f. Rendering & Visualization (main_visualize_reconstruction.py → rendering/)
- renderer: rasterizes mesh with textures & lighting
- visualization_renderer: overlays results on original frames
→ Outputs: video reconstructions, qualitative figures

##### Architectural Patterns & Principles
- Modular pipeline: Separate preprocessing / optimization / visualization phases.
- Config‐driven: Hydra centralizes all experiment parameters (config/config.yaml).
- Extensibility: Loss modules and callbacks follow plug‐in pattern under scene_optim/.
- Decoupling: Model definitions (pose vs. texture) live in separate sub‐packages, enabling independent swapping.

##### Final Checklist
- Annotation of key technologies/frameworks

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Data Repository
  subgraph "Data Repository"
    cop3d["CoP3D Dataset"]:::datastore
    smal_priors["SMAL Priors"]:::datastore
    densepose["DensePose Weights"]:::datastore
  end

  config["Config (Hydra)"]:::config

  %% Preprocessing Stage
  subgraph "Preprocessing Stage"
    preprocess_script["main_preprocess_scene.py"]:::compute
    cse["CSE Embedding"]:::compute
    pnp["PnP Initialization"]:::compute
    flow["Optical Flow (Optional)"]:::compute
  end

  %% Data Loader
  data_loader["Data Loader"]:::data

  %% Model Init & Inference
  subgraph "Model Init & Inference"
    model_init["Model Initialization"]:::model
    infer["Inferencer"]:::model
  end

  smal_model["SMAL Model Interface"]:::model

  %% Optimization Stage
  subgraph "Optimization Stage"
    optimize_script["main_optimize_scene.py"]:::compute
    scene_opt["Scene Optimizer"]:::compute
    subgraph "Loss Modules"
      losses["Loss Functions"]:::compute
    end
    subgraph "Callbacks"
      callbacks["Callbacks"]:::compute
    end
    subgraph "Evaluators"
      evaluators["Evaluators"]:::compute
    end
  end

  %% Visualization Stage
  subgraph "Visualization Stage"
    vis_script["main_visualize_reconstruction.py"]:::compute
    renderer["Renderer"]:::compute
    vis_render["Visualization Renderer"]:::compute
  end

  %% Data Flows
  cop3d -->|frames,masks,cameras| preprocess_script
  smal_priors -->|SMAL Priors| preprocess_script
  densepose -->|CSE configs & weights| preprocess_script
  config --> preprocess_script
  preprocess_script --> cse
  preprocess_script --> pnp
  preprocess_script --> flow
  cse --> data_loader
  pnp --> data_loader
  flow --> data_loader
  data_loader --> model_init
  data_loader --> smal_model
  config --> model_init
  config --> optimize_script
  model_init --> infer
  smal_model --> infer
  infer --> scene_opt
  optimize_script --> scene_opt
  scene_opt --> losses
  scene_opt --> callbacks
  scene_opt --> evaluators
  scene_opt --> renderer
  config --> vis_script
  vis_script --> renderer
  renderer --> vis_render

  %% Click Events
  click config "https://github.com/facebookresearch/animalavatar/blob/main/config/config.yaml"
  click config "https://github.com/facebookresearch/animalavatar/blob/main/config/keys.py"
  click preprocess_script "https://github.com/facebookresearch/animalavatar/blob/main/main_preprocess_scene.py"
  click cse "https://github.com/facebookresearch/animalavatar/blob/main/cse_embedding/cse_embedder.py"
  click cse "https://github.com/facebookresearch/animalavatar/blob/main/cse_embedding/preprocess_cse.py"
  click cse "https://github.com/facebookresearch/animalavatar/blob/main/cse_embedding/functional_map.py"
  click cse "https://github.com/facebookresearch/animalavatar/blob/main/cse_embedding/cse_keypoints.py"
  click cse "https://github.com/facebookresearch/animalavatar/blob/main/cse_embedding/visualize_cse.py"
  click pnp "https://github.com/facebookresearch/animalavatar/blob/main/pnp/preprocess_pnp.py"
  click pnp "https://github.com/facebookresearch/animalavatar/blob/main/pnp/camera_estimation.py"
  click pnp "https://github.com/facebookresearch/animalavatar/blob/main/pnp/p_n_p_optim.py"
  click pnp "https://github.com/facebookresearch/animalavatar/blob/main/pnp/visualize_pnp.py"
  click flow "https://github.com/facebookresearch/animalavatar/blob/main/optical_flow/cse_optical_flow_preprocess.py"
  click flow "https://github.com/facebookresearch/animalavatar/blob/main/optical_flow/optical_flow_model.py"
  click flow "https://github.com/facebookresearch/animalavatar/blob/main/optical_flow/flow_checker.py"
  click data_loader "https://github.com/facebookresearch/animalavatar/blob/main/data/cop3d_dataloader.py"
  click data_loader "https://github.com/facebookresearch/animalavatar/blob/main/data/custom_dataloader.py"
  click data_loader "https://github.com/facebookresearch/animalavatar/blob/main/data/input_cop.py"
  click data_loader "https://github.com/facebookresearch/animalavatar/blob/main/data/utils.py"
  click smal_model "https://github.com/facebookresearch/animalavatar/blob/main/smal/initialize_smal.py"
  click smal_model "https://github.com/facebookresearch/animalavatar/blob/main/smal/smal_torch.py"
  click smal_model "https://github.com/facebookresearch/animalavatar/blob/main/smal/smal_utils.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/model_initialization.py"
  click infer "https://github.com/facebookresearch/animalavatar/blob/main/model/inferencer.py"
  click infer "https://github.com/facebookresearch/animalavatar/blob/main/model/logger.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/positional_embedding.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/pose_models/model_canonical.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/pose_models/model_fixed.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/pose_models/model_mlp.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/pose_models/model_util.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/texture_models/model_cse.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/texture_models/model_duplex.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/texture_models/trajectory_sampler.py"
  click model_init "https://github.com/facebookresearch/animalavatar/blob/main/model/texture_models/model_utils.py"
  click optimize_script "https://github.com/facebookresearch/animalavatar/blob/main/main_optimize_scene.py"
  click scene_opt "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/scene_optimizer.py"
  click callbacks "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/callbacks/callback_early_stop.py"
  click callbacks "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/callbacks/callback_eval.py"
  click callbacks "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/callbacks/callback_general.py"
  click evaluators "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/evaluators/psnr_evaluator.py"
  click evaluators "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/evaluators/lpips_evaluator.py"
  click evaluators "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/evaluators/iou_evaluator.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/loss_optim.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/losses_optim/loss_optim_arap_fast_reg.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/losses_optim/loss_optim_arap_reg.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/losses_optim/loss_optim_chamfer.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/losses_optim/loss_optim_color_rend.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/losses_optim/loss_optim_kps.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/losses_optim/loss_optim_regularization.py"
  click losses "https://github.com/facebookresearch/animalavatar/blob/main/scene_optim/losses_optim/loss_optim_tv_reg.py"
  click vis_script "https://github.com/facebookresearch/animalavatar/blob/main/main_visualize_reconstruction.py"
  click renderer "https://github.com/facebookresearch/animalavatar/blob/main/rendering/renderer.py"
  click vis_render "https://github.com/facebookresearch/animalavatar/blob/main/rendering/visualization_renderer.py"

  %% Styles
  classDef compute fill:#ADD8E6,stroke:#000,stroke-width:1px
  classDef data fill:#90EE90,stroke:#000,stroke-width:1px
  classDef model fill:#FFA500,stroke:#000,stroke-width:1px
  classDef datastore fill:#d3d3d3,stroke:#000,stroke-width:1px
  classDef config fill:#FFD700,stroke:#000,stroke-width:1px
```
