# nvlabs/eg3d architecture

> Architecture diagram of the GitHub repository nvlabs/eg3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/eg3d
- Repository: https://github.com/nvlabs/eg3d

## Overview

##### Project Type and Purpose
- This is a research‐oriented ML framework for Efficient Geometry‐aware 3D Generative Adversarial Networks (EG3D).
- It provides end-to-end scripts and libraries for:
- Dataset preprocessing (FFHQ, AFHQ, ShapeNet Cars)
- Model training (3D-aware GAN with hybrid explicit-implicit architecture)
- Evaluation (FID, IS, KID, etc.)
- Interactive visualization

##### High-Level Components
A. Data Preprocessing
- CLI scripts in `dataset_preprocessing/` (subfolders: `ffhq`, `afhq`, `shapenet_cars`)
- Reads raw archives or image folders → aligns/crops faces or samples camera views → writes standardized ZIP datasets + `dataset.json`
B. Training Pipeline
- Entry point: `eg3d/train.py`
- Core modules in `eg3d/training/` (augment, dataset loader, networks_stylegan2/stylegan3, loss, training_loop)
- Custom CUDA/PyTorch ops under `eg3d/torch_utils/ops` for bias_act, upfirdn2d, filtered_lrelu
- Outputs: model snapshots (`*.pkl`), training statistics (`training_stats.jsonl`), image grids, TensorBoard logs
C. Metrics & Evaluation
- Scripts: `eg3d/calc_metrics.py` and `eg3d/eg3d/metrics/` for FID, IS, KID, PPL, precision-recall, etc.
- Consumes model pickles + dataset → writes JSONL metric logs
D. Generation
- Scripts: `eg3d/gen_samples.py`, `eg3d/gen_videos.py`
E. Interactive Visualization
- `eg3d/visualizer.py`
- GUI widgets under `eg3d/gui_utils/` and `eg3d/viz/` (ImGui via GLFW/OpenGL)
- Allows on-the-fly parameter tweaking (pose, truncation, layer selection)

##### Interactions and Data Flow
- Raw Data → [Preprocessing] → Standardized Dataset ZIPs
- Dataset ZIPs → [Training] → Model Pickles + Logs
- Model Pickles →
- [Metrics] → Evaluation Reports
- [Interactive Visualizer] → Live Exploration

##### Architectural Patterns & Design Principles
- Separation of concern:
- Data pipelines (dataset_tool, preprocessing scripts)
- Model definition (networks_stylegan2/stylegan3)
- Rendering (neural renderer decoupled from feature generation)
- Training loop orchestrator (dnnlib + custom training_loop)
- Custom C++/CUDA extensions for performance-critical ops
- Plugin-style GUI widgets to facilitate extension

##### Key Technologies & Frameworks
- Python 3.8+, PyTorch ≥1.11 with custom CUDA extensions
- dnnlib (NVIDIA’s lightweight Deep Learning library)
- Conda environment (`eg3d/environment.yml`) for reproducibility
- ImGui + GLFW for interactive GUI
- External submodule Deep3DFaceRecon_pytorch for pose estimation

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    RawData["Raw Images/Archives"]:::data

    %% Preprocessing Stage
    subgraph "Data Preprocessing" 
        PreprocessCLI["dataset_preprocessing/ (ffhq, afhq, shapenet_cars)"]:::compute
        DatasetTool["eg3d/dataset_tool.py"]:::compute
        Deep3DSub["Deep3DFaceRecon_pytorch (Submodule)"]:::external
    end
    PreprocessCLI -->|align/crop, sample views| DatasetTool
    DatasetTool -->|"ZIP + dataset.json"| DatasetZIP

    %% Intermediate Storage
    DatasetZIP["Standardized Dataset ZIPs"]:::data

    %% Training Stage
    subgraph "Training Pipeline" 
        TrainEntry["eg3d/train.py"]:::compute
        subgraph "Core Modules"
            Augment["augment.py"]:::compute
            DatasetLoader["dataset.py"]:::compute
            Loss["loss.py"]:::compute
            Loop["training_loop.py"]:::compute
            Networks2["networks_stylegan2.py"]:::compute
            Networks3["networks_stylegan3.py"]:::compute
            Triplane["triplane.py"]:::compute
            DualDisc["dual_discriminator.py"]:::compute
            SR["superresolution.py"]:::compute
            VolRender["volumetric_rendering/"]:::compute
        end
        subgraph "Performance Layer (CUDA/PyTorch Ops)"
            Bias["bias_act.*"]:::compute
            Upfirdn["upfirdn2d.*"]:::compute
            FiltLRelu["filtered_lrelu.*"]:::compute
            ConvGrad["conv2d_gradfix.py"]:::compute
            ConvResample["conv2d_resample.py"]:::compute
        end
    end
    DatasetZIP -->|load dataset| TrainEntry
    TrainEntry --> Augment
    TrainEntry --> DatasetLoader
    TrainEntry --> Loss
    TrainEntry --> Loop
    TrainEntry --> Networks2
    TrainEntry --> Networks3
    TrainEntry --> Triplane
    TrainEntry --> DualDisc
    TrainEntry --> SR
    TrainEntry --> VolRender
    TrainEntry --> Bias
    TrainEntry --> Upfirdn
    TrainEntry --> FiltLRelu
    TrainEntry --> ConvGrad
    TrainEntry --> ConvResample
    Loop -->|writes| ModelPickle
    Loop -->|writes| TrainLogs

    %% Model Outputs
    ModelPickle["Model Snapshots (.pkl)"]:::data
    TrainLogs["Training Stats, Image Grids, TensorBoard Logs"]:::data

    %% Metrics & Evaluation
    subgraph "Metrics & Evaluation"
        CalcMetrics["eg3d/calc_metrics.py"]:::compute
        subgraph "Metric Modules"
            FID["frechet_inception_distance.py"]:::compute
            IS["inception_score.py"]:::compute
            KID["kernel_inception_distance.py"]:::compute
            PPL["perceptual_path_length.py"]:::compute
            PR["precision_recall.py"]:::compute
            EQ["equivariance.py"]:::compute
            MMain["metric_main.py"]:::compute
            MUtil["metric_utils.py"]:::compute
        end
    end
    ModelPickle -->|"use pickle + dataset"| CalcMetrics
    CalcMetrics -->|writes| MetricsLogs

    MetricsLogs["Metrics JSONL Reports"]:::data

    %% Generation
    subgraph "Generation"
        GenSamples["eg3d/gen_samples.py"]:::compute
        GenVideos["eg3d/gen_videos.py"]:::compute
    end
    ModelPickle -->|"load G_ema"| GenSamples
    ModelPickle -->|"load G_ema"| GenVideos
    GenSamples -->|outputs images, .mrc shapes| GenOutputs
    GenVideos -->|outputs videos| GenOutputs

    GenOutputs["Generated Images/Videos/Shapes"]:::data
    GenOutputs -->|view shapes| ChimeraX

    ChimeraX["ChimeraX Viewer"]:::external

    %% Interactive Visualization
    subgraph "Interactive Visualization"
        Visualizer["eg3d/visualizer.py"]:::ui
        GUIUtils["eg3d/gui_utils/"]:::ui
        VizWidgets["eg3d/viz/"]:::ui
    end
    ModelPickle -->|"load G_ema"| Visualizer
    Visualizer --> GUIUtils
    Visualizer --> VizWidgets

    %% Configuration
    Config["environment.yml + CLI Flags"]:::external
    Config --> PreprocessCLI
    Config --> TrainEntry
    Config --> CalcMetrics
    Config --> GenSamples
    Config --> Visualizer

    %% Click Events
    click PreprocessCLI "https://github.com/nvlabs/eg3d/tree/main/dataset_preprocessing/ffhq/"
    click PreprocessCLI "https://github.com/nvlabs/eg3d/tree/main/dataset_preprocessing/afhq/"
    click PreprocessCLI "https://github.com/nvlabs/eg3d/tree/main/dataset_preprocessing/shapenet_cars/"
    click PreprocessCLI "https://github.com/nvlabs/eg3d/blob/main/dataset_preprocessing/mirror_dataset.py"
    click PreprocessCLI "https://github.com/nvlabs/eg3d/tree/main/dataset_preprocessing/rebalance_ffhq/"
    click DatasetTool "https://github.com/nvlabs/eg3d/blob/main/eg3d/dataset_tool.py"
    click TrainEntry "https://github.com/nvlabs/eg3d/blob/main/eg3d/train.py"
    click Augment "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/augment.py"
    click DatasetLoader "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/dataset.py"
    click Loss "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/loss.py"
    click Loop "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/training_loop.py"
    click Networks2 "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/networks_stylegan2.py"
    click Networks3 "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/networks_stylegan3.py"
    click Triplane "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/triplane.py"
    click DualDisc "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/dual_discriminator.py"
    click SR "https://github.com/nvlabs/eg3d/blob/main/eg3d/training/superresolution.py"
    click VolRender "https://github.com/nvlabs/eg3d/tree/main/eg3d/training/volumetric_rendering/"
    click Bias "https://github.com/nvlabs/eg3d/blob/main/eg3d/torch_utils/ops/bias_act.*"
    click Upfirdn "https://github.com/nvlabs/eg3d/blob/main/eg3d/torch_utils/ops/upfirdn2d.*"
    click FiltLRelu "https://github.com/nvlabs/eg3d/blob/main/eg3d/torch_utils/ops/filtered_lrelu.*"
    click ConvGrad "https://github.com/nvlabs/eg3d/blob/main/eg3d/torch_utils/ops/conv2d_gradfix.py"
    click ConvResample "https://github.com/nvlabs/eg3d/blob/main/eg3d/torch_utils/ops/conv2d_resample.py"
    click CalcMetrics "https://github.com/nvlabs/eg3d/blob/main/eg3d/calc_metrics.py"
    click FID "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/frechet_inception_distance.py"
    click IS "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/inception_score.py"
    click KID "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/kernel_inception_distance.py"
    click PPL "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/perceptual_path_length.py"
    click PR "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/precision_recall.py"
    click EQ "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/equivariance.py"
    click MMain "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/metric_main.py"
    click MUtil "https://github.com/nvlabs/eg3d/blob/main/eg3d/metrics/metric_utils.py"
    click GenSamples "https://github.com/nvlabs/eg3d/blob/main/eg3d/gen_samples.py"
    click GenVideos "https://github.com/nvlabs/eg3d/blob/main/eg3d/gen_videos.py"
    click Visualizer "https://github.com/nvlabs/eg3d/blob/main/eg3d/visualizer.py"
    click GUIUtils "https://github.com/nvlabs/eg3d/tree/main/eg3d/gui_utils/"
    click VizWidgets "https://github.com/nvlabs/eg3d/tree/main/eg3d/viz/"
    click Config "https://github.com/nvlabs/eg3d/blob/main/eg3d/environment.yml"

    %% Styles
    classDef data fill:#bbdefb,stroke:#1976d2,color:#0d47a1
    classDef compute fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20
    classDef ui fill:#e1bee7,stroke:#6a1b9a,color:#4a148c
    classDef external fill:#eeeeee,stroke:#9e9e9e,color:#616161
```
