# cremebrule/digital-cousins architecture

> Architecture diagram of the GitHub repository cremebrule/digital-cousins, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cremebrule/digital-cousins
- Repository: https://github.com/cremebrule/digital-cousins

## Overview

##### Main System Components

The system should be divided into these major components:

a) Core Pipeline Components:
- Scene Generation Pipeline (/pipeline)
- Feature Extraction System (/models)
- Environment Wrappers (/envs)
- Skills System (/skills)

b) External Dependencies:
- Foundation Models (CLIP, DINO, SAM, etc.)
- Simulation Environments (OmniGibson, Robomimic)
- GPT Integration

**Input Image → ACDC Pipeline → Interactive Scene → Policy Training → Deployment**

**[Input Processing]**
- ↓
**[Feature Extraction] → [Scene Generation] → [Digital Cousin Creation]**
- ↓                        ↓                    ↓
**[Environment Setup] ←→ [Skill System] ←→ [Policy Training]**

##### Component Details

a) Feature Extraction Block:
- Models from /models:
- CLIP
- Depth Anything V2
- DINO V2
- Grounded SAM V2
- Perspective Fields
- Feature Matcher

b) Environment Integration Block:
- OmniGibson Environment
- Robomimic Environment
- Custom Wrappers:
- Cabinet Open Wrapper
- Skill Wrapper
- Data Collection Wrapper

c) Pipeline Processing Block:
- ACDC Pipeline
- Extraction Pipeline
- Generation Pipeline
- Matching Pipeline

##### Implementation Notes

- The automated nature of the pipeline
- Integration with multiple foundation models
- The relationship between digital twins and cousins
- Policy training workflow
- Environment interaction capabilities

##### Technical Details

- Input: RGB image
- Output: Interactive 3D scene
- Supported environments: OmniGibson, Robomimic
- Key technologies: CLIP, DINO, SAM, GPT
- Training capabilities: BC (Behavioral Cloning)

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        RGB[RGB Image]
    end

    subgraph "Feature Extraction"
        FE[Feature Extraction System]:::core
        CLIP[CLIP]:::model
        DINO[DINO V2]:::model
        SAM[Grounded SAM V2]:::model
        DEPTH[Depth Anything V2]:::model
        PF[Perspective Fields]:::model
        FM[Feature Matcher]:::model
        GPT[GPT Integration]:::model

        FE --> CLIP
        FE --> DINO
        FE --> SAM
        FE --> DEPTH
        FE --> PF
        FE --> FM
        FE --> GPT
    end

    subgraph "Pipeline Processing"
        ACDC[ACDC Pipeline]:::pipeline
        EP[Extraction Pipeline]:::pipeline
        GP[Generation Pipeline]:::pipeline
        MP[Matching Pipeline]:::pipeline

        ACDC --> EP
        EP --> GP
        GP --> MP
    end

    subgraph "Environment Integration"
        OG[OmniGibson]:::env
        RM[Robomimic]:::env

        subgraph "Environment Wrappers"
            CW[Cabinet Wrapper]:::wrapper
            SW[Skill Wrapper]:::wrapper
            DCW[Data Collection Wrapper]:::wrapper
            SCW[Skill Collection Wrapper]:::wrapper
        end

        OG --> CW
        OG --> SW
        RM --> DCW
        RM --> SCW
    end

    subgraph "Skills System"
        SB[Skill Base]:::skill
        OCS[Open/Close Skill]:::skill
    end

    subgraph "Configuration"
        DC[Default Config]:::config
        TC[Training Config]:::config
    end

    subgraph "Utilities"
        DU[Dataset Utils]:::util
        SU[Scene Utils]:::util
        PU[Processing Utils]:::util
        TU[Transform Utils]:::util
    end

    RGB --> FE
    FE --> ACDC
    MP --> OG
    MP --> RM
    OG --> SB
    RM --> SB
    SB --> OCS

    %% Click Events
    click FE "https://github.com/cremebrule/digital-cousins/tree/main/digital_cousins/models/"
    click CLIP "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/models/clip.py"
    click DINO "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/models/dino_v2.py"
    click SAM "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/models/grounded_sam_v2.py"
    click DEPTH "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/models/depth_anything_v2.py"
    click PF "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/models/perspective_fields.py"
    click FM "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/models/feature_matcher.py"
    click GPT "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/models/gpt.py"
    click ACDC "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/pipeline/acdc.py"
    click EP "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/pipeline/extraction.py"
    click GP "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/pipeline/generation.py"
    click MP "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/pipeline/matching.py"
    click OG "https://github.com/cremebrule/digital-cousins/tree/main/digital_cousins/envs/omnigibson/"
    click RM "https://github.com/cremebrule/digital-cousins/tree/main/digital_cousins/envs/robomimic/"
    click CW "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/envs/omnigibson/cabinet_open_wrapper.py"
    click SW "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/envs/omnigibson/skill_wrapper.py"
    click DCW "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/envs/robomimic/data_collection_wrapper.py"
    click SCW "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/envs/robomimic/skill_collection_wrapper.py"
    click SB "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/skills/skill_base.py"
    click OCS "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/skills/open_or_close_skill.py"
    click DC "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/configs/default.yaml"
    click TC "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/configs/training/bc_base.json"
    click DU "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/utils/dataset_utils.py"
    click SU "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/utils/scene_utils.py"
    click PU "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/utils/processing_utils.py"
    click TU "https://github.com/cremebrule/digital-cousins/blob/main/digital_cousins/utils/transform_utils.py"

    %% Styling
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef model fill:#4CAF50,stroke:#2E7D32,color:white
    classDef pipeline fill:#1976D2,stroke:#0D47A1,color:white
    classDef env fill:#FFC107,stroke:#FFA000,color:black
    classDef wrapper fill:#FF9800,stroke:#F57C00,color:black
    classDef skill fill:#9C27B0,stroke:#6A1B9A,color:white
    classDef config fill:#795548,stroke:#4E342E,color:white
    classDef util fill:#607D8B,stroke:#455A64,color:white
```
