How cremebrule/digital-cousins works
Main System Components
The system should be divided into these major components:
a) Core Pipeline Components:
• Scene Generation Pipeline (/pipeline)
• Feature Extraction System (/models)
• Environment Wrappers (/envs)
• Skills System (/skills)
b) External Dependencies:
• Foundation Models (CLIP, DINO, SAM, etc.)
• Simulation Environments (OmniGibson, Robomimic)
• GPT Integration
Input Image → ACDC Pipeline → Interactive Scene → Policy Training → Deployment
[Input Processing]
• ↓
[Feature Extraction] → [Scene Generation] → [Digital Cousin Creation]
• ↓ ↓ ↓
[Environment Setup] ←→ [Skill System] ←→ [Policy Training]
Component Details
a) Feature Extraction Block:
• Models from /models:
• CLIP
• Depth Anything V2
• DINO V2
• Grounded SAM V2
• Perspective Fields
• Feature Matcher
b) Environment Integration Block:
• OmniGibson Environment
• Robomimic Environment
• Custom Wrappers:
• Cabinet Open Wrapper
• Skill Wrapper
• Data Collection Wrapper
c) Pipeline Processing Block:
• ACDC Pipeline
• Extraction Pipeline
• Generation Pipeline
• Matching Pipeline
Implementation Notes
• The automated nature of the pipeline
• Integration with multiple foundation models
• The relationship between digital twins and cousins
• Policy training workflow
• Environment interaction capabilities
Technical Details
• Input: RGB image
• Output: Interactive 3D scene
• Supported environments: OmniGibson, Robomimic
• Key technologies: CLIP, DINO, SAM, GPT
• Training capabilities: BC (Behavioral Cloning)