# vlongle/articulate-anything architecture

> Architecture diagram of the GitHub repository vlongle/articulate-anything, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vlongle/articulate-anything
- Repository: https://github.com/vlongle/articulate-anything

## Overview

##### Main System Components

The system follows an actor-critic architecture with the following key components:

a) Core Components:
- Actor System (articulate_anything/agent/actor/)
- Critic System (articulate_anything/agent/critic/)
- Multimodal Input Processing
- Physics Simulation Engine

b) Input Processing Components:
- Text Processing
- Image Processing
- Video Processing
- PartNet-Mobility Dataset Interface

**[Text Input] [Image Input] [Video Input] [PartNet-Mobility Dataset]**
- ↓           ↓           ↓              ↓
- [Multimodal Input Processing Pipeline]

- [Actor System]
- ↓
**[Mesh Retrieval] ← [Joint Prediction] → [Link Placement]**
- ↓
- [Critic System]

**[Physics Simulation] → [URDF Generation] → [3D Articulation]**

##### Component Details

Actor System (Blue Components):
- Category Selector
- Object Selector
- Text Layout Planner
- Joint Prediction
- Link Placement

Critic System (Green Components):
- Joint Critic
- Link Critic
- Success Evaluation

Support Systems (Gray Components):
- Configuration Management (Hydra)
- Physics Simulation (PyBullet/Sapien)
- Preprocessing Utilities
- Visualization Tools

##### Data Flow Indicators

- Input processing flow
- Actor-critic feedback loop
- Generation pipeline
- Validation and correction paths

##### Integration Points

Connections with:
- Vision-Language Models (Gemini/GPT/Claude)
- CLIP embeddings
- Physics engines
- PartNet-Mobility dataset

##### Additional Elements

Include:
- Configuration flow from conf/ directory
- Dataset preprocessing pipeline
- Model API integration points
- Output generation pathway

##### The preprocessing and validation pipeline

Keep the abstraction level high enough to show system architecture while including critical implementation details that affect the system's behavior.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input["Input Layer"]
        direction TB
        TextIn[Text Input]:::input
        ImageIn[Image Input]:::input
        VideoIn[Video Input]:::input
        PartNet[PartNet-Mobility Dataset]:::input
    end

    subgraph Processing["Processing Layer"]
        direction TB
        CLIP["CLIP Utils"]:::processing
        VideoProc["Video Processing"]:::processing
        MeshProc["Mesh Processing"]:::processing
        PartnetUtils["PartNet Interface"]:::processing
    end

    subgraph Actor["Actor System"]
        direction TB
        CatSel["Category Selector"]:::actor
        ObjSel["Object Selector"]:::actor
        TextLayout["Text Layout Planner"]:::actor
        MeshRet["Mesh Retrieval"]:::actor
        JointPred["Joint Prediction"]:::actor
        LinkPlace["Link Placement"]:::actor
    end

    subgraph Critic["Critic System"]
        direction TB
        JointCritic["Joint Critic"]:::critic
        LinkCritic["Link Critic"]:::critic
    end

    subgraph Config["Configuration"]
        direction TB
        MainConf["Main Config"]:::config
        JointActorConf["Joint Actor Config"]:::config
        JointCriticConf["Joint Critic Config"]:::config
        LinkActorConf["Link Actor Config"]:::config
        LinkCriticConf["Link Critic Config"]:::config
    end

    subgraph Physics["Physics & Visualization"]
        direction TB
        PyBullet["PyBullet Integration"]:::physics
        Sapien["Sapien Integration"]:::physics
        SapienRender["Sapien Rendering"]:::physics
        VizTools["Visualization Tools"]:::physics
    end

    subgraph Preprocess["Preprocessing"]
        direction TB
        PartnetPrep["PartNet Preprocessing"]:::preprocess
        PrepUtils["Preprocessing Utils"]:::preprocess
        PartnetEmbed["PartNet Embeddings"]:::preprocess
    end

    %% Connections
    Input --> Processing
    Processing --> Actor
    Actor --> Critic
    Critic --> Actor
    Actor --> Physics
    Config --> Actor
    Config --> Critic
    Preprocess --> Processing

    %% Click Events
    click CatSel "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/actor/mesh_retrieval/category_selector.py"
    click ObjSel "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/actor/mesh_retrieval/obj_selector.py"
    click TextLayout "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/actor/mesh_retrieval/text_layout_planner.py"
    click JointPred "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/actor/joint_prediction/joint_actor.py"
    click LinkPlace "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/actor/link_placement/link_placement.py"
    click MeshRet "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/actor/mesh_retrieval/partnet_mesh_retrieval.py"
    click JointCritic "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/critic/joint_prediction/joint_critic.py"
    click LinkCritic "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/agent/critic/link_placement/link_critic.py"
    click PyBullet "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/physics/pybullet_utils.py"
    click Sapien "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/physics/sapien_simulate.py"
    click SapienRender "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/physics/sapien_render.py"
    click MainConf "https://github.com/vlongle/articulate-anything/blob/main/conf/config.yaml"
    click JointActorConf "https://github.com/vlongle/articulate-anything/blob/main/conf/joint_actor/default.yaml"
    click JointCriticConf "https://github.com/vlongle/articulate-anything/blob/main/conf/joint_critic/default.yaml"
    click LinkActorConf "https://github.com/vlongle/articulate-anything/blob/main/conf/link_actor/default.yaml"
    click LinkCriticConf "https://github.com/vlongle/articulate-anything/blob/main/conf/link_critic/default.yaml"
    click PartnetPrep "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/preprocess/preprocess_partnet.py"
    click PrepUtils "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/preprocess/preprocess_utils.py"
    click PartnetEmbed "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/preprocess/create_partnet_embeddings.py"
    click CLIP "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/utils/clip_utils.py"
    click VideoProc "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/utils/cotracker_utils.py"
    click MeshProc "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/utils/mesh_utils.py"
    click PartnetUtils "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/utils/partnet_utils.py"
    click VizTools "https://github.com/vlongle/articulate-anything/blob/main/articulate_anything/utils/viz.py"

    %% Styles
    classDef input fill:#ff9900,stroke:#333,stroke-width:2px
    classDef processing fill:#6699ff,stroke:#333,stroke-width:2px
    classDef actor fill:#3366cc,stroke:#333,stroke-width:2px
    classDef critic fill:#33cc33,stroke:#333,stroke-width:2px
    classDef config fill:#ffcc00,stroke:#333,stroke-width:2px
    classDef physics fill:#9966ff,stroke:#333,stroke-width:2px
    classDef preprocess fill:#ff66cc,stroke:#333,stroke-width:2px
```
