# microsoft/som architecture

> Architecture diagram of the GitHub repository microsoft/som, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/som
- Repository: https://github.com/microsoft/som

## Overview

##### Project Type and Purpose
- Type: Research prototype / toolkit combining image segmentation models with an LLM (GPT‑4V) via visual prompts.
- Purpose: Generate spatial “marks” on images (via segmentation backends), overlay them, and feed interleaved visual+text prompts to GPT‑4V for grounded reasoning.

##### File Structure
- Top‑level scripts & entrypoints:
– demo_som.py, demo_gpt4v_som.py, client.py ⇒ user‐facing demos/CLI utilities
– deploy.py + entrypoint.sh + Dockerfile + GitHub Actions template ⇒ containerization & AWS EC2 deployment
- Core library & plugins:
– task_adapter/… ⇒ module‐per‐segmentation‐model plugin (sam, seem, semantic_sam)
– ops/… ⇒ custom C++/CUDA operations (ms_deform_attn) to accelerate segmentation
- Configuration & assets:
– configs/*.yaml ⇒ model/hyperparameter settings
– download_ckpt.sh ⇒ weight‐fetching script
– assets/… ⇒ logos, demo videos
- Benchmark & examples:
– benchmark/…, examples/, gpt4v.py ⇒ evaluation harness & quick‐start usage

##### Extract Key Insights from README
- Multiple third‐party segmentation models are supported via plugin pattern.
- Pretrained checkpoints are downloaded and loaded at runtime.
- The workflow is: image → mark generator → mark overlay → GPT‑4V API call → response → UI/CLI output.
- Deployment can be via Docker on AWS EC2 (GitHub Actions CI/CD generates the image).

##### Main Components & Their Relationships
a. User Interface / CLI
– demo scripts (demo_som.py, demo_gpt4v_som.py) or client.py
– Reads local image, accepts user inputs (interactive mark selection or automatic mode).
b. SoM Core Library
– Orchestrates the pipeline: loads config → invokes Task Adapters → collects masks → overlays marks.
– Task Adapters load specific segmentation models (SAM, SEEM, Semantic‑SAM) through a common API.
c. Custom Ops Layer
– ops/modules + ops/functions + ops/src: compiles and provides optimized kernels for ms_deform_attn.
– Invoked by segmentation backends for attention operations.
d. Model Weights Storage
– Local filesystem (downloaded via download_ckpt.sh) or container image volume.
e. External Services
– GPT‑4V API (OpenAI) accessed by client library in Python (requires OPENAI_API_KEY).
f. Deployment Infrastructure
– Docker container (Dockerfile + entrypoint.sh)
– AWS EC2 provisioning via deploy.py + GitHub Actions (docker-build-ec2.yml.j2)

##### Architectural Patterns & Design Principles
- Plugin Pattern: task_adapter/* folders each implement a uniform interface for different segmentation engines.
- Layered Architecture: UI Layer → Application Orchestration → Model Inference → External API.
- Separation of Concerns: heavy compute in custom ops, model logic in adapters, prompt logic in core, deployment scripts isolated.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface / CLI
    subgraph "User Interface / CLI"
        DemoSom["demo_som.py"]:::python
        DemoGpt4vSom["demo_gpt4v_som.py"]:::python
        Client["client.py"]:::python
    end

    %% SoM Core Library
    subgraph "SoM Core Library"
        Core["gpt4v.py"]:::python
        Examples["examples/"]:::python
        Configs["configs/"]:::storage
    end

    %% Task Adapters
    subgraph "Task Adapters"
        subgraph "SAM Adapter"
            SAMAuto["inference_sam_m2m_auto.py"]:::python
            SAMInteract["inference_sam_m2m_interactive.py"]:::python
        end
        subgraph "SEEM Adapter"
            SEEMAuto["automatic_mask_generator.py"]:::python
            SEEMInteract["inference_seem_interactive.py"]:::python
            SEEMPano["inference_seem_pano.py"]:::python
            SEEMM2M["interactive_seem_m2m_auto.py"]:::python
        end
        subgraph "Semantic-SAM Adapter"
            SS_Auto["automatic_mask_generator.py"]:::python
            SS_m2m["inference_semsam_m2m_auto.py"]:::python
            SS_box["interactive_idino_1o1_box.py"]:::python
            SS_m2m2["interactive_idino_m2m.py"]:::python
            SS_pred["interactive_predictor.py"]:::python
        end
        UtilsVis["visualizer.py"]:::python
    end

    %% Custom Ops Layer
    subgraph "Custom Ops Layer"
        MSFunc["ms_deform_attn_func.py"]:::ops
        MSModule["ms_deform_attn.py"]:::ops
        subgraph "ops/src"
            CPUcpp["ms_deform_attn_cpu.cpp"]:::ops
            CPUh["ms_deform_attn_cpu.h"]:::ops
            CUCuda["ms_deform_attn_cuda.cu"]:::ops
            CUCuh["ms_deform_attn_cuda.h"]:::ops
            IM2Col["ms_deform_im2col_cuda.cuh"]:::ops
            Header1["ms_deform_attn.h"]:::ops
            Vision["vision.cpp"]:::ops
        end
        Make["make.sh"]:::ops
        Setup["setup.py"]:::ops
    end

    %% Configuration & Checkpoint Retrieval
    subgraph "Configuration & Checkpoint Retrieval"
        YAML1["seem_focall_unicl_lang_v1.yaml"]:::storage
        YAML2["semantic_sam_only_sa-1b_swinL.yaml"]:::storage
        CKPT["download_ckpt.sh"]:::storage
    end

    %% Assets
    Assets["assets/som_gpt4v_demo.mp4"]:::storage

    %% External GPT-4V API
    External["GPT-4V API"]:::external

    %% Deployment Infrastructure
    subgraph "Deployment Infrastructure"
        Dockerfile["Dockerfile"]:::deploy
        Entrypoint["entrypoint.sh"]:::deploy
        DeployPy["deploy.py"]:::deploy
        Req["deploy_requirements.txt"]:::deploy
        GHAction["docker-build-ec2.yml.j2"]:::deploy
        AWS["AWS EC2"]:::deploy
    end

    %% Data Flows
    Client -->|invokes| DemoSom
    Client -->|invokes| DemoGpt4vSom
    DemoSom -->|reads image+input| Core
    DemoGpt4vSom -->|reads image+input| Core
    Core -->|loads config| Configs
    CKPT -->|downloads weights| Core
    Core -->|invoke| SAMAuto
    Core -->|invoke| SAMInteract
    Core -->|invoke| SEEMAuto
    Core -->|invoke| SEEMInteract
    Core -->|invoke| SEEMPano
    Core -->|invoke| SEEMM2M
    Core -->|invoke| SS_Auto
    Core -->|invoke| SS_m2m
    Core -->|invoke| SS_box
    Core -->|invoke| SS_m2m2
    Core -->|invoke| SS_pred
    SAMAuto -->|calls| MSModule
    SAMInteract -->|calls| MSModule
    SEEMAuto -->|calls| MSModule
    SEEMInteract -->|calls| MSModule
    SEEMPano -->|calls| MSModule
    SEEMM2M -->|calls| MSModule
    SS_Auto -->|calls| MSModule
    SS_m2m -->|calls| MSModule
    SS_box -->|calls| MSModule
    SS_m2m2 -->|calls| MSModule
    SS_pred -->|calls| MSModule
    Core -->|overlay marks| Core
    Core -->|send JSONprompt+image| External
    External -->|response| Core
    Core -->|output| DemoSom
    Core -->|output| DemoGpt4vSom

    GHAction -->|builds image| Dockerfile
    Dockerfile -->|container image| AWS
    Entrypoint -->|starts container| AWS
    DeployPy -->|provisions| AWS
    Req -->|installs deps| AWS

    %% Click Events
    click DemoSom "https://github.com/microsoft/som/blob/main/demo_som.py"
    click DemoGpt4vSom "https://github.com/microsoft/som/blob/main/demo_gpt4v_som.py"
    click Client "https://github.com/microsoft/som/blob/main/client.py"
    click Core "https://github.com/microsoft/som/blob/main/gpt4v.py"
    click Examples "https://github.com/microsoft/som/tree/main/examples/"
    click Configs "https://github.com/microsoft/som/tree/main/configs/"
    click SAMAuto "https://github.com/microsoft/som/blob/main/task_adapter/sam/tasks/inference_sam_m2m_auto.py"
    click SAMInteract "https://github.com/microsoft/som/blob/main/task_adapter/sam/tasks/inference_sam_m2m_interactive.py"
    click SEEMAuto "https://github.com/microsoft/som/blob/main/task_adapter/seem/tasks/automatic_mask_generator.py"
    click SEEMInteract "https://github.com/microsoft/som/blob/main/task_adapter/seem/tasks/inference_seem_interactive.py"
    click SEEMPano "https://github.com/microsoft/som/blob/main/task_adapter/seem/tasks/inference_seem_pano.py"
    click SEEMM2M "https://github.com/microsoft/som/blob/main/task_adapter/seem/tasks/interactive_seem_m2m_auto.py"
    click SS_Auto "https://github.com/microsoft/som/blob/main/task_adapter/semantic_sam/tasks/automatic_mask_generator.py"
    click SS_m2m "https://github.com/microsoft/som/blob/main/task_adapter/semantic_sam/tasks/inference_semsam_m2m_auto.py"
    click SS_box "https://github.com/microsoft/som/blob/main/task_adapter/semantic_sam/tasks/interactive_idino_1o1_box.py"
    click SS_m2m2 "https://github.com/microsoft/som/blob/main/task_adapter/semantic_sam/tasks/interactive_idino_m2m.py"
    click SS_pred "https://github.com/microsoft/som/blob/main/task_adapter/semantic_sam/tasks/interactive_predictor.py"
    click UtilsVis "https://github.com/microsoft/som/blob/main/task_adapter/utils/visualizer.py"
    click MSFunc "https://github.com/microsoft/som/blob/main/ops/functions/ms_deform_attn_func.py"
    click MSModule "https://github.com/microsoft/som/blob/main/ops/modules/ms_deform_attn.py"
    click CPUcpp "https://github.com/microsoft/som/blob/main/ops/src/cpu/ms_deform_attn_cpu.cpp"
    click CPUh "https://github.com/microsoft/som/blob/main/ops/src/cpu/ms_deform_attn_cpu.h"
    click CUCuda "https://github.com/microsoft/som/blob/main/ops/src/cuda/ms_deform_attn_cuda.cu"
    click CUCuh "https://github.com/microsoft/som/blob/main/ops/src/cuda/ms_deform_attn_cuda.h"
    click IM2Col "https://github.com/microsoft/som/blob/main/ops/src/cuda/ms_deform_im2col_cuda.cuh"
    click Header1 "https://github.com/microsoft/som/blob/main/ops/src/ms_deform_attn.h"
    click Vision "https://github.com/microsoft/som/blob/main/ops/src/vision.cpp"
    click Make "https://github.com/microsoft/som/blob/main/ops/make.sh"
    click Setup "https://github.com/microsoft/som/blob/main/ops/setup.py"
    click YAML1 "https://github.com/microsoft/som/blob/main/configs/seem_focall_unicl_lang_v1.yaml"
    click YAML2 "https://github.com/microsoft/som/blob/main/configs/semantic_sam_only_sa-1b_swinL.yaml"
    click CKPT "https://github.com/microsoft/som/blob/main/download_ckpt.sh"
    click Assets "https://github.com/microsoft/som/blob/main/assets/som_gpt4v_demo.mp4"
    click Dockerfile "https://github.com/microsoft/som/tree/main/Dockerfile"
    click Entrypoint "https://github.com/microsoft/som/blob/main/entrypoint.sh"
    click DeployPy "https://github.com/microsoft/som/blob/main/deploy.py"
    click Req "https://github.com/microsoft/som/blob/main/deploy_requirements.txt"
    click GHAction "https://github.com/microsoft/som/blob/main/docker-build-ec2.yml.j2"

    %% Styles
    classDef python fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1
    classDef ops fill:#C8E6C9,stroke:#388E3C,color:#1B5E20
    classDef storage fill:#FFE0B2,stroke:#F57C00,color:#E65100
    classDef external fill:#E1BEE7,stroke:#8E24AA,color:#4A148C
    classDef deploy fill:#CFD8DC,stroke:#607D8B,color:#263238
```
