# joejoe1313/paligemma-image-segmentation architecture

> Architecture diagram of the GitHub repository joejoe1313/paligemma-image-segmentation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joejoe1313/paligemma-image-segmentation
- Repository: https://github.com/joejoe1313/paligemma-image-segmentation

## Overview

##### Project type and purpose
- This is a Dockerized microservice for image-segmentation. It exposes a REST API (FastAPI) that accepts an image URL + prompt, runs a PaliGemma 2 mix model plus a VAE checkpoint to generate segmentation masks, and returns JSON to the client.
- Use case: any client (web UI, script, notebook) uploads an image and prompt, the service returns object masks.

##### File structure
- Top‐level:
- Dockerfile & docker-compose.yml → containerization & orchestration
- .github/workflows/docker-publish.yml → CI/CD pipeline
- requirements.txt → Python deps
- example.ipynb → usage demo
- Directories:
- app/
– main.py (FastAPI app, endpoint definitions)
– segmentation.py (image-segmentation logic)
- models/
– huggingface/ (placeholder for model artifacts)
– vae-oid.npz (VAE checkpoint)
- This suggests a single‐container service wrapping both API and ML logic.

##### README for additional insights
- Key technologies: FastAPI, transformers, JAX/Flax, PyTorch, numpy, Docker.

a. Main components
- User / Client Application
- FastAPI Service (main.py)
- Segmentation Logic (segmentation.py)
- PaliGemma 2 Mix Model
- VAE Model Checkpoint
- External dependencies (transformers, JAX/Flax, PyTorch, numpy)
- Docker Container boundary
b. Relationships & interactions
1. Client → HTTP POST /segment → FastAPI Service
2. FastAPI loads segmentation module → invokes PaliGemma model
3. PaliGemma → intermediate segmentation → calls VAE checkpoint
5. API → JSON response → Client
c. Architectural patterns
- Layered architecture: Presentation (Client) → API layer → Domain logic → Model layer
- Single‐service container (monolith + ML assets)
- CI/CD pipeline triggers Docker image build & publish
d. Technologies & frameworks
- FastAPI (Uvicorn)
- Python ML stack: transformers, JAX/Flax, PyTorch, numpy
- Docker & docker-compose
- Optional: GitHub Actions for publishing

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client
    Client["User / Client"]:::client

    %% Docker Container Boundary
    subgraph "Docker Container"
        direction TB
        API["FastAPI Service<br>(main.py)"]:::api
        SegLogic["Segmentation Logic<br>(segmentation.py)"]:::logic
        PGModel["PaliGemma 2 Mix Model<br>(models/huggingface/)"]:::model
        VAE["VAE Model Checkpoint<br>(vae-oid.npz)"]:::model
        Deps["Dependencies<br>(transformers, JAX/Flax, PyTorch, numpy)"]:::deps
    end

    %% Peripheral Infrastructure
    subgraph "Container Configuration" 
        direction TB
        DockerfileNode["Dockerfile"]:::infra
        Compose["docker-compose.yml"]:::infra
    end

    CI["CI/CD Pipeline<br>(.github/workflows/docker-publish.yml)"]:::infra

    %% Data Flows
    Client -->|"HTTP POST /segment"| API
    API -->|"invokes"| SegLogic
    SegLogic -->|"calls"| PGModel
    PGModel -->|"loads checkpoint"| VAE
    API -->|"returns JSON with masks & coords"| Client

    %% Dependencies
    Deps -.-> API
    Deps -.-> SegLogic

    %% CI/CD Flow
    CI -->|"triggers build"| DockerfileNode
    CI -->|"triggers build"| Compose

    %% Click Events
    click API "https://github.com/joejoe1313/paligemma-image-segmentation/blob/main/app/main.py"
    click SegLogic "https://github.com/joejoe1313/paligemma-image-segmentation/blob/main/app/segmentation.py"
    click PGModel "https://github.com/joejoe1313/paligemma-image-segmentation/tree/main/models/huggingface/"
    click VAE "https://github.com/joejoe1313/paligemma-image-segmentation/blob/main/models/vae-oid.npz"
    click Deps "https://github.com/joejoe1313/paligemma-image-segmentation/blob/main/requirements.txt"
    click DockerfileNode "https://github.com/joejoe1313/paligemma-image-segmentation/tree/main/Dockerfile"
    click Compose "https://github.com/joejoe1313/paligemma-image-segmentation/blob/main/docker-compose.yml"
    click CI "https://github.com/joejoe1313/paligemma-image-segmentation/blob/main/.github/workflows/docker-publish.yml"

    %% Styles
    classDef client fill:#f9f,stroke:#333,stroke-width:1px
    classDef api fill:#88f,stroke:#333,stroke-width:1px
    classDef logic fill:#8fd,stroke:#333,stroke-width:1px
    classDef model fill:#dfd,stroke:#333,stroke-width:1px
    classDef deps fill:#fdc,stroke:#333,stroke-width:1px
    classDef infra fill:#eee,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
