How joejoe1313/paligemma-image-segmentation works
Project type and purpose
• This is a Dockerized microservice for image-segmentation. It exposes a REST API (FastAPI) that accepts an image URL + prompt, runs a PaliGemma 2 mix model plus a VAE checkpoint to generate segmentation masks, and returns JSON to the client.
• Use case: any client (web UI, script, notebook) uploads an image and prompt, the service returns object masks.
File structure
• Top‐level:
• Dockerfile & docker-compose.yml → containerization & orchestration
• .github/workflows/docker-publish.yml → CI/CD pipeline
• requirements.txt → Python deps
• example.ipynb → usage demo
• Directories:
• app/
– main.py (FastAPI app, endpoint definitions)
– segmentation.py (image-segmentation logic)
• models/
– huggingface/ (placeholder for model artifacts)
– vae-oid.npz (VAE checkpoint)
• This suggests a single‐container service wrapping both API and ML logic.
README for additional insights
• Key technologies: FastAPI, transformers, JAX/Flax, PyTorch, numpy, Docker.
a. Main components
• User / Client Application
• FastAPI Service (main.py)
• Segmentation Logic (segmentation.py)
• PaliGemma 2 Mix Model
• VAE Model Checkpoint
• External dependencies (transformers, JAX/Flax, PyTorch, numpy)
• Docker Container boundary
b. Relationships & interactions
1. Client → HTTP POST /segment → FastAPI Service
2. FastAPI loads segmentation module → invokes PaliGemma model
3. PaliGemma → intermediate segmentation → calls VAE checkpoint
5. API → JSON response → Client
c. Architectural patterns
• Layered architecture: Presentation (Client) → API layer → Domain logic → Model layer
• Single‐service container (monolith + ML assets)
• CI/CD pipeline triggers Docker image build & publish
d. Technologies & frameworks
• FastAPI (Uvicorn)
• Python ML stack: transformers, JAX/Flax, PyTorch, numpy
• Docker & docker-compose
• Optional: GitHub Actions for publishing