# jamesjg/foodsam architecture

> Architecture diagram of the GitHub repository jamesjg/foodsam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jamesjg/foodsam
- Repository: https://github.com/jamesjg/foodsam

## Overview

##### Project Type & Purpose
- FoodSAM is a full-stack research prototype for “any food” segmentation tasks: semantic, instance, panoptic, and promptable segmentation.
- It integrates three core models:
a. Segment Anything Model (SAM) for class-agnostic mask proposals
b. Semantic segmenter (from mmsegmentation) for coarse food category masks
c. Object detector (from UniDet) for non-food/background classes and prompt prioritization

##### High-Level Architectural Components
1. Data Ingestion
- CLI / Python API accepts single image or folder
- Configuration loader (YAML / Python) reads model weights and parameters
2. Model Inference Engines
- SAM Engine: loads pretrained SAM, generates a set of binary masks per image
- Semantic Segmenter Engine: runs mmsegmentation model, produces per-pixel food class probabilities / mask
3. Mask-Category Matching & Enhancement
- Match SAM masks to semantic predictions via overlap heuristic
- Assign each mask a food category ID (using category_id_files) or non-food class
- Enhancement/Merge Module: integrates coarse semantic and SAM masks to refine boundaries
4. Output Generators
- Semantic Segmentation: refined per-pixel food categories
- Instance Segmentation: labeled masks from SAM + category assignment
- Panoptic Segmentation: unified “stuff” (semantic) + “things” (instances)
- Promptable Segmentation: selective mask generation guided by object detector prompts
5. Evaluation & Visualization
- Uses FoodSeg103 and UECFOODPIXCOMPLETE datasets for metric computation
- Visualizer module plots masks and computes mIoU, mAcc, aAcc

##### Relationships & Data Flow
- Data Ingestion → (parallel) SAM, Semantic Segmenter, Object Detector
- Their outputs converge at Mask-Category Matching → goes into Enhancement/Merge
- Enhanced masks split into three pipelines: Semantic / Instance / Panoptic (plus optional Promptable)
- Final results passed to Visualization & Evaluation

##### Architectural Patterns & Design Principles
- Modular pipeline: each model is an independent component with well-defined I/O
- Configuration-driven: YAML/Python configs decouple weights, hyperparameters, and dataset paths
- Reuse & extension: builds on external frameworks (Segment-Anything, mmsegmentation, detectron2/UniDet)
- Zero-shot capability: minimal retraining; model fusion via mask merges rather than monolithic training

##### Technologies & Frameworks
- PyTorch (core training & inference)
- Segment-Anything (SAM)
- mmsegmentation (semantic segmentation)
- Detectron2 / UniDet (object detection)
- NumPy (mask arrays), YAML for configs
- CLI wrappers in FoodSAM/semantic.py and FoodSAM/panoptic.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        DL1["CLI Semantic"]:::grey
        DL2["CLI Panoptic"]:::grey
        Config["Config Loader"]:::grey
    end

    %% Inference Engines
    subgraph "Inference Engines"
        SAM["SAM Engine\n(SAM, Facebook Research)"]:::blue
        SEM["Semantic Engine\n(mmseg FPN)"]:::blue
        DET["Detector Engine\n(UniDet RCNN)"]:::blue
    end

    %% Post-Processing
    subgraph "Mask Matching & Enhancement"
        Match["Mask-Category Matching"]:::green
        Enhance["Enhancement/Merge"]:::green
    end

    %% Output Generators
    subgraph "Output Generators"
        SemOut["Semantic Segmentation"]:::orange
        InstOut["Instance Segmentation"]:::orange
        PanOut["Panoptic Segmentation"]:::orange
        PromptOut["Promptable Segmentation"]:::orange
    end

    %% Visualization & Evaluation
    subgraph "Visualization & Evaluation"
        Viz["Visualization & Evaluation"]:::grey
    end

    %% Data Flow
    DL1 -->|images| SAM
    DL1 -->|images| SEM
    DL1 -->|images| DET
    DL2 -->|images| SAM
    DL2 -->|images| SEM
    DL2 -->|images| DET
    Config --> SAM
    Config --> SEM
    Config --> DET

    DET -.->|prompt injection| SAM

    SAM -->|binary masks| Match
    SEM -->|coarse semantic map| Match
    DET -->|bboxes + classes| Match
    Match --> Enhance

    Enhance -->|refined semantic map| SemOut
    Enhance -->|labeled masks| InstOut
    Enhance -->|panoptic map| PanOut
    Enhance -->|promptable masks| PromptOut

    SemOut --> Viz
    InstOut --> Viz
    PanOut --> Viz
    PromptOut --> Viz

    %% Click Events
    click DL1 "https://github.com/jamesjg/foodsam/blob/main/FoodSAM/semantic.py"
    click DL2 "https://github.com/jamesjg/foodsam/blob/main/FoodSAM/panoptic.py"
    click Config "https://github.com/jamesjg/foodsam/tree/main/configs/"
    click SAM "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/panoramic_segment.py"
    click SEM "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/predict_semantic_mask.py"
    click DET "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/object_detection.py"
    click Match "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/enhance_semantic_masks.py"
    click Enhance "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/enhance_semantic_masks.py"
    click SemOut "https://github.com/jamesjg/foodsam/blob/main/FoodSAM/semantic.py"
    click InstOut "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/panoramic_segment.py"
    click PanOut "https://github.com/jamesjg/foodsam/blob/main/FoodSAM/panoptic.py"
    click PromptOut "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/panoramic_segment.py"
    click Viz "https://github.com/jamesjg/foodsam/blob/main/FoodSAM_tools/evaluate_foodseg103.py"

    %% Styles
    classDef blue fill:#cce5ff,stroke:#004085,color:#004085;
    classDef grey fill:#e2e3e5,stroke:#6c757d,color:#383d41,stroke-dasharray: 5 5;
    classDef green fill:#d4edda,stroke:#155724,color:#155724;
    classDef orange fill:#ffe5b4,stroke:#cc8400,color:#cc8400;
```
