How jamesjg/foodsam works
Project Type & Purpose
• FoodSAM is a full-stack research prototype for “any food” segmentation tasks: semantic, instance, panoptic, and promptable segmentation.
• It integrates three core models:
a. Segment Anything Model (SAM) for class-agnostic mask proposals
b. Semantic segmenter (from mmsegmentation) for coarse food category masks
c. Object detector (from UniDet) for non-food/background classes and prompt prioritization
High-Level Architectural Components
1. Data Ingestion
• CLI / Python API accepts single image or folder
• Configuration loader (YAML / Python) reads model weights and parameters
2. Model Inference Engines
• SAM Engine: loads pretrained SAM, generates a set of binary masks per image
• Semantic Segmenter Engine: runs mmsegmentation model, produces per-pixel food class probabilities / mask
3. Mask-Category Matching & Enhancement
• Match SAM masks to semantic predictions via overlap heuristic
• Assign each mask a food category ID (using category_id_files) or non-food class
• Enhancement/Merge Module: integrates coarse semantic and SAM masks to refine boundaries
4. Output Generators
• Semantic Segmentation: refined per-pixel food categories
• Instance Segmentation: labeled masks from SAM + category assignment
• Panoptic Segmentation: unified “stuff” (semantic) + “things” (instances)
• Promptable Segmentation: selective mask generation guided by object detector prompts
5. Evaluation & Visualization
• Uses FoodSeg103 and UECFOODPIXCOMPLETE datasets for metric computation
• Visualizer module plots masks and computes mIoU, mAcc, aAcc
Relationships & Data Flow
• Data Ingestion → (parallel) SAM, Semantic Segmenter, Object Detector
• Their outputs converge at Mask-Category Matching → goes into Enhancement/Merge
• Enhanced masks split into three pipelines: Semantic / Instance / Panoptic (plus optional Promptable)
• Final results passed to Visualization & Evaluation
Architectural Patterns & Design Principles
• Modular pipeline: each model is an independent component with well-defined I/O
• Configuration-driven: YAML/Python configs decouple weights, hyperparameters, and dataset paths
• Reuse & extension: builds on external frameworks (Segment-Anything, mmsegmentation, detectron2/UniDet)
• Zero-shot capability: minimal retraining; model fusion via mask merges rather than monolithic training
Technologies & Frameworks
• PyTorch (core training & inference)
• Segment-Anything (SAM)
• mmsegmentation (semantic segmentation)
• Detectron2 / UniDet (object detection)
• NumPy (mask arrays), YAML for configs
• CLI wrappers in FoodSAM/semantic.py and FoodSAM/panoptic.py