How zdenek-valehrach/segmenstory works
Project type and purpose
• SegmenStory is a monolithic, front-end-centric “full-stack” application built with Streamlit. It binds a simple UI (Streamlit app) to two external AI services:
• Mask2Former segmentation via Hugging Face API
• Generative text (Perplexity “sonar” or OpenAI gpt-4o)
• There is no persistent database; all processing happens at runtime in a single Python process.
File structure
• Top‐level:
• app.py – Streamlit UI entry point
• config.py – defaults for all API keys and parameters
• models/
• segmentation.py – wrapper for Mask2Former calls and mask decoding
• llm.py – wrapper for Perplexity/OpenAI text generation
• utils/
• image_utils.py – resizing, format conversion, mask overlay
• prompt_utils.py – prompt‐building logic
• coco_class_map.py – Czech translation and normalization of COCO class names
• .devcontainer – environment definition
README for additional insights
• Describes stepwise workflow: image upload → segmentation → mask visualization → segment selection → prompt generation → LLM call → story display
• Highlights key modules you already see in the tree
• Streamlit UI (app.py + Streamlit runtime)
• Core Processing modules:
– Segmentation Module (models/segmentation.py + image_utils)
– LLM Module (models/llm.py + prompt_utils + coco_class_map)
• Configuration Manager (config.py / sidebar input)
– Hugging Face Mask2Former API
– Perplexity API
– OpenAI API
1. User → Streamlit UI: upload image, enter API keys, choose provider/topic
2. UI → Segmentation Module: raw image → resized image → HF API call
4. UI → LLM Module: selected segment class + optional user topic → prompt_utils + coco_class_map → built prompt
5. LLM Module → chosen API (Perplexity or OpenAI) → raw story → artifact cleaner → UI for display
6. Config Manager → provides API keys and default params to both modules
c. Architectural patterns & principles:
• Modular monolith with clear separation of concerns (UI, segmentation, LLM, utils, config)
• Pipeline pattern: data flows through discrete stages
• Dependency inversion: high-level modules (app.py) depend on abstractions (wrappers in models/)
d. Key technologies & frameworks:
• Python 3.x, Streamlit for UI
• Hugging Face REST API (Mask2Former)
• OpenAI REST API, Perplexity REST API
• PIL, NumPy for image processing
• Base64 encoding/decoding for masks