# yogendra-yatnalkar/storyboard-ai architecture

> Architecture diagram of the GitHub repository yogendra-yatnalkar/storyboard-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yogendra-yatnalkar/storyboard-ai
- Repository: https://github.com/yogendra-yatnalkar/storyboard-ai

## Overview

1. **Repository shape**
   - This repo is split into two major subsystems:
     - **`genai-pipeline/`**: the end-to-end Storyboard AI application that turns a prompt into a narrated whiteboard video.
     - **`sam3-hosting/`**: an optional self-hosted SAM 3 segmentation service and the full SAM3 model codebase.
   - The architectural boundary is clear: the pipeline can run without SAM3, but if `SAM_API_URL` is configured it calls the SAM3 service as a remote dependency.

2. **Primary runtime flow**
   - The main entrypoint is **`genai-pipeline/pipeline.py`**, which acts as an interactive CLI orchestrator.
   - It collects user choices such as topic, research mode, language, reference-image usage, fast mode, and whether to use Veo.
   - From there, the pipeline fans out into specialized tools to research, write, generate visuals, synthesize audio, and assemble the final video.

3. **Orchestration layer / Director Agent**
   - The architectural “brain” is the **Director Agent** in `genai-pipeline/tools/director.py`.
   - It decomposes one high-level prompt into scene-level tasks and delegates work to tool modules.
   - This is the main coordination boundary: downstream modules are mostly specialized workers rather than autonomous controllers.

4. **Tool-based multimodal pipeline**
   - The `genai-pipeline/tools/` package is organized as a set of single-purpose stages:
     - research and reference lookup: `research.py`, `reference_search.py`
     - script and prompt shaping: `narration_refiner.py`, `image_prompt_tool.py`
     - image generation: `image_gen.py`
     - animation and segmentation: `draw_animation.py`, `segmentation.py`, `divider.py`
     - audio generation and processing: `tts.py`, `transcribe_audio.py`
     - subtitle generation and burning: `subtitle.py`, `video_subtitle.py`
     - video assembly: `video_gen.py`, `concatenate_videos.py`, `merge_audio_video.py`
   - These stages form a mostly linear pipeline with optional branches for richer visuals or narration handling.

5. **Data flow through the pipeline**
   - Input starts as a prompt and optional language/research settings.
   - The Director produces a scene plan and narration/script.
   - Per scene, the system may fetch reference material, generate a custom image, create animation strokes or video segments, synthesize speech, and generate subtitles.
   - Final outputs are merged into a single video artifact, while scene-local intermediates are kept for inspection and debugging.

6. **SAM3 integration boundary**
   - SAM3 is treated as an optional external service rather than a hard dependency.
   - When enabled, `genai-pipeline` sends image-related requests to a SAM3 endpoint; when disabled, the pipeline falls back to single-pass drawing.
   - This means the whiteboard animation quality can scale from “no segmentation server” to “multi-pass segmentation-assisted drawing” without changing the main pipeline structure.

7. **SAM3 hosting subsystem**
   - `sam3-hosting/` contains a deployable **FastAPI-style service** plus the full SAM3 model package.
   - The host includes:
     - model inference and agent code under `sam3/agent/`
     - model definitions under `sam3/model/`
     - performance kernels and ops under `sam3/perflib/`
     - evaluation tooling under `sam3/eval/`
     - training and dataset code under `sam3/train/`
   - In the architecture diagram, this should be represented as a separate inference service that the pipeline calls remotely.

8. **Deployment and infrastructure**
   - `sam3-hosting/Dockerfile` indicates the SAM3 service is containerized.
   - The README describes deployment to **Google Cloud Run with GPU acceleration**, making the model server an externally hosted compute component.
   - Environment setup uses `.env` values such as Google API credentials and optional Hugging Face access for model weights.

9. **External services**
   - The pipeline depends materially on Google’s GenAI stack:
     - LLM for planning and script generation
     - image generation for line-art visuals
     - TTS for narration
     - Veo for optional video segments
   - The repo also supports `GOOGLE_GENAI_USE_VERTEXAI`, so the same orchestration can target either Google Developer API or Vertex AI depending on deployment.

10. **Persistence and outputs**
   - Generated artifacts are written under **`genai-pipeline/output/run_<timestamp>/`**.
   - The main deliverable is `storyboard_final_video.mp4`.
   - Scene folders contain intermediates such as generated images, narration audio, subtitles, masks, and scene-level sketch videos.
   - This output layout is important for understanding the pipeline boundary between transient processing and final deliverables.

11. **Testing and validation**
   - `genai-pipeline/test_scripts/` contains tool-level and pipeline-level test drivers, including offline Veo/SAM3 checks.
   - `sam3-hosting/sam3/perflib/tests/` provides low-level correctness checks for segmentation-related ops.
   - The presence of these separate test surfaces reflects the repo’s split between orchestration validation and model/inference validation.

12. **What is not part of the runtime path**
   - Much of `sam3-hosting/sam3/train/` and `sam3-hosting/sam3/eval/` is supporting ML infrastructure, not the default production path.
   - For an architecture diagram, treat those directories as offline training/evaluation capabilities around the SAM3 service, not as part of the Storyboard AI request lifecycle.

13. **Best diagram abstraction**
   - A clear architecture diagram would show:
     - **User prompt / CLI**
     - **Director Agent / pipeline orchestrator**
     - **Parallel or sequential tool stages**
     - **External Google GenAI services**
     - **Optional SAM3 segmentation service**
     - **Output assembly and final MP4**
   - The key design idea is an agentic multimodal pipeline with an optional remote segmentation accelerator.

Main entry points: [`genai-pipeline/tools/director.py`](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/director.py) (Director), [`genai-pipeline/tools/video_gen.py`](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/video_gen.py) (Video assembly), [`sam3-hosting/sam3/sam3/model_builder.py`](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/sam3/sam3/model_builder.py) (SAM3 model), [`genai-pipeline/pipeline.py`](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/pipeline.py) (Pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_genai_pipeline["genai-pipeline"]
  node_user_input["Prompt & settings<br/>cli input<br/>[pipeline.py]"]
  node_pipeline_orchestrator["Pipeline<br/>runtime orchestrator<br/>[pipeline.py]"]
  node_director_agent["Director<br/>scene planner<br/>[director.py]"]
  node_research_tools["Research<br/>info gathering<br/>[research.py]"]
  node_reference_search["References<br/>reference lookup"]
  node_script_tools["Script shaping<br/>narration prep"]
  node_image_generation["Image gen<br/>visual generation<br/>[image_gen.py]"]
  node_animation_tools["Animation<br/>draw pipeline<br/>[draw_animation.py]"]
  node_segmentation_bridge["Segmentation<br/>sam3 client<br/>[segmentation.py]"]
  node_audio_tools["Audio<br/>tts/transcribe<br/>[tts.py]"]
  node_subtitle_tools["Subtitles<br/>captioning<br/>[subtitle.py]"]
  node_video_assembly["Video assembly<br/>merge stage<br/>[video_gen.py]"]
  node_run_output[("Run output<br/>artifact store<br/>[README.md]")]
end

subgraph group_sam3_hosting["sam3-hosting"]
  node_sam3_service["SAM3 service<br/>remote inference<br/>[README.md]"]
  node_sam3_model_core["SAM3 model<br/>vision model stack<br/>[model_builder.py]"]
  node_sam3_ops{{"Perf ops<br/>kernels & ops<br/>[compile.py]"}}
  node_sam3_train_eval["Train & eval<br/>offline ml tooling<br/>[train.py]"]
end

subgraph group_external_services["External services"]
  node_genai_api(("Google GenAI<br/>llm/image/tts/video api"))
  node_vertexai_flag(("Vertex AI<br/>deployment mode<br/>[.env.example]"))
end

node_user_input -->|"start run"| node_pipeline_orchestrator
node_pipeline_orchestrator -->|"plan scenes"| node_director_agent
node_director_agent -->|"research"| node_research_tools
node_director_agent -->|"find refs"| node_reference_search
node_director_agent -->|"shape script"| node_script_tools
node_director_agent -->|"request visuals"| node_image_generation
node_director_agent -->|"build motion"| node_animation_tools
node_animation_tools -->|"optional assist"| node_segmentation_bridge
node_segmentation_bridge -->|"remote call"| node_sam3_service
node_sam3_service -->|"infer"| node_sam3_model_core
node_sam3_model_core -->|"use ops"| node_sam3_ops
node_director_agent -->|"narrate"| node_audio_tools
node_director_agent -->|"caption"| node_subtitle_tools
node_director_agent -->|"assemble"| node_video_assembly
node_image_generation -->|"scene assets"| node_video_assembly
node_audio_tools -->|"audio track"| node_video_assembly
node_subtitle_tools -->|"subtitle track"| node_video_assembly
node_video_assembly -->|"write artifacts"| node_run_output
node_pipeline_orchestrator -->|"LLM/media"| node_genai_api
node_genai_api -->|"backend mode"| node_vertexai_flag
node_sam3_service -->|"hosts"| node_sam3_model_core
node_sam3_train_eval -.->|"supports"| node_sam3_model_core

click node_user_input "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/pipeline.py"
click node_pipeline_orchestrator "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/pipeline.py"
click node_director_agent "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/director.py"
click node_research_tools "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/research.py"
click node_reference_search "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/reference_search.py"
click node_script_tools "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/narration_refiner.py"
click node_image_generation "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/image_gen.py"
click node_animation_tools "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/draw_animation.py"
click node_segmentation_bridge "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/segmentation.py"
click node_audio_tools "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/tts.py"
click node_subtitle_tools "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/subtitle.py"
click node_video_assembly "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/video_gen.py"
click node_run_output "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/output/README.md"
click node_vertexai_flag "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/.env.example"
click node_sam3_service "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/README.md"
click node_sam3_model_core "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/sam3/sam3/model_builder.py"
click node_sam3_ops "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/sam3/sam3/perflib/compile.py"
click node_sam3_train_eval "https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/sam3/sam3/train/train.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_user_input,node_pipeline_orchestrator,node_director_agent,node_research_tools,node_reference_search,node_script_tools,node_image_generation,node_animation_tools,node_segmentation_bridge,node_audio_tools,node_subtitle_tools,node_video_assembly,node_run_output toneBlue
class node_sam3_service,node_sam3_model_core,node_sam3_ops,node_sam3_train_eval toneAmber
class node_genai_api,node_vertexai_flag toneMint
```

## Components

### genai-pipeline

Storyboard AI orchestration and media assembly

- [Prompt & settings](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/pipeline.py): cli input · Interactive topic, language, research, and generation options · `genai-pipeline/pipeline.py`
- [Pipeline](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/pipeline.py): runtime orchestrator · Main CLI flow that coordinates the storyboard generation run · `genai-pipeline/pipeline.py`
- [Director](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/director.py): scene planner · Breaks the prompt into scene-level tasks and delegates work · `genai-pipeline/tools/director.py`
- [Research](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/research.py): info gathering · Fetches background facts and context · `genai-pipeline/tools/research.py`
- [References](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/reference_search.py): reference lookup · Finds supporting images and source material · `genai-pipeline/tools/reference_search.py`
- [Script shaping](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/narration_refiner.py): narration prep · Refines narration and scene prompts · `genai-pipeline/tools/narration_refiner.py`
- [Image gen](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/image_gen.py): visual generation · Creates storyboard visuals and image assets · `genai-pipeline/tools/image_gen.py`
- [Animation](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/draw_animation.py): draw pipeline · Builds sketch motion and segmented drawing sequences · `genai-pipeline/tools/draw_animation.py`
- [Segmentation](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/segmentation.py): sam3 client · Optional callout to remote SAM3-assisted segmentation · `genai-pipeline/tools/segmentation.py`
- [Audio](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/tts.py): tts/transcribe · Synthesizes narration and can transcribe generated audio · `genai-pipeline/tools/tts.py`
- [Subtitles](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/subtitle.py): captioning · Generates and burns subtitles into scene or final videos · `genai-pipeline/tools/subtitle.py`
- [Video assembly](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/tools/video_gen.py): merge stage · Concatenates scenes and merges audio with video · `genai-pipeline/tools/video_gen.py`
- [Run output](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/output/README.md): artifact store · Timestamped run directory holding intermediates and final MP4 · `genai-pipeline/output/README.md`

### sam3-hosting

Optional remote segmentation service and SAM3 model stack

- [SAM3 service](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/README.md): remote inference · Containerized segmentation endpoint used when configured by the pipeline · `sam3-hosting/README.md`
- [SAM3 model](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/sam3/sam3/model_builder.py): vision model stack · Core SAM3 agent, inference, and model implementation · `sam3-hosting/sam3/sam3/model_builder.py`
- [Perf ops](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/sam3/sam3/perflib/compile.py): kernels & ops · Performance-critical mask, NMS, and connected-components primitives · `sam3-hosting/sam3/sam3/perflib/compile.py`
- [Train & eval](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/sam3-hosting/sam3/sam3/train/train.py): offline ml tooling · Training, benchmark, and evaluation infrastructure around SAM3 · `sam3-hosting/sam3/sam3/train/train.py`

### External services

Google GenAI and optional cloud-hosted model endpoints

- **Google GenAI**: llm/image/tts/video api · External planning, image, speech, and optional video generation services
- [Vertex AI](https://github.com/yogendra-yatnalkar/storyboard-ai/blob/main/genai-pipeline/.env.example): deployment mode · Optional Google GenAI backend selected via environment configuration · `genai-pipeline/.env.example`

## Connections

- Prompt & settings → Pipeline: start run · User choices enter the main CLI flow
- Pipeline → Director: plan scenes · The pipeline hands the prompt to the Director Agent
- Director → Research: research · Gather background facts and context
- Director → References: find refs · Locate reference imagery and supporting sources
- Director → Script shaping: shape script · Refine narration and scene prompts
- Director → Image gen: request visuals · Generate storyboard images for scenes
- Director → Animation: build motion · Create sketch animation and drawing passes
- Animation → Segmentation: optional assist · Use SAM3 when segmentation support is configured
- Segmentation → SAM3 service: remote call · Send image-related segmentation requests to the hosted service
- SAM3 service → SAM3 model: infer · Service executes SAM3 model logic
- SAM3 model → Perf ops: use ops · Model relies on optimized kernels and mask primitives
- Director → Audio: narrate · Synthesize or process scene narration audio
- Director → Subtitles: caption · Generate subtitles from script or audio
- Director → Video assembly: assemble · Combine scene assets into video outputs
- Image gen → Video assembly: scene assets · Visuals become inputs to the final assembly stage
- Audio → Video assembly: audio track · Narration is merged into the final render
- Subtitles → Video assembly: subtitle track · Caption files are burned in or attached
- Video assembly → Run output: write artifacts · Store scene intermediates and the final MP4 under the run directory
- Pipeline → Google GenAI: LLM/media · Use Google services for planning and generation
- Google GenAI → Vertex AI: backend mode · Deployment can target Vertex AI or developer API
- SAM3 service → SAM3 model: hosts · Containerized deployment serves the model stack
- Train & eval → SAM3 model: supports · Offline training and evaluation support the model stack