yogendra-yatnalkar/storyboard-ai

1. Repository shape

- This repo is split into two major subsystems:

- genai-pipeline/: the end-to-end Storyboard AI application that turns a prompt into a narrated whiteboard video.

- sam3-hosting/: an optional self-hosted SAM 3 segmentation service and the full SAM3 model codebase.

- The architectural boundary is clear: the pipeline can run without SAM3, but if SAM_API_URL is configured it calls the SAM3 service as a remote dependency.

2. Primary runtime flow

- The main entrypoint is genai-pipeline/pipeline.py, which acts as an interactive CLI orchestrator.

- It collects user choices such as topic, research mode, language, reference-image usage, fast mode, and whether to use Veo.

- From there, the pipeline fans out into specialized tools to research, write, generate visuals, synthesize audio, and assemble the final video.

3. Orchestration layer / Director Agent

- The architectural “brain” is the Director Agent in genai-pipeline/tools/director.py.

- It decomposes one high-level prompt into scene-level tasks and delegates work to tool modules.

- This is the main coordination boundary: downstream modules are mostly specialized workers rather than autonomous controllers.

4. Tool-based multimodal pipeline

- The genai-pipeline/tools/ package is organized as a set of single-purpose stages:

- research and reference lookup: research.py, reference_search.py

- script and prompt shaping: narration_refiner.py, image_prompt_tool.py

- image generation: image_gen.py

- animation and segmentation: draw_animation.py, segmentation.py, divider.py

- audio generation and processing: tts.py, transcribe_audio.py

- subtitle generation and burning: subtitle.py, video_subtitle.py

- video assembly: video_gen.py, concatenate_videos.py, merge_audio_video.py

- These stages form a mostly linear pipeline with optional branches for richer visuals or narration handling.

5. Data flow through the pipeline

- Input starts as a prompt and optional language/research settings.

- The Director produces a scene plan and narration/script.

- Per scene, the system may fetch reference material, generate a custom image, create animation strokes or video segments, synthesize speech, and generate subtitles.

- Final outputs are merged into a single video artifact, while scene-local intermediates are kept for inspection and debugging.

6. SAM3 integration boundary

- SAM3 is treated as an optional external service rather than a hard dependency.

- When enabled, genai-pipeline sends image-related requests to a SAM3 endpoint; when disabled, the pipeline falls back to single-pass drawing.

- This means the whiteboard animation quality can scale from “no segmentation server” to “multi-pass segmentation-assisted drawing” without changing the main pipeline structure.

7. SAM3 hosting subsystem

- sam3-hosting/ contains a deployable FastAPI-style service plus the full SAM3 model package.

- The host includes:

- model inference and agent code under sam3/agent/

- model definitions under sam3/model/

- performance kernels and ops under sam3/perflib/

- evaluation tooling under sam3/eval/

- training and dataset code under sam3/train/

- In the architecture diagram, this should be represented as a separate inference service that the pipeline calls remotely.

8. Deployment and infrastructure

- sam3-hosting/Dockerfile indicates the SAM3 service is containerized.

- The README describes deployment to Google Cloud Run with GPU acceleration, making the model server an externally hosted compute component.

- Environment setup uses .env values such as Google API credentials and optional Hugging Face access for model weights.

9. External services

- The pipeline depends materially on Google’s GenAI stack:

- LLM for planning and script generation

- image generation for line-art visuals

- TTS for narration

- Veo for optional video segments

- The repo also supports GOOGLE_GENAI_USE_VERTEXAI, so the same orchestration can target either Google Developer API or Vertex AI depending on deployment.

10. Persistence and outputs

- Generated artifacts are written under genai-pipeline/output/run_<timestamp>/.

- The main deliverable is storyboard_final_video.mp4.

- Scene folders contain intermediates such as generated images, narration audio, subtitles, masks, and scene-level sketch videos.

- This output layout is important for understanding the pipeline boundary between transient processing and final deliverables.

11. Testing and validation

- genai-pipeline/test_scripts/ contains tool-level and pipeline-level test drivers, including offline Veo/SAM3 checks.

- sam3-hosting/sam3/perflib/tests/ provides low-level correctness checks for segmentation-related ops.

- The presence of these separate test surfaces reflects the repo’s split between orchestration validation and model/inference validation.

12. What is not part of the runtime path

- Much of sam3-hosting/sam3/train/ and sam3-hosting/sam3/eval/ is supporting ML infrastructure, not the default production path.

- For an architecture diagram, treat those directories as offline training/evaluation capabilities around the SAM3 service, not as part of the Storyboard AI request lifecycle.

13. Best diagram abstraction

- A clear architecture diagram would show:

- User prompt / CLI

- Director Agent / pipeline orchestrator

- Parallel or sequential tool stages

- External Google GenAI services

- Optional SAM3 segmentation service

- Output assembly and final MP4

- The key design idea is an agentic multimodal pipeline with an optional remote segmentation accelerator.