How balajiraoiim/openshorts works
1. What this repo is
- OpenShorts.app is a self-hosted, full-stack AI video platform.
- The architecture centers on a Python backend that runs media/AI pipelines and a React dashboard that operators use to trigger jobs, inspect results, and publish content.
2. Top-level split: backend vs. dashboard
- The dashboard/ directory is a separate Vite + React app with its own Dockerfile and package lock, so the UI is independently built and served.
- The Python files at the repo root (main.py, app.py, editor.py, saasshorts.py, etc.) form the backend/service layer and pipeline modules.
3. User-facing subsystems
- The dashboard is organized around three major product areas reflected in the UI components:
- Clip Generator
- AI Shorts / UGC videos
- YouTube Studio
- There is also a public gallery experience for generated videos and avatars, with SEO-oriented pages.
4. Frontend architecture
- The dashboard is a single-page app built with React 18, Vite, and Tailwind CSS.
- Key component boundaries in dashboard/src/components/ suggest feature-specific UI modules rather than a monolithic screen:
- media ingestion: MediaInput, KeyInput
- generation flows: SaaShortsTab, ThumbnailStudio
- content editing and overlays: HookModal, SubtitleModal, TranslateModal
- results and browsing: ResultCard, Gallery, UGCGallery, GalleryCard
- processing feedback: ProcessingAnimation
- This UI is likely a thin orchestration layer over backend jobs rather than doing heavy processing in-browser.
5. Backend orchestration layer
- The backend appears to be the system of record for job creation, processing, and asset management.
- main.py and app.py are the likely server entrypoints; the README says the backend serves both APIs and SEO/public pages.
- The backend is responsible for coordinating pipeline steps, storing results, and exposing them to the dashboard and gallery pages.
6. Clip Generator pipeline
- The README describes a pipeline that transforms long videos or uploads into shorts:
- ingest from YouTube or local file
- transcribe with Whisper
- detect scenes with PySceneDetect
- analyze moments with Gemini
- cut clips with FFmpeg
- reframe vertically using face/subject tracking
- add subtitles, hook text, and effects
- back up and publish outputs
- This subsystem is media-processing heavy and likely runs as async jobs rather than request/response work.
7. AI Shorts / UGC video pipeline
- saasshorts.py and related modules likely implement the second major generation flow: marketing-style UGC videos from a product URL or description.
- The pipeline spans analysis, script generation, avatar creation, voiceover, talking-head generation, b-roll generation, compositing, and final publishing.
- This is a separate domain from clip extraction: it synthesizes a new video from text, product context, and AI-generated assets.
8. YouTube Studio subsystem
- The “studio” feature is a third pipeline focused on creator utilities:
- title generation
- thumbnail generation
- description/chapter generation
- direct publishing to YouTube
- The presence of thumbnail.py, translate.py, and editor.py suggests specialized post-processing modules that support this studio workflow.
9. Shared media and storage boundary
- The architecture distinguishes between transient local processing and durable storage.
- s3_uploader.py plus the README’s S3 setup indicate two buckets/boundaries:
- a private bucket for generated clips and working outputs
- a public bucket for gallery assets and avatars
- The gallery pages depend on these public assets being directly reachable, while job outputs can remain private until published.
10. External AI/service integrations
- The core system depends on multiple external services that define major architectural boundaries:
- Google Gemini for analysis, script generation, and viral-moment detection
- fal.ai for image/video/actor generation
- ElevenLabs for voiceover and dubbing
- Upload-Post for social publishing to TikTok, Instagram, and YouTube
- These integrations are not optional implementation details; they materially shape the end-to-end workflow and cost model.
11. Publishing and distribution
- After generation, content can be distributed to social platforms through the Upload-Post integration.
- The platform supports direct publishing and scheduled publishing, so the backend likely manages a publish queue or deferred jobs separate from the media creation pipeline.
12. Job control and operational concerns
- The README mentions semaphore-based concurrency (MAX_CONCURRENT_JOBS) and automatic cleanup, implying background job management is a first-class part of the backend design.
- This suggests the backend is built to queue and throttle expensive media/AI jobs, rather than executing them synchronously in API handlers.
13. Validation and quality gates
- Files like verify_aesthetic.py, verify_custom_hook.py, and verify_hooks.py indicate post-generation validation checks.
- These likely act as quality gates for generated assets before they are surfaced in the UI, stored publicly, or published externally.
14. Deployment shape
- The repo is containerized with a root Dockerfile, a docker-compose.yml, and a separate dashboard/Dockerfile.
- That implies at least two deployable services: the backend service and the frontend dashboard service, with shared environment configuration via .env.
15. Diagram-friendly mental model
- Draw this as: Dashboard UI → Backend API/orchestrator → AI/media pipelines → storage/public gallery/publishing services.
- The important boundaries are:
- browser vs. server
- request handling vs. async job execution
- transient processing vs. durable S3/public assets
- internal generation vs. external AI and social APIs
Main entry points: App.jsx Dashboard App · main.py Main Service · saasshorts.py UGC Pipeline · s3_uploader.py S3 Uploader
Components 18
Dashboard UI
Separate React/Vite frontend for operators and browsing
- Dashboard App · Operator UI for generation, inspection, and publishing
dashboard/src/App.jsx - Media Input · Collects source video or URL inputs
dashboard/src/components/MediaInput.jsx - Shorts Tab · Main clip-generation workflow panel
dashboard/src/components/SaaShortsTab.jsx - Thumbnail Studio · YouTube studio tools for titles and thumbnails
dashboard/src/components/ThumbnailStudio.jsx - Gallery View · Browsing surface for generated videos
dashboard/src/components/Gallery.jsx - UGC Gallery · Public gallery for avatars and generated marketing videos
dashboard/src/components/UGCGallery.jsx - Processing UI · Generation progress and result presentation
dashboard/src/components/ProcessingAnimation.jsx
Python Backend
Server, orchestration, and media pipeline modules
- Backend API · HTTP entrypoint and request orchestration
app.py - Main Service · Primary backend runtime and job control
main.py - Clip Pipeline · Transforms long-form video into shorts
- UGC Pipeline · Generates marketing-style videos from product context
saasshorts.py - Studio Tools · Titles, thumbnails, translation, and editor support
thumbnail.py - Quality Gates · Post-generation checks before publish or gallery exposure
verify_hooks.py - S3 Uploader · Moves working outputs and public assets into buckets
s3_uploader.py
External Services
Third-party AI, storage, and publishing boundaries
- Gemini · Analysis, scripting, and moment detection
- fal.ai · Image, avatar, and video generation
- ElevenLabs · Voiceover and dubbing generation
- Upload-Post · Social distribution to YouTube, TikTok, and Instagram
Connections 24
- Dashboard App → Media Input · Collects source inputs
- Dashboard App → Shorts Tab · Launches clip generation
- Dashboard App → Thumbnail Studio · Opens creator tools
- Dashboard App → Gallery View · Browses generated clips
- Dashboard App → UGC Gallery · Browses public UGC assets
- Dashboard App → Processing UI · Shows job progress and results
- Shorts Tab → Backend API · Submits generation jobs
- Thumbnail Studio → Backend API · Requests titles, thumbnails, and edits
- Gallery View → Main Service · Fetches gallery data
- UGC Gallery → S3 Uploader · Loads public assets
- Backend API → Main Service · Delegates request handling
- Main Service → Clip Pipeline · Runs shorts workflow
- Main Service → UGC Pipeline · Runs UGC generation workflow
- Main Service → Studio Tools · Invokes studio utilities
- Main Service → Quality Gates · Checks generated outputs
- Clip Pipeline → Gemini · Analyzes content and moments
- Clip Pipeline → S3 Uploader · Stores clips and artifacts
- UGC Pipeline → Gemini · Generates scripts and analysis
- UGC Pipeline → fal.ai · Creates avatars and video assets
- UGC Pipeline → ElevenLabs · Generates narration
- UGC Pipeline → S3 Uploader · Uploads final outputs
- Studio Tools → Upload-Post · Publishes or schedules content
- Quality Gates → S3 Uploader · Approves assets for durable storage
- S3 Uploader → Upload-Post · Supplies publishable media