How aniketk100/ai-powered-video-captioning-language-conversion-platform works
1. What this repo is
- CaptionFlow is a two-tier video captioning system: a React/Vite client for interactive caption editing and styling, and an Express backend for AI transcription, translation, style conversion, and video export.
- The main architectural boundary is browser UI vs. server-side media/AI processing.
2. Client-side application shell
- The frontend lives under client/ and is organized as a single-page app with Home, Editor, and NotFound pages.
- main.jsx bootstraps the app, while App.jsx likely wires routing and the overall layout.
- Dockerfile, vite.config.js, and tailwind.config.js show the client is built and served as a Vite-based React app with Tailwind styling.
3. Editor is the core UX surface
- The Editor page is the central workflow screen where users upload media, inspect captions, adjust styling, and export results.
- The editor is composed of specialized panels: upload, video preview/player, captions, timeline, style controls, and export modal.
- This is the main place where data from the backend is manipulated and visualized.
4. UI is split by functional surface
- components/player handles video playback, caption overlays, crop overlays, and preview rendering.
- components/timeline handles transport controls, waveform display, and word-level timeline blocks.
- components/captions handles caption segments and individual word badges for fine-grained editing.
- components/style handles template selection, font/color picking, and style preview.
- components/upload handles drag-and-drop ingestion and upload progress feedback.
- components/export contains the export dialog for subtitle and burned-video output.
5. Client state management
- The client uses multiple Zustand stores under client/src/store to separate concerns:
- useProjectStore for project/caption data
- usePlayerStore for playback and playhead state
- useStyleStore for caption appearance settings
- useUIStore for modal/panel/UI interaction state
- This suggests a shared editor state model rather than prop-drilling across many panels.
6. Client service layer
- client/src/services/api.js is the network boundary to the backend.
- ffmpegService.js and waveformService.js imply some local or browser-side media preprocessing/analysis support, used to keep the editor responsive.
- Utilities under client/src/utils handle subtitle parsing/generation (SRT, VTT, ASS), time formatting, language lists, and caption-style presets.
- The client can therefore both consume backend-generated captions and generate exportable subtitle files locally.
7. Backend API layer
- The backend is an Express app in server/src/app.js.
- Routes and controllers are organized around workflow actions: transcription, translation, style conversion, and export.
- The presence of rateLimiter, upload middleware, and errorHandler indicates the API is designed to handle large media uploads and expensive AI/media jobs safely.
8. Backend processing services
- groqService.js is the transcription/LLM integration point, using Groq-hosted models for Whisper transcription and post-processing.
- translationService.js handles language translation, with the README indicating MyMemory and Azure as translation providers/fallbacks.
- ffmpegService.js handles media extraction and rendering tasks such as audio extraction and video export.
- cloudinaryService.js is the optional external storage integration for video assets.
- captionService.js centralizes caption transformation logic, likely shared by transcription, translation, styling, and export flows.
9. Persistence and project data
- server/src/models/Project.js is the MongoDB/Mongoose persistence model for saved project state.
- The architecture implies caption segments, style settings, and source media references are stored as a project record so editing can continue after the initial transcription job.
- MongoDB is therefore the durable system of record, while the upload directory is temporary workspace.
10. Media ingestion and temporary storage
- Incoming uploads pass through server/src/middleware/upload.js and are written into server/uploads/.
- cleanupTemp.js indicates temporary media and generated artifacts are explicitly cleaned up after processing.
- This is an important boundary: large binaries are handled on the server filesystem during processing, but only project metadata and final outputs are persisted or returned.
11. Primary data flow
- User uploads a video in the client.
- The browser sends the file to the backend transcription endpoint.
- The backend extracts audio with FFmpeg, sends it to Groq/Whisper, post-processes captions, and stores results in MongoDB.
- The client then loads the project into the editor, where the user edits text, timing, style, and layout through the various panels.
- Updated project data is saved back to the backend, then exported as subtitle files or as a burned-in video.
12. Export paths
- There are two distinct export modes:
- Text subtitle export: SRT, VTT, and ASS generation can happen through client utilities.
- Video export: server-side FFmpeg rendering burns subtitles into the video, which is more compute-heavy and stays behind the backend boundary.
- The export modal is the UX entry point; the actual rendering logic lives on the server.
13. Infrastructure and deployment shape
- The repo is dockerized at both service levels: client/Dockerfile, server/Dockerfile, and a top-level docker-compose.yml.
- This suggests a locally orchestrated stack with at least three runtime concerns: frontend dev server/build, backend API, and MongoDB.
- External services materially affecting the architecture are Groq, MongoDB, optional Cloudinary, and translation providers.
14. How to diagram it
- Draw the client as a stateful editor UI with multiple panels fed by shared project/player/style stores.
- Draw the server as a processing API with distinct media/AI services around a MongoDB project model and temporary upload storage.
- Show the main arrows as: upload media → transcribe/process → persist project → edit UI updates → translate/style/export → subtitle files or rendered video.
Main entry points: app.js API app · Editor.jsx Editor · transcribeController.js Transcribe API · ffmpegService.js FFmpeg service
Components 30
Client
React/Vite editor UI and shared state
- App shell · Bootstraps routing and top-level layout
client/src/App.jsx - Home · Entry page for the application
client/src/pages/Home.jsx - Editor · Main caption editing and export workflow
client/src/pages/Editor.jsx - Player · Video playback, overlays, and preview rendering
client/src/components/player/VideoPlayer.jsx - Timeline · Transport controls, waveform, and word-level timing
client/src/components/timeline/TimelinePanel.jsx - Captions · Segments and word badges for caption refinement
client/src/components/captions/CaptionPanel.jsx - Style · Templates, colors, and caption styling controls
client/src/components/style/StylePanel.jsx - Upload · Drag-and-drop media ingestion and upload progress
client/src/components/upload/UploadZone.jsx - Export · Subtitle and rendered-video export entry point
client/src/components/export/ExportModal.jsx - Project store · Shared project and caption data
client/src/store/useProjectStore.js - Player store · Playback and playhead state
client/src/store/usePlayerStore.js - Style store · Caption appearance settings
client/src/store/useStyleStore.js - UI store · Modal and panel interaction state
client/src/store/useUIStore.js - API client · Frontend service layer for backend requests
client/src/services/api.js - Subtitle utils · SRT, VTT, ASS generation and parsing helpers
client/src/utils/generateSRT.js
Server
Express API and media/AI processing
- API app · Backend application bootstrap and routing
server/src/app.js - Transcribe API · Handles transcription workflow requests
server/src/controllers/transcribeController.js - Translate API · Handles language conversion requests
server/src/controllers/translateController.js - Style API · Handles caption style conversion requests
server/src/controllers/styleController.js - Export API · Handles server-side video export requests
server/src/controllers/exportController.js - Caption service · Central caption transformation and normalization
server/src/services/captionService.js - Groq service · Whisper transcription and post-processing integration
server/src/services/groqService.js - Translation service · Translation orchestration and fallback providers
server/src/services/translationService.js - FFmpeg service · Audio extraction and video rendering tasks
server/src/services/ffmpegService.js - Cloudinary service · Optional external video asset storage
server/src/services/cloudinaryService.js - Project model · Persistent project, caption, and style state
server/src/models/Project.js - Upload middleware · Handles incoming media uploads
server/src/middleware/upload.js - Rate limit · Protects expensive API endpoints
server/src/middleware/rateLimiter.js - Error handler · Central API error handling
server/src/middleware/errorHandler.js - Temp cleanup · Removes temporary media and generated artifacts
server/src/utils/cleanupTemp.js
Connections 35
- App shell → Home: routes to · Landing experience
- App shell → Editor: routes to · Primary workflow screen
- Editor → Upload: uses · Media ingestion
- Editor → Player: uses · Preview and overlays
- Editor → Timeline: uses · Timing controls
- Editor → Captions: uses · Caption editing
- Editor → Style: uses · Appearance controls
- Editor → Export: uses · Export workflow
- Player → Player store: reads state · Playback and playhead
- Timeline → Player store: reads state · Shared transport state
- Captions → Project store: edits · Caption segments and project data
- Style → Style store: edits · Caption styling settings
- Upload → API client: uploads via · Sends source media to backend
- Export → API client: requests · Triggers export jobs
- API client → API app: calls · Frontend-to-backend boundary
- API app → Rate limit: protects with · Limits expensive requests
- API app → Upload middleware: accepts media through · Upload handling
- API app → Error handler: handles errors with · Central failure path
- API app → Transcribe API: routes to · Transcription endpoint
- API app → Translate API: routes to · Translation endpoint
- API app → Style API: routes to · Style conversion endpoint
- API app → Export API: routes to · Export endpoint
- Transcribe API → FFmpeg service: uses · Extracts audio from media
- Transcribe API → Groq service: uses · AI transcription
- Transcribe API → Caption service: uses · Normalizes caption output
- Translate API → Translation service: uses · Language conversion providers
- Style API → Caption service: uses · Style transformation logic
- Export API → FFmpeg service: uses · Server-side rendering
- Transcribe API → Project model: persists · Stores project state
- Translate API → Project model: persists · Updates translated captions
- Style API → Project model: persists · Stores style settings
- Export API → Project model: reads · Uses project data for rendering
- FFmpeg service → Temp cleanup: cleans up with · Removes temporary artifacts
- FFmpeg service → Cloudinary service: optionally uses · External asset storage
- Project store → Subtitle utils: supports export · Client-side subtitle file generation