How vishwakneelamegam/openai-video-narrator works
Main Components
• Client/Webpage: The vision.html file handles the live video stream from the user’s browser. It also displays the narration output every 15 frames.
• Server (Backend): The server.py file initiates a FastAPI application (run via uvicorn on port 3000) which handles incoming video data and processes it.
• LLM/Processing Module: The llm.py file likely contains logic to call OpenAI vision for analyzing video frames and generating narration.
• Configuration: The configs.py file stores configuration details like the OpenAI key, domain, vision_prompt, max_frames, and the ngrok tunnel domain.
• External Tunneling: ngrok is used to expose the FastAPI server to the public internet.
• External Service: The OpenAI vision service processes the video frames for narration.
Relationships and Data Flow
• The Client (vision.html) captures and continuously sends video frames (1 frame per second) to the FastAPI server.
• The server (server.py) receives these frames and may perform preliminary processing before handing them off to the LLM module.
• The LLM module (llm.py) then interacts with OpenAI’s vision API (using the credentials and settings from configs.py) to generate narration.
• The processed narration is then returned by the server to be displayed on the vision.html page.
• The configuration module (configs.py) supports overall system settings, ensuring proper keys and parameters are applied.
• ngrok, as an external tunneling service, allows external access to the server by tunneling through port 3000.
Architectural Patterns and Design Principles
• Separation of Concerns: Components are split into configuration (configs.py), processing (llm.py), server logic (server.py), and client display (vision.html).
• API-driven interaction: The use of FastAPI indicates RESTful endpoints that manage the video stream and narration data.
• Modular Design: Each file/module has a specific responsibility which makes it easier to maintain and update (e.g., updating prompts or frame size via configs.py).
Relevant Technologies and Frameworks
• FastAPI and Uvicorn for building and serving the web API.
• OpenAI Vision API for generating video narration.
• ngrok for secure tunneling.
• HTML/JavaScript (embedded in vision.html) for handling the client-side video stream.