jordanrendric/claude-video-vision

1. What this repo is

This is a Claude Code plugin plus MCP server that turns videos into multimodal context for Claude. The plugin is mostly a thin command/skill layer; the real work happens in the Node.js MCP server, which extracts frames, gets audio/transcripts, and packages both for Claude.

2. Top-level boundary

The architecture splits cleanly into:

- Claude-side UX: slash commands, skill prompts, plugin metadata

- MCP server: video ingestion, extraction, backend selection, session/cache handling

- External tooling/services: ffmpeg, optional yt-dlp, and one of several audio backends

3. Claude Code integration

The .claude-plugin, commands/, and skills/ directories define how users invoke the system from Claude Code. /watch-video and /setup-video-vision are the main entry points, and the video-perception skill frames the task as perception rather than interpretation.

4. MCP server as the orchestration core

mcp-server/src/index.ts likely registers the MCP tools and routes requests into the pipeline. The tools under src/tools/ form the public API:

- video_watch for the main analysis flow

- video_analyze for pre-analysis / structure probing

- video_detail for focused follow-up on specific moments

- video_info for metadata-only inspection

- video_configure and video_setup for environment/config management

5. Input normalization and source handling

src/utils/video-source.ts and related utilities indicate the server accepts both local files and YouTube URLs. For YouTube, the server downloads media via yt-dlp and preserves source metadata/captions, so transcript provenance can be tracked later. This is an important boundary: source acquisition happens before extraction, but after user intent is resolved.

6. Frame extraction pipeline

The src/extractors/frames.ts module is the visual path. It uses ffmpeg to sample frames at configurable FPS, time ranges, and resolution, and then converts them into image payloads for Claude. The README’s “adaptive extraction” implies the tool can vary density based on the question, with frame format and resolution controlled by config.

7. Audio/transcript pipeline

Audio processing is abstracted behind src/backends/. There are three implementations:

- gemini-api.ts for cloud speech/event transcription

- openai.ts for OpenAI Whisper API

- local.ts for offline Whisper via whisper.cpp or Python Whisper

The extractor layer (audio.ts, audio-chunker.ts, analyzers.ts) prepares audio segments and normalizes results into a common transcript/event shape.

8. Result composition

The server merges two independently produced streams: base64-encoded frames with timestamps, and audio transcription with timestamps/events. That combined bundle is what Claude receives. The README makes clear the server is not interpreting the video itself; it is providing evidence in a form Claude can reason over.

9. Session and cache boundaries

src/session/manager.ts and src/session/manifest.ts suggest persisted session state for extracted media, cached artifacts, and incremental drill-down. The config also includes retention settings like session_max_age_days and downloads_max_age_days, indicating the system manages storage hygiene over time.

10. Configuration and setup

src/config.ts and video-configure/video-setup manage user settings stored under ~/.claude-video-vision/config.json. The setup wizard verifies dependencies like ffmpeg, yt-dlp, and backend-specific requirements, so runtime behavior depends heavily on local environment readiness.

11. Operational tooling

mcp-server/scripts/ includes token measurement and backend test scripts, which suggests the team is concerned with payload size and backend behavior under real video workloads. The test suite under mcp-server/tests/ covers backends, extractors, sessions, utilities, and tool behavior, showing that the extraction pipeline is the main correctness surface.

12. Deployment model

The MCP server is designed to be installed and run via npx from the published package, while the plugin metadata under .claude-plugin/ and GitHub workflows handle packaging/release. In practice, the runtime architecture is “Claude Code client + local/remote MCP server + local media tools + optional AI APIs.”

13. Key data flow to diagram

A good architecture diagram should show this order:

user command or skill prompt → MCP tool invocation → source resolution/download → frame extraction and audio transcription in parallel → session/cache persistence → combined multimodal payload returned to Claude for reasoning.

14. Main architectural constraint

The critical design choice is that this system is a perception layer, not a summarizer. It preserves raw visual evidence and timestamped audio provenance, then hands that evidence to Claude. That separation is the core boundary around which the repository is organized.