# jordanrendric/claude-video-vision architecture

> Architecture diagram of the GitHub repository jordanrendric/claude-video-vision, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jordanrendric/claude-video-vision
- Repository: https://github.com/jordanrendric/claude-video-vision

## Overview

1. **What this repo is**
   This is a Claude Code plugin plus MCP server that turns videos into multimodal context for Claude. The plugin is mostly a thin command/skill layer; the real work happens in the Node.js MCP server, which extracts frames, gets audio/transcripts, and packages both for Claude.

2. **Top-level boundary**
   The architecture splits cleanly into:
   - **Claude-side UX**: slash commands, skill prompts, plugin metadata
   - **MCP server**: video ingestion, extraction, backend selection, session/cache handling
   - **External tooling/services**: `ffmpeg`, optional `yt-dlp`, and one of several audio backends

3. **Claude Code integration**
   The `.claude-plugin`, `commands/`, and `skills/` directories define how users invoke the system from Claude Code. `/watch-video` and `/setup-video-vision` are the main entry points, and the `video-perception` skill frames the task as perception rather than interpretation.

4. **MCP server as the orchestration core**
   `mcp-server/src/index.ts` likely registers the MCP tools and routes requests into the pipeline. The tools under `src/tools/` form the public API:
   - `video_watch` for the main analysis flow
   - `video_analyze` for pre-analysis / structure probing
   - `video_detail` for focused follow-up on specific moments
   - `video_info` for metadata-only inspection
   - `video_configure` and `video_setup` for environment/config management

5. **Input normalization and source handling**
   `src/utils/video-source.ts` and related utilities indicate the server accepts both local files and YouTube URLs. For YouTube, the server downloads media via `yt-dlp` and preserves source metadata/captions, so transcript provenance can be tracked later. This is an important boundary: source acquisition happens before extraction, but after user intent is resolved.

6. **Frame extraction pipeline**
   The `src/extractors/frames.ts` module is the visual path. It uses `ffmpeg` to sample frames at configurable FPS, time ranges, and resolution, and then converts them into image payloads for Claude. The README’s “adaptive extraction” implies the tool can vary density based on the question, with frame format and resolution controlled by config.

7. **Audio/transcript pipeline**
   Audio processing is abstracted behind `src/backends/`. There are three implementations:
   - `gemini-api.ts` for cloud speech/event transcription
   - `openai.ts` for OpenAI Whisper API
   - `local.ts` for offline Whisper via `whisper.cpp` or Python Whisper
   The extractor layer (`audio.ts`, `audio-chunker.ts`, `analyzers.ts`) prepares audio segments and normalizes results into a common transcript/event shape.

8. **Result composition**
   The server merges two independently produced streams: base64-encoded frames with timestamps, and audio transcription with timestamps/events. That combined bundle is what Claude receives. The README makes clear the server is not interpreting the video itself; it is providing evidence in a form Claude can reason over.

9. **Session and cache boundaries**
   `src/session/manager.ts` and `src/session/manifest.ts` suggest persisted session state for extracted media, cached artifacts, and incremental drill-down. The config also includes retention settings like `session_max_age_days` and `downloads_max_age_days`, indicating the system manages storage hygiene over time.

10. **Configuration and setup**
   `src/config.ts` and `video-configure`/`video-setup` manage user settings stored under `~/.claude-video-vision/config.json`. The setup wizard verifies dependencies like `ffmpeg`, `yt-dlp`, and backend-specific requirements, so runtime behavior depends heavily on local environment readiness.

11. **Operational tooling**
   `mcp-server/scripts/` includes token measurement and backend test scripts, which suggests the team is concerned with payload size and backend behavior under real video workloads. The test suite under `mcp-server/tests/` covers backends, extractors, sessions, utilities, and tool behavior, showing that the extraction pipeline is the main correctness surface.

12. **Deployment model**
   The MCP server is designed to be installed and run via `npx` from the published package, while the plugin metadata under `.claude-plugin/` and GitHub workflows handle packaging/release. In practice, the runtime architecture is “Claude Code client + local/remote MCP server + local media tools + optional AI APIs.”

13. **Key data flow to diagram**
   A good architecture diagram should show this order:
   user command or skill prompt → MCP tool invocation → source resolution/download → frame extraction and audio transcription in parallel → session/cache persistence → combined multimodal payload returned to Claude for reasoning.

14. **Main architectural constraint**
   The critical design choice is that this system is a **perception layer**, not a summarizer. It preserves raw visual evidence and timestamped audio provenance, then hands that evidence to Claude. That separation is the core boundary around which the repository is organized.

Main entry points: [`mcp-server/src/index.ts`](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/index.ts) (MCP entry), [`mcp-server/src/tools/video-watch.ts`](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-watch.ts) (Watch tool), [`commands/watch-video.md`](https://github.com/jordanrendric/claude-video-vision/blob/main/commands/watch-video.md) (Watch video), [`.claude-plugin`](https://github.com/jordanrendric/claude-video-vision/blob/main/.claude-plugin) (.claude-plugin)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_claude_ux["Claude UX"]
  node_plugin_meta[".claude-plugin<br/>plugin manifest<br/>[.claude-plugin]"]
  node_watch_command["Watch video<br/>slash command<br/>[watch-video.md]"]
  node_setup_command["Setup video<br/>slash command"]
  node_video_skill["Video perception<br/>skill prompt<br/>[SKILL.md]"]
end

subgraph group_mcp_server["MCP Server"]
  node_mcp_entry["MCP entry<br/>server bootstrap<br/>[index.ts]"]
  node_video_watch_tool["Watch tool<br/>analysis tool<br/>[video-watch.ts]"]
  node_video_analyze_tool["Analyze tool<br/>analysis tool<br/>[video-analyze.ts]"]
  node_video_detail_tool["Detail tool<br/>follow-up tool<br/>[video-detail.ts]"]
  node_video_info_tool["Info tool<br/>metadata tool<br/>[video-info.ts]"]
  node_video_setup_tool["Setup tool<br/>config tool<br/>[video-setup.ts]"]
  node_video_config_tool["Configure tool<br/>config tool<br/>[video-configure.ts]"]
  node_source_utils["Source resolver<br/>input normalization<br/>[video-source.ts]"]
  node_frame_extractor["Frame extractor<br/>visual pipeline<br/>[frames.ts]"]
  node_audio_pipeline["Audio pipeline<br/>[audio.ts]"]
  node_audio_backends["Audio backends<br/>backend adapters<br/>[gemini-api.ts]"]
  node_session_store[("Session store<br/>state cache<br/>[manager.ts]")]
  node_manifest["Manifest<br/>session record<br/>[manifest.ts]"]
  node_config_core["Config core<br/>runtime config<br/>[config.ts]"]
end

subgraph group_external["External Tools"]
  node_ffmpeg["ffmpeg<br/>local binary"]
  node_ytdlp["yt-dlp<br/>download tool"]
  node_speech_apis(("Speech APIs<br/>external services"))
end

subgraph group_ops["Ops & Release"]
  node_release_ci["Release CI<br/>workflow<br/>[release.yml]"]
end

node_plugin_meta -->|"installs"| node_watch_command
node_plugin_meta -->|"installs"| node_setup_command
node_watch_command -->|"guides"| node_video_skill
node_setup_command -->|"invokes"| node_video_setup_tool
node_watch_command -->|"calls"| node_mcp_entry
node_mcp_entry -->|"routes"| node_video_watch_tool
node_mcp_entry -->|"routes"| node_video_analyze_tool
node_mcp_entry -->|"routes"| node_video_detail_tool
node_mcp_entry -->|"routes"| node_video_info_tool
node_mcp_entry -->|"routes"| node_video_config_tool
node_mcp_entry -->|"routes"| node_video_setup_tool
node_video_watch_tool -->|"normalizes"| node_source_utils
node_video_watch_tool -->|"extracts"| node_frame_extractor
node_video_watch_tool -->|"extracts"| node_audio_pipeline
node_source_utils -->|"downloads"| node_ytdlp
node_frame_extractor -->|"uses"| node_ffmpeg
node_audio_pipeline -->|"delegates"| node_audio_backends
node_audio_backends -.->|"optionally calls"| node_speech_apis
node_video_watch_tool -->|"persists"| node_session_store
node_session_store -->|"records"| node_manifest
node_video_config_tool -->|"reads/writes"| node_config_core
node_video_setup_tool -->|"verifies"| node_config_core
node_release_ci -->|"packages"| node_plugin_meta
node_release_ci -->|"ships"| node_mcp_entry

click node_plugin_meta "https://github.com/jordanrendric/claude-video-vision/blob/main/.claude-plugin"
click node_watch_command "https://github.com/jordanrendric/claude-video-vision/blob/main/commands/watch-video.md"
click node_setup_command "https://github.com/jordanrendric/claude-video-vision/blob/main/commands/setup-video-vision.md"
click node_video_skill "https://github.com/jordanrendric/claude-video-vision/blob/main/skills/video-perception/SKILL.md"
click node_mcp_entry "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/index.ts"
click node_video_watch_tool "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-watch.ts"
click node_video_analyze_tool "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-analyze.ts"
click node_video_detail_tool "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-detail.ts"
click node_video_info_tool "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-info.ts"
click node_video_setup_tool "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-setup.ts"
click node_video_config_tool "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-configure.ts"
click node_source_utils "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/utils/video-source.ts"
click node_frame_extractor "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/extractors/frames.ts"
click node_audio_pipeline "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/extractors/audio.ts"
click node_audio_backends "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/backends/gemini-api.ts"
click node_session_store "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/session/manager.ts"
click node_manifest "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/session/manifest.ts"
click node_config_core "https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/config.ts"
click node_release_ci "https://github.com/jordanrendric/claude-video-vision/blob/main/.github/workflows/release.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_plugin_meta,node_watch_command,node_setup_command,node_video_skill toneBlue
class node_mcp_entry,node_video_watch_tool,node_video_analyze_tool,node_video_detail_tool,node_video_info_tool,node_video_setup_tool,node_video_config_tool,node_source_utils,node_frame_extractor,node_audio_pipeline,node_audio_backends,node_session_store,node_manifest,node_config_core toneAmber
class node_ffmpeg,node_ytdlp,node_speech_apis toneMint
class node_release_ci toneRose
```

## Components

### Claude UX

Claude Code-facing commands, skills, and plugin metadata.

- [.claude-plugin](https://github.com/jordanrendric/claude-video-vision/blob/main/.claude-plugin): plugin manifest · Marketplace and plugin metadata for Claude Code installation. · `.claude-plugin`
- [Watch video](https://github.com/jordanrendric/claude-video-vision/blob/main/commands/watch-video.md): slash command · Main user entry point for video analysis. · `commands/watch-video.md`
- [Setup video](https://github.com/jordanrendric/claude-video-vision/blob/main/commands/setup-video-vision.md): slash command · Guides environment setup and dependency checks. · `commands/setup-video-vision.md`
- [Video perception](https://github.com/jordanrendric/claude-video-vision/blob/main/skills/video-perception/SKILL.md): skill prompt · Frames the task as perception and evidence gathering. · `skills/video-perception/SKILL.md`

### MCP Server

Node.js server that orchestrates video ingestion, extraction, and response composition.

- [MCP entry](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/index.ts): server bootstrap · Registers tools and routes MCP requests into the pipeline. · `mcp-server/src/index.ts`
- [Watch tool](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-watch.ts): analysis tool · Primary end-to-end video analysis flow. · `mcp-server/src/tools/video-watch.ts`
- [Analyze tool](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-analyze.ts): analysis tool · Pre-analysis and structure probing for a video. · `mcp-server/src/tools/video-analyze.ts`
- [Detail tool](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-detail.ts): follow-up tool · Focused drill-down on specific moments or segments. · `mcp-server/src/tools/video-detail.ts`
- [Info tool](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-info.ts): metadata tool · Metadata-only inspection of a source video. · `mcp-server/src/tools/video-info.ts`
- [Setup tool](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-setup.ts): config tool · Performs environment setup and dependency validation. · `mcp-server/src/tools/video-setup.ts`
- [Configure tool](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/tools/video-configure.ts): config tool · Reads and updates runtime configuration. · `mcp-server/src/tools/video-configure.ts`
- [Source resolver](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/utils/video-source.ts): input normalization · Accepts local files and YouTube URLs, and prepares source metadata. · `mcp-server/src/utils/video-source.ts`
- [Frame extractor](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/extractors/frames.ts): visual pipeline · Uses ffmpeg to sample frames with configurable density and resolution. · `mcp-server/src/extractors/frames.ts`
- [Audio pipeline](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/extractors/audio.ts): audio pipeline · Chunks and normalizes audio for transcription and event extraction. · `mcp-server/src/extractors/audio.ts`
- [Audio backends](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/backends/gemini-api.ts): backend adapters · Pluggable transcription backends for Gemini, OpenAI, or local Whisper. · `mcp-server/src/backends/gemini-api.ts`
- [Session store](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/session/manager.ts): state cache · Persists session manifests, cached artifacts, and drill-down state. · `mcp-server/src/session/manager.ts`
- [Manifest](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/session/manifest.ts): session record · Describes extracted media, provenance, and cached results. · `mcp-server/src/session/manifest.ts`
- [Config core](https://github.com/jordanrendric/claude-video-vision/blob/main/mcp-server/src/config.ts): runtime config · Loads user settings, retention policy, and backend selection. · `mcp-server/src/config.ts`

### External Tools

Local binaries and optional remote APIs used during processing.

- **ffmpeg**: local binary · Samples and converts video frames and audio.
- **yt-dlp**: download tool · Downloads remote video sources and preserves source metadata.
- **Speech APIs**: external services · Optional remote transcription services used by audio backends.

### Ops & Release

Packaging, workflows, and maintenance scripts around the core runtime.

- [Release CI](https://github.com/jordanrendric/claude-video-vision/blob/main/.github/workflows/release.yml): workflow · Build and publish automation for the package and plugin. · `.github/workflows/release.yml`

## Connections

- .claude-plugin → Watch video: installs · Plugin metadata exposes Claude Code actions.
- .claude-plugin → Setup video: installs · Plugin metadata exposes setup actions.
- Watch video → Video perception: guides · The command invokes the perception-oriented skill.
- Setup video → Setup tool: invokes · Setup command maps to MCP environment setup.
- Watch video → MCP entry: calls · User intent reaches the MCP server through the main tool path.
- MCP entry → Watch tool: routes · Server registers the primary analysis tool.
- MCP entry → Analyze tool: routes · Server registers the pre-analysis tool.
- MCP entry → Detail tool: routes · Server registers the drill-down tool.
- MCP entry → Info tool: routes · Server registers the metadata tool.
- MCP entry → Configure tool: routes · Server registers the config tool.
- MCP entry → Setup tool: routes · Server registers the setup tool.
- Watch tool → Source resolver: normalizes · The watch flow resolves local files or remote URLs first.
- Watch tool → Frame extractor: extracts · Visual evidence is sampled into timestamped frames.
- Watch tool → Audio pipeline: extracts · Audio is chunked and prepared for transcription.
- Source resolver → yt-dlp: downloads · Remote sources are fetched through yt-dlp when needed.
- Frame extractor → ffmpeg: uses · Frame sampling depends on ffmpeg.
- Audio pipeline → Audio backends: delegates · Chunked audio is sent to a selectable backend.
- Audio backends → Speech APIs: optionally calls · Cloud backends use remote speech services.
- Watch tool → Session store: persists · Extracted artifacts and incremental state are cached.
- Session store → Manifest: records · Session manager stores manifests for later reuse.
- Configure tool → Config core: reads/writes · Configuration changes flow through the config core.
- Setup tool → Config core: verifies · Setup validates environment and backend prerequisites.
- Release CI → .claude-plugin: packages · Release automation publishes the plugin metadata.
- Release CI → MCP entry: ships · Release automation packages the MCP server.