oxbshw/watch-skill

Purpose and Entry Points

Watch Skill is a Python 3.11+ video-memory and agent-verification engine. It converts media, streams, meetings, and screen recordings into a persistent timestamped evidence index, then supports retrieval, answering, extraction, and self-verification loops.

Primary surfaces share the core: CLI (src/watch_skill/surfaces/cli/main.py), stdio MCP server (src/watch_skill/surfaces/mcp/server.py), REST application (src/watch_skill/surfaces/api/app.py), and framework adapters (src/watch_skill/integrations/_core.py).

1. Orchestration and Shared Contracts

The top-level watch workflow coordinates acquisition, perception, transcription, indexing, and reporting. Batch and background-style work are represented separately.

• src/watch_skill/watch.py

• src/watch_skill/jobs.py

• src/watch_skill/batch.py

Configuration and explicit domain errors form cross-cutting boundaries for storage, privacy, providers, budgets, and user-facing repair paths.

• src/watch_skill/config.py

• src/watch_skill/errors.py

2. Media Acquisition Boundary

Acquisition normalizes heterogeneous inputs—local/direct media, supported hosted sources, and streams—into common media/source types. Resolver logic selects download mechanisms; cache prevents unnecessary reacquisition.

• src/watch_skill/acquire/resolver.py

• src/watch_skill/acquire/sources.py

• src/watch_skill/acquire/cache.py

External runtime dependencies include download tooling such as yt-dlp and Cobalt-compatible acquisition, represented by src/watch_skill/acquire/ytdlp.py and src/watch_skill/acquire/cobalt.py.

3. Perception and Transcription Pipeline

Media is processed into scene-aware frames, OCR evidence, and timestamped transcript segments. Perception uses budget controls and scene/deduplication logic before costly visual interpretation.

• src/watch_skill/perceive/engine.py

• src/watch_skill/perceive/scenes.py

• src/watch_skill/perceive/ocr_backends.py

Transcription follows a provider ladder: captions/VTT and local audio processing first, with optional cloud speech services and diarization.

• src/watch_skill/transcribe/ladder.py

• src/watch_skill/transcribe/local.py

• src/watch_skill/transcribe/cloud.py

4. Persistent Evidence Index and Retrieval

The index persists videos, frames, transcripts, OCR, embeddings, and usage data. Retrieval combines full-text and vector search, with multilingual normalization as a core retrieval concern.

• src/watch_skill/index/db.py

• src/watch_skill/index/store.py

• src/watch_skill/index/retrieval.py

Embedding generation is isolated from storage/query logic, allowing provider/model evolution without changing the evidence model.

• src/watch_skill/index/embeddings.py

• src/watch_skill/index/textnorm.py

5. Evidence-Grounded Answering and Derived Knowledge

The answer engine retrieves indexed evidence, applies a cost/quality ladder, localizes supporting moments, and returns confidence-aware cited answers. Crops provide focused visual evidence when needed.

• src/watch_skill/answer/engine.py

• src/watch_skill/answer/ladder.py

• src/watch_skill/answer/localize.py

Structured extraction and cross-video library synthesis consume the same index rather than reprocessing source media.

• src/watch_skill/extract/chapters.py

• src/watch_skill/library/synthesize.py

Lessons capture reported mistakes, classify applicability, inject corrections into later work, and support replayable evaluations.

• src/watch_skill/lessons/store.py

• src/watch_skill/lessons/inject.py

6. Vision Provider Boundary

Visual QA is provider-pluggable behind a client/model registry, with pricing and cost accounting separated from pipeline behavior. It supports hosted vendors, OpenAI-compatible endpoints, and local Ollama-style operation.

• src/watch_skill/vision/client.py

• src/watch_skill/vision/registry.py

• src/watch_skill/vision/cost.py

7. THE LOOP Verification Subsystem

THE LOOP is a parallel capture → critique → iteration → proof flow for browser, desktop/window, generated-video, game, and monitored-stream work. Capture artifacts and diffs are evaluated against plain-language criteria; results can be indexed and rendered for review.

• src/watch_skill/loop/runner.py

• src/watch_skill/loop/capture.py

• src/watch_skill/loop/critic.py

Reporting/viewing is an output boundary: standalone evidence reports and loop viewers package artifacts for sharing without requiring a live service.

• src/watch_skill/viewer.py

• src/watch_skill/loop/viewer.py

Runtime and Infrastructure

The package runs locally via Python/uv/pipx, optionally in Docker (Dockerfile). Local-first processing uses media/OCR/transcription binaries and models; health commands diagnose and bootstrap these dependencies.

• src/watch_skill/health/doctor.py

• src/watch_skill/health/binaries.py

Persistent local storage is central to reuse and offline workflows. Network calls are limited to acquisition sources and opt-in vision/cloud-transcription providers; MCP uses stdio, while REST is an optional HTTP service.