browser-use/video-use

1. What this repo is

- video-use is not a conventional app; it is an agent-oriented video editing skill. The primary “runtime” is a Claude Code–style agent with shell access that reads the repo docs, runs helper scripts, and iterates on edits.

2. Core control plane

- The human gives a folder of raw footage and a goal.

- The agent follows SKILL.md and install.md, inventories the sources, proposes an edit strategy, then drives the rest of the pipeline.

- The repo is designed so outputs stay in <videos_dir>/edit/, while the skill repo itself remains clean.

3. Input/output contract

- Inputs: raw video/audio takes in a working folder, plus configuration via .env and the agent session.

- Outputs: edited media artifacts, especially final.mp4, plus intermediate transcripts, packed notes, render previews, and session memory like project.md.

4. Transcription subsystem

- helpers/transcribe.py and helpers/transcribe_batch.py form the ingestion layer for source media.

- The README indicates these rely on ElevenLabs Scribe to produce word-level timestamps, speaker diarization, and audio events like laughter or applause.

- This is the foundation for all cut decisions: the LLM reasons over text, not raw frames.

5. Transcript packing layer

- helpers/pack_transcripts.py compresses the per-take transcript output into a compact markdown view such as takes_packed.md.

- This packed transcript is the agent’s main reading surface and is intentionally small enough to support precise editing decisions without frame-by-frame inspection.

6. On-demand visual inspection

- helpers/timeline_view.py generates a composite visual for selected time ranges: filmstrip, waveform, speaker/word labels, and cut candidates.

- This is used only at decision points or sanity checks, so visual data acts as a targeted supplement to transcript reasoning rather than the primary representation.

7. Render and finishing pipeline

- helpers/render.py is the execution boundary that turns the chosen edit decisions into actual media output.

- The README highlights the main finishing operations: filler-word removal, dead-space cuts, 30 ms audio fades, subtitle burn-in, color grading, and overlay insertion.

- helpers/grade.py suggests grading is treated as a separate concern, likely reusable across segments or styles.

8. Self-evaluation loop

- After rendering, the system re-inspects the output around each cut boundary using the same timeline-view approach.

- This creates a render → inspect → fix → re-render loop, capped to a small number of retries, so problems like pops, jumps, or hidden captions are caught before the user sees the final result.

9. Animation overlay subsystem

- The skills/manim-video subtree is a specialized sub-skill for generating animation overlays.

- Its references/ directory acts as a knowledge base for Manim-specific topics like scene planning, rendering, camera/3D, graphs, equations, and production quality.

- This is a parallel capability, not the core edit pipeline: overlays can be generated by separate sub-agents and then composed into the main edit.

10. Tooling and external dependencies

- The repo is Python-based (pyproject.toml) and expects ffmpeg for media processing.

- yt-dlp is optional for downloading online sources.

- .env.example indicates API-key based integration, with ElevenLabs explicitly required for transcription.

- The repo is built to be used under different agent hosts, but the architecture assumes an LLM that can read docs, invoke shell commands, and manage iterative media workflows.

11. Important architectural boundary

- The key design boundary is between LLM reasoning and media execution.

- The LLM does not “watch” the whole video; it reasons from packed transcripts and selective timeline composites, then delegates concrete media operations to helper scripts and ffmpeg-based tooling.

- That separation is what makes the system scalable across content types: talking heads, tutorials, travel, interviews, montages, and more.