How gaebalai/study-recoding-pipeline works
1. What this repository is
- A minimal two-stage media pipeline: it turns a script JSON into ElevenLabs speech, then uses Remotion to compose and render an MP4.
- The repository is intentionally split between a Node.js orchestration layer and a Remotion rendering layer, with generated assets written to output/ and remotion/public/audio/.
2. Primary end-to-end flow
- The canonical path is:
examples/sample.json → scripts/synthesize.js → output/audio/<slug>/*.mp3 → scripts/make-props.js → examples/<slug>.props.json → Remotion render → output/videos/<slug>.mp4
- This is the main architecture to diagram: input script, audio synthesis, props generation, static asset staging, video rendering, final MP4 output.
3. Node.js pipeline boundary
- scripts/ contains the pipeline executables that run outside Remotion.
- synthesize.js is the integration point with ElevenLabs and is responsible for producing per-segment MP3 files.
- make-props.js bridges the content model to the Remotion component model by converting the script JSON plus synthesis results into a props file.
- render.js is the top-level orchestrator that loads env vars, coordinates the prior steps, and invokes Remotion rendering.
4. External service boundary: ElevenLabs
- ElevenLabs is the only external SaaS dependency called at runtime.
- The pipeline depends on API key, voice IDs, model selection, timeout, and retry behavior configured through environment variables and speaker config.
- Speech synthesis is where cost and reliability concerns live; the repo includes caching and retry controls to reduce repeated API usage.
5. Audio generation and caching
- Synthesized voice is stored under output/audio/<slug>/.
- A per-slug cache file (.cache.json) deduplicates calls by hashing the combination of text, voice, model, and voice settings.
- This makes audio generation idempotent across reruns unless the cache is cleared or forced regeneration is requested.
6. Remotion project boundary
- remotion/ is a self-contained Remotion app with its own package metadata, TypeScript config, linting, and build/runtime config.
- remotion/src/index.ts and remotion/src/Root.tsx are the entry points for the Remotion composition.
- remotion/src/StudyVideo.tsx is the main composition component that consumes props and assembles the final video.
7. Composition data model
- remotion/src/types.ts defines the rendering-side types used to describe the script, speakers, and layout data.
- remotion/src/speaker-config.json is a key configuration boundary: it maps logical speaker keys to ElevenLabs voice IDs, model IDs, and voice settings.
- Environment variables can override parts of this config, but per-speaker settings take precedence where supported.
8. Static asset staging for rendering
- Remotion reads audio as static files, so synthesized MP3s are copied into remotion/public/audio/.
- This is a deliberate boundary between generated audio and the renderer’s asset lookup mechanism.
- The repository also includes remotion/public/characters/ for placeholder speaker imagery used in the sample video layouts.
9. Input examples and layout coverage
- examples/ contains sample script JSON files that illustrate the supported layout types: bullets/code, compare, image, table, and video.
- These examples are both documentation and test fixtures for the rendering pipeline.
- examples/sample-video.json highlights an extra asset dependency: the user must supply a local clip for embedded video content.
10. Generated artifacts and cleanliness
- output/ is the primary build output directory for both audio and rendered video.
- examples/*.props.json is generated render input, not source.
- The README’s cleanup commands show that the system expects these artifacts to be disposable and reproducible from source inputs.
11. Configuration and execution environment
- The repo is Node-based, with the Remotion app requiring a separate install under remotion/.
- .env.example and the scripts’ automatic .env loading indicate a local-file configuration model rather than a centralized config service.
- The architecture relies on environment variables to control the runtime behavior of synthesis and render steps without changing source code.
12. Docs as operational support
- docs/quickstart.md, docs/setup.md, and related authoring templates support the pipeline by defining input format and setup steps.
- These docs do not add runtime components, but they clarify the contract between the script JSON authoring layer and the rendering pipeline.
- For an architecture diagram, the docs can be shown as support material around the main flow rather than as executable subsystems.
Main entry points: synthesize.js Speech synth · StudyVideo.tsx Study video · make-props.js Props builder · sample.json Sample script
Components 18
Inputs & examples
Source scripts, layout samples, and authoring docs that define the content contract.
- Sample script · Canonical script JSON used to drive the pipeline.
examples/sample.json - Layout samples · Representative inputs for compare, image, table, and video layouts.
examples/sample-compare.json - Authoring docs · Setup and content guidance that define the expected input contract.
docs/quickstart.md
Node pipeline
Scripts that synthesize audio, prepare Remotion props, and orchestrate the end-to-end run.
- Render runner · Top-level script that loads env vars, coordinates synthesis, and invokes video rendering.
scripts/render.js - Speech synth · Calls ElevenLabs and writes per-segment MP3 files with caching and retry behavior.
scripts/synthesize.js - Props builder · Converts script JSON plus synthesis results into Remotion-ready props.
scripts/make-props.js
Remotion app
Self-contained rendering app that turns props and staged assets into video.
- App entry · TypeScript entrypoint for the Remotion app.
remotion/src/index.ts - Root entry · Remotion bootstrap that registers the composition tree.
remotion/src/Root.tsx - Study video · Main composition component that assembles the final video from props and staged assets.
remotion/src/StudyVideo.tsx - Render types · Type definitions for script, speakers, and layout data used by the renderer.
remotion/src/types.ts - Speaker config · Maps logical speakers to ElevenLabs voice IDs, model IDs, and voice settings.
remotion/src/speaker-config.json - Audio staging · Rendered MP3 files copied here so Remotion can read them as static media.
remotion/public - Character art · Placeholder character imagery used by sample layouts.
remotion/public/characters
Generated artifacts
Disposable build outputs produced by synthesis, props generation, and rendering.
- Synthesis output · Per-slug audio files produced by synthesis and cached for reruns.
- Video output · Final rendered MP4 files produced by Remotion.
- Props file · Intermediate Remotion input generated from the script and synthesis results.
Configuration
Environment and speaker mapping that control runtime behavior.
- Env template · Example environment variables for API keys, model selection, timeout, and retry controls.
.env.example
External service
Third-party SaaS used for speech synthesis.
- ElevenLabs · External speech synthesis service used to generate the narration audio.
Connections 21
- Sample script → Speech synth: script JSON · The canonical input script feeds audio synthesis.
- Layout samples → Props builder: input patterns · Sample layouts document the supported content shapes.
- Authoring docs → Sample script: authoring guide · Docs explain how to author scripts consumed by the pipeline.
- Env template → Render runner: env vars · Local configuration controls synthesis and render behavior.
- Env template → Speech synth: api settings · API key and retry/timeout settings affect synthesis calls.
- Render runner → Speech synth: run first · The orchestrator invokes audio generation before rendering.
- Speech synth → ElevenLabs: speech request · Audio is generated through the external speech API.
- ElevenLabs → Synthesis output: MP3 files · Returned narration is stored in the audio output cache.
- Synthesis output → Props builder: audio manifest · Synthesized audio metadata is consumed when building render props.
- Sample script → Props builder: content model · The script structure is transformed into rendering props.
- Props builder → Props file: write props · The bridge emits Remotion-ready props as a generated file.
- Props file → App entry: render input · The generated props are loaded by the Remotion app.
- App entry → Root entry: bootstrap · The app entry initializes the composition tree.
- Root entry → Study video: register comp · Root exposes the main study video composition.
- Speaker config → Study video: voice mapping · Speaker configuration informs narration and voice selection.
- Render types → Study video: data shapes · Types define the props and layout contract used by the composition.
- Audio staging → Study video: static audio · The composition reads staged MP3 assets from the public directory.
- Character art → Study video: art assets · Placeholder imagery supports sample layouts.
- Render runner → Video output: final render · The orchestrator triggers the MP4 render output.
- Speech synth → Audio staging: copy audio · Generated MP3s are staged into Remotion's static asset directory.
- Synthesis output → Audio staging: staged assets · Audio output is mirrored into the renderer's public path.