Building an AI-Narrated Demo Video Pipeline with Vision LLMs and ffmpeg

I used Claude to help build the pipeline and draft this post. I ran the script against real screen recordings and did the debugging — especially the AV sync rabbit hole — but a fair chunk of the scaffolding came out of a back-and-forth with Claude. Side project either way, not something I'd ship or recommend verbatim.

Product demos are painful to produce by hand. You record the screen, miss a click, re-record, write a script, record voiceover, realize the narration doesn't match what's on screen, and start over. I had a pile of raw screen recordings — no narration — and wanted one script that takes them in and outputs a finished product video: narrated MP4, subtitles, one command. No video editor, no manual timeline scrubbing.

Fair warning: this is a fun project. Not a product, not a library, not something with users. I built it because the problem annoyed me and because feeding screen recordings to a vision model and having macOS robot voice narrate them back felt absurd in a way I wanted to explore. At several points I felt silly — sitting there tuning setpts multipliers so macOS's built-in robot voice wouldn't talk over a loading spinner. I'm almost certainly burning more tokens than any sane person would on what is, at the end of the day, a demo video you could have recorded in one take with your own voice in twenty minutes.

I'm writing it up anyway because the ffmpeg sync problem turned out to be genuinely interesting. If you're looking for a production-grade video pipeline, this isn't it. If you want to see what happens when you throw a vision LLM at screen recordings and then fight ffmpeg until the audio fits — read on.

What It Produces

You point the script at one or more screen recordings and get a single product video — an MP4 with narration baked in. That's the deliverable. Everything else is cache or debug output you can delete once you're happy with the result.

• Final MP4 — narrated product demo, ready to share
• SRT / ASS subtitle sidecars (optional; soft track in the MP4 by default, or burn-in with a flag)
• narration.json — editable narration script, cached so you can re-run sync/TTS without calling the vision API again

While developing, the script also writes intermediates you can inspect when something sounds wrong — extracted frames, per-cue audio files, and retimed video segments. These aren't part of the product; they're scaffolding.

The pipeline is designed for iteration. Vision API calls are the expensive step; everything after that can be rebuilt in seconds if you tweak sync parameters or TTS settings.

Screen recordings (.mov)
        │
        ▼
  ffmpeg frame extract  ──►  cached JPEGs (1 frame / 6s)
        │
        ▼
  Vision LLM (Claude)   ──►  narration.json  [{start, text}, ...]
        │
        ▼
  TTS per cue (macOS say) ──►  raw_*.mp3 → seg_*.mp3 (optional atempo)
        │
        ▼
  Hybrid AV sync        ──►  retimed video + delayed audio mix
        │
        ▼
  ffmpeg mux            ──►  final MP4 + subtitles

How I Got Here

I didn't get sync right on the first try. The approaches that failed:

What shipped: high-level vision prompts (skip UI chrome, name features) plus hybrid video retiming — slow the video first, speed the speech only for whatever overflow remains.

Stack and Constants

These constants ended up mattering more than I expected. They're the knobs you turn when the output feels rushed or frozen:

System dependencies (not in npm): ffmpeg, ffprobe, macOS say, and Python 3 + Pillow for the PNG subtitle fallback.

Step 1 — Frame Extraction

Each source clip gets frames extracted into a cache directory. If frames for that clip already exist, extraction is skipped entirely.

For clips longer than seven seconds, ffmpeg samples at one frame every six seconds, scaled to 1280px wide:

ffmpeg -y -i source.mov \
  -vf "fps=1/6,scale=1280:-1" -q:v 3 \
  frames/slug_%04d.jpg

Short clips get a single frame at t=0. The script returns a list of { t, path } objects — timestamp in seconds plus the JPEG path — which become the timeline the vision model sees.

Why six seconds? Too sparse and the model misses short UI transitions (a modal opening and closing within four seconds). Too dense and you burn tokens on nearly identical frames — a form with a cursor blinking looks the same at t=12, t=18, and t=24. Six seconds worked well for feature-length clips where the screen stays on each view long enough to read. For faster-paced recordings I'd drop to 3–4 seconds.

Step 2 — Vision LLM Narration

Feature guides

For each clip, the script maintains a feature guide: a bullet list of capabilities to call out when they appear on screen. This is the highest-leverage input in the whole pipeline. Without it, the model narrates whatever it sees — including loading spinners and calendar widgets. With it, the model knows what matters.

Feature demo beats to cover when visible:
- Cmd+K opens command palette from any page
- Universal search: filter actions and pages as you type
- Search people, clients, invoices by name or ID
- Search help requests by ticket reference → jump to record
- One shortcut replaces clicking through module menus

The guide lives in code as part of a clip definition: [filename, slug, title, featureGuide]. You can edit it between runs without touching the rest of the pipeline.

The prompt

Each clip gets one API call. The message is a multimodal array: a text instruction, then alternating timestamp labels and base64 JPEG frames:

const content = [
  { type: 'text', text: promptWithFeatureGuide },
  { type: 'text', text: 'Frame ~0s:' },
  { type: 'image', source: { type: 'base64', media_type: 'image/jpeg', data: frameData } },
  { type: 'text', text: 'Frame ~6s:' },
  { type: 'image', source: { /* ... */ } },
  // ...
];

const res = await client.messages.create({
  model: visionModel,
  max_tokens: 6000,
  temperature: 0.25,
  messages: [{ role: 'user', content }],
});

Temperature is low (0.25) because you want consistent, factual narration — not creative variation between runs. max_tokens: 6000 is generous; a typical clip returns 8–14 segments totaling ~500 tokens.

The prompt rules matter more than the model choice:

DO:
- 1–2 sentences per segment (20–38 words)
- Keep each line short enough to say in ~5–8 seconds
- Call out specific search targets, shortcuts, and module names when visible
- Tie each line to what is on screen at that timestamp

DO NOT:
- Generic marketing fluff with no feature names
- Date-picker / calendar / scroll / loading-spinner narration
- Character-by-character typing play-by-play

That last category is where vision models waste words. They describe UI chrome that viewers don't care about. Explicit "DO NOT" rules cut that noise significantly — the difference between "the user clicks the date field, opens the calendar, selects March 15" and "the expense date is prefilled from the receipt."

Segment count scales with clip length: 2–3 segments under 15 seconds, 8–10 under 90 seconds, up to 14 for longer recordings.

JSON output and parsing

The model returns a JSON array:

[
  {"start": 8, "text": "Opening the command palette with Cmd+K gives you universal search across people, invoices, and help requests — no need to click through module menus."},
  {"start": 22, "text": "Typing a ticket reference pulls up the matching record and an open shortcut directly from the palette."}
]

Each start is a second offset within that clip. The script strips markdown fences if the model wraps the array in ```json blocks, then validates strictly increasing timestamps.

The JSON is written to narration.json and becomes the editable source of truth. If a line sounds wrong, edit the text and re-run with --build-only — no vision API call needed.

Multiple recordings → one video

If you have several screen recordings (one per feature), the script concatenates them into a single source timeline before sync. Each clip is analyzed independently — frames extracted, vision API called, narration segments returned — then segment timestamps are offset so clip 2 starts where clip 1 ends:

let offset = 0;
for (const clip of clips) {
  for (const seg of clip.segments) {
    globalCues.push({ ...seg, start: seg.start + offset });
  }
  offset += clip.duration;
}

All sync, audio, and subtitle logic runs against this one continuous timeline. One recording in works the same way — just skip the concat step. The output is always a single MP4.

Step 3 — TTS Per Cue

Each narration line gets its own audio file via macOS's built-in say command. I used the Samantha voice — one of the default English voices on macOS (say -v ? lists them):

function tts(text, outMp3) {
  const aiff = outMp3.replace('.mp3', '.aiff');
  run(`say -v Samantha -r 200 -o "${aiff}" "${text.replace(/"/g, '\\"')}"`);
  run(`ffmpeg -y -i "${aiff}" -af volume=3.5 -ar 44100 -ac 1 "${outMp3}"`);
  fs.unlinkSync(aiff);
  return probeDuration(outMp3);
}

Each cue produces a raw_NNN.mp3. The natural duration (audioDur) is probed via ffprobe and attached to the cue — this is the number the sync algorithm must fit against the on-screen slot.

Later, if the sync planner decides speech needs compression, raw_*.mp3 is speed-adjusted into seg_*.mp3 via ffmpeg's atempo filter. The atempo filter only accepts values between 0.5 and 2.0, so larger adjustments require chaining:

function buildAtempoChain(tempo) {
  const filters = [];
  let t = tempo;
  while (t > 2.0) { filters.push('atempo=2.0'); t /= 2.0; }
  while (t < 0.5) { filters.push('atempo=0.5'); t /= 0.5; }
  filters.push(`atempo=${t.toFixed(4)}`);
  return filters.join(',');
}

In practice, MAX_AUDIO_TEMPO is 1.48×, so chaining rarely triggers — but the helper is there if you raise the cap.

Step 4 — Hybrid AV Sync

This is where I spent most of the engineering time.

The problem, concretely

The vision model writes "start": 22 because a feature appears at 22 seconds. TTS for that line might be 9.2 seconds long. The next feature appears at 28 seconds — a 6-second slot. The narration needs 9.2 seconds; the slot offers 6. Without intervention, you get overlapping audio or truncated speech.

Three approaches compared

Hybrid retiming is what shipped.

Building the source timeline

buildSourceTimeline() walks the sorted cues and splits the source video into alternating passive and narrated segments:

[passive 0–8s] [narrated 8–28s] [passive 28–35s] [narrated 35–52s] ...

For each narrated segment:

• sourceStart = cue timestamp in the source video
• sourceSlot = time until the next cue (minimum 0.5s)
• Scene changes in that slot are counted via ffmpeg's scene detection filter
• planNarratedTiming() computes stretch factors

Passive segments pass through at 1× speed with no audio. Narrated segments get retimed.

Motion-aware stretch planning

function planNarratedTiming(sourceSlot, audioDur, sceneChanges = null) {
  const motionCps = sceneChanges != null ? sceneChanges / sourceSlot : 999;
  let maxStretch = 1.25;
  if (motionCps < 5.5) {
    maxStretch = Math.min(maxStretch, 1.1);  // static UI — don't freeze it
  }

  const maxContentDur = sourceSlot * maxStretch;
  const naturalContentDur = Math.max(sourceSlot, audioDur + 0.25);

  let contentDur, audioTempo = 1;

  if (naturalContentDur <= maxContentDur) {
    contentDur = naturalContentDur;           // video stretch alone is enough
  } else {
    contentDur = maxContentDur;
    audioTempo = Math.min(Math.max(audioDur / (contentDur - 0.25), 1), 1.48);
  }

  const outDur = contentDur + 0.35;           // pause after narration
  const videoPts = outDur / sourceSlot;       // setpts multiplier
  return { contentDur, outDur, videoPts, audioTempo, motionCps };
}

Worked example: A 6-second slot, 9.2-second narration, high motion (cursor moving, pages loading):

• naturalContentDur = max(6, 9.2 + 0.25) = 9.45s
• maxContentDur = 6 × 1.25 = 7.5s — narration doesn't fit with video stretch alone
• contentDur = 7.5s, audioTempo = 9.2 / (7.5 − 0.25) ≈ 1.27×
• outDur = 7.85s, videoPts = 7.85 / 6 ≈ 1.31 — video plays 31% slower

Same slot but low motion (static form, 2 scene changes in 6s → 0.33 cps):

• maxStretch capped at 1.1× → maxContentDur = 6.6s
• More overflow goes to audio speedup instead of video freeze

Scene detection uses ffmpeg's select filter:

ffmpeg -ss START -t DURATION -i input.mp4 \
  -vf "select='gt(scene,0.015)',showinfo" -f null -
# count "Parsed_showinfo" lines in stderr

Video retiming with setpts

Each timeline segment is cut from the source, stretched via setpts, and written as a partial MP4:

# -t BEFORE -i limits INPUT duration.
# -t AFTER -i would cap OUTPUT and undo the setpts stretch.
ffmpeg -y -ss 22.000 -t 6.000 -i source.mp4 \
  -vf "setpts=1.308333*PTS,fps=30" -an -pix_fmt yuv420p part_0003.mp4

I lost an afternoon to that -t placement bug. If you put -t after -i, ffmpeg caps the output duration, clipping the stretched video back to the original length — completely defeating the slowdown.

Partials are concatenated:

ffmpeg -y -f concat -safe 0 -i concat.txt \
  -c:v libx264 -preset fast -crf 23 -pix_fmt yuv420p -an retimed.mp4

Frame-rounding reconciliation

After concat, the script probes each partial's actual duration and reconciles against the planned timeline via applyPartDurations(). ffmpeg frame rounding (30fps quantization) can introduce sub-second drift across many segments; probing and rewriting outStart/outEnd on each segment keeps subtitles and audio delays aligned with the actual video.

There's also a safety pass: if a retimed partial ends up shorter than its narration audio, a second setpts correction pass stretches it to fit.

What the logs tell you

On a typical run:

Analyzing motion per segment...
  audio 1.27× @ 187s — Opening the command palette with Cmd+K gives you…
  3 low-motion segments capped at 1.1× video slow

Retiming video to match narration...

Output 612s (source 487s) — 18 video slowed, 7 audio sped

Hybrid sync makes the output longer than the source — often 15–25%. That's the tradeoff for keeping narration intelligible instead of chipmunk-fast.

Step 5 — Audio Mix

Narration cues are mixed onto a single AAC track. Each cue's MP3 is delayed to its output timestamp (post-retiming, not source timestamp):

// For each cue: delay to outStart, then mix all tracks
filters.push(`[${i}:a]adelay=${delayMs}|${delayMs}[a${i}]`);
// ...
const mix = `[a0][a1]...[aN]amix=inputs=N:duration=longest:normalize=0[outmix]`;
const master = `[outmix]apad=whole_dur=${totalDuration},loudnorm=I=-16:TP=-1.5:LRA=11[out]`;

adelay positions each cue at the right moment in the retimed timeline. amix overlays them (with normalize=0 so ffmpeg doesn't attenuate overlapping segments — there shouldn't be any). loudnorm brings the quiet macOS TTS output to broadcast-ish levels (-16 LUFS integrated).

Step 6 — Subtitles and Mux

Subtitle timestamps come from the reconciled timeline — outStart to outStart + playbackAudioDur for each narrated segment. The script writes both SRT and ASS; ASS uses alignment 8 (top center) so subtitles don't cover bottom navigation bars.

Three mux paths, tried in order:

1. Fast mux (default) — copy video and audio streams, attach SRT as a soft mov_text track. No re-encode. Good for iteration in QuickTime or IINA.

2. ASS burn-in — requires ffmpeg compiled with libass. One-pass subtitles filter. My Homebrew ffmpeg didn't have libass, so this path wasn't available out of the box.

3. PNG overlay fallback — a Python script (Pillow) renders each subtitle as a top-aligned PNG with a semi-transparent black background. ffmpeg builds a transparent video track from timed PNGs and overlays it:

draw.rectangle([0, 0, width, height], fill=(0, 0, 0, 200))
draw.multiline_text((width // 2, 12), text, anchor='ma', align='center')

The overlay track is concat-demuxed with precise durations matching each subtitle segment, then composited with overlay=(W-w)/2:28. PNGs and timing metadata are cached — if subtitle text hasn't changed, the overlay track is reused.

Trigger burn-in with --burn-subs when you need subtitles baked in for sharing on platforms that ignore soft tracks.

Running It

One script, one output video:

node demo-pipeline.mjs                      # full run: vision → TTS → sync → mux
node demo-pipeline.mjs --build-only           # skip vision API, reuse narration.json
node demo-pipeline.mjs --build-only --skip-tts  # reuse existing audio files
node demo-pipeline.mjs --burn-subs            # bake subtitles into the final MP4

Full run flow:

1. For each input recording: extract frames (cached) → vision API → collect narration segments
2. If multiple recordings: concat into one source video; offset segment timestamps
3. Write narration.json
4. TTS each cue → probe natural durations
5. Build timeline → plan stretch factors → apply audio fits
6. Retime video → reconcile part durations
7. Write subtitles from reconciled timestamps
8. Mix narration audio → mux into the final MP4

Typical iteration loop: Run full once → watch the output → edit narration.json to fix a bad line → --build-only → watch again → tweak sync constants → --build-only --skip-tts → final --burn-subs export. Vision API is called once; everything else is free.

Gotchas

Narration drift from queue scheduling. The non-overlapping approach (scheduleNonOverlapping) is ~15 lines and tempting. It fixes double-audio but makes the demo unwatchable — narration describes actions long after they happened. Don't use it for screen-synced demos.

The setpts / -t trap. Documented above, worth repeating: -t before -i limits input; -t after -i limits output. Getting this wrong produces video that looks unchanged despite setpts in the filter chain.

ffmpeg libass availability. macOS ffmpeg builds often lack libass. Plan for soft subtitles during development and burn-in only for the final export. The PNG overlay path works everywhere but adds minutes to the build.

Hardcoded paths. Input clips and output copies use fixed directories. Fine for a personal tool; parameterize with CLI flags before sharing.

macOS-only TTS. say locks speech synthesis to macOS. Swapping to a cloud TTS API is straightforward — the sync algorithm only cares about MP3 duration, not how the MP3 was produced.

Output duration growth. Hybrid sync makes the video longer. Budget ~20% extra runtime vs. source. If that's unacceptable, you'll need shorter narration prompts (fewer words) rather than tighter sync caps — going below 1.25× video stretch makes speech sound chipmunk-fast.

No automated tests. I validated output by watching the video. For a recurring pipeline I'd add: duration assertions (output ≥ source), narration JSON schema validation, and a dry-run mode that prints the timeline without calling ffmpeg.

API cost / token guilt. Vision calls with 10–15 frames per recording add up. I re-ran the vision step more times than I'd like to admit while tweaking prompts — each time sending a batch of JPEGs to a model that costs real money to describe a screen I could see myself. Caching narration.json helps (--build-only), but the honest answer is: a human voiceover on a single clean recording would have been cheaper, faster, and probably better. I did this because it was fun to automate, not because it was economical.

The one thing worth taking away if you read this far: the LLM part was easy — frames in, JSON out. The hard part was AV sync. If you ever try something like this, expect to spend most of your time fighting ffmpeg, not tuning prompts.