Short version: a scenario drives the product through surfaces (browser, terminal/TUI). Each surface captures its own native artifact (a Playwright video for the browser, an asciicast for the terminal) and stamps a shared timeline.json as a side effect of acting. Recordings are an off-by-default cost: assertions are vitest's job, not the film's. When E2E_FILM=1, the surfaces slow down and "beat" so the result is followable. A post-step film.ts then cuts each native recording on the focus timeline and concatenates them onto one canvas, so a multi-window run plays back as a single screen recording of a developer tabbing between apps.
The entry point is scenario(name, options, body) in e2e/src/scenario.ts.
It computes a per-run directory (runs/<target>/<slug>), builds an Effect
context of only the surfaces this target can provide, runs the body, and writes a small
result.json plus whatever artifacts the surfaces dropped into the run dir. There is no
central recorder: each surface owns its own capture, and the scenario file itself is shipped as
test.ts because "the test source is the review artifact."
flowchart TB Scenario["scenario(name, options, body)"] --> Ctx["contextFor(target, dir):
add only the surfaces
the target's capabilities allow"] Ctx --> Body["run body Effect"] Body --> Browser["Browser surface
(Playwright)"] Body --> Cli["Cli / chat-theater
(PTY)"] Browser --> RunDir[("runs/<target>/<slug>/")] Cli --> RunDir Browser -. side effect .-> TL["timeline.json"] Cli -. side effect .-> TL RunDir --> Result["result.json + test.ts"] Result --> Film{"both
terminal.cast +
session.mp4?"} Film -->|yes| FilmTs["scripts/film.ts
cut + concat → film.mp4"] Film -->|no| Done["parts stay as artifacts"]
Both capturing surfaces wrap their resource in Effect's acquireUseRelease. That matters
for evidence: a vitest timeout interrupts the fiber, and the release step still runs, so the
browser closes and flushes its video, or the PTY recording is written, even on a failing or hung run.
A bare promise would leak Chromium and lose the footage you most want.
e2e/src/surfaces/browser.ts launches Chromium and opens a context with three capture
mechanisms turned on at once:
recordVideo: { dir, size: 1280×800 }. Playwright writes a .webm as the page runs.context.tracing.start({ screenshots, snapshots, sources }), the time-travel DOM + network + console bundle you drag into trace.playwright.dev.step(label, fn) helper opens a tracing group, runs the action, then saves a numbered PNG named from the step label.
The context is dark-mode and seeded with the identity's cookies, so the recording is of a real logged-in
session against the target's actual web UI, never a stub. On release the surface stops tracing
(trace.zip), closes the context to flush the video, then transcodes:
// webm plays nowhere reliable; mp4 plays everywhere (Safari/iOS don't do webm) ffmpeg -y -i <recorded.webm> \ -c:v libx264 -preset veryfast -crf 26 \ -pix_fmt yuv420p -movflags +faststart \ <run>/session.mp4 // if ffmpeg is missing, fall back to copying the raw .webm
That transcode is the entire reason the artifact is a portable session.mp4. The
+faststart flag moves the MP4 index to the front so it streams inline in a browser or a
GitHub PR without downloading the whole file first.
This is the path our sibling's PR used. The OpenHands agent's policies-duplicate scenario yields only Browser and Api, so it produced exactly this: session.mp4 + trace.zip + step screenshots, recorded with E2E_FILM=1 for the watchable pace. No terminal track means no splice: the browser video is the film.
The CLI surface (e2e/src/surfaces/cli.ts) drives a real pseudo-terminal through
@kitlangton/terminal-control. A scenario can type, press keys, wait for rendered text, and
assert on the screen. With record set, the session's internal JSONL recording is converted
to asciicast v2 (terminal.cast), the same format asciinema plays, using a
single streaming UTF-8 decoder so multi-byte characters split across PTY chunks survive.
On top of the raw CLI surface sits chat theater (e2e/src/clients/chat-theater.ts),
which is the clever bit for product demos. It renders a fake "developer chatting with an agent" TUI, but
every tool spinner brackets a real MCP call:
theater.tool(
{ name: "execute", input: code, result: (v) => ... },
realMcpCall, // the spinner lasts exactly as long as this Effect runs
)
No inference and no third-party agent binary: the chat is a renderer painting genuine calls, so the
recording reads like an agent session while staying fully deterministic. The renderer runs inside the
recorded PTY and the theater feeds it events as base64 lines, producing the terminal.cast.
The keystone idea is in e2e/src/timeline.ts. The film needs to know which window the
developer was looking at, and when, so it can cut between the terminal and the browser at the right
moments. Rather than make scenarios annotate that, the surfaces stamp it as a side effect of normal work:
enterFocus(dir, "browser").markFocus(dir, "terminal").markRecordingStart, the anchor that maps wall time to that recording's own clock.
So timeline.json ends up with anchors (when each camera started),
focus (the ordered window transitions), and nav (URLs). Any scenario gets a
faithful focus track "for free," and the cut list is decided by the operations themselves, for any number
of hops, not by hand-authored markers.
sequenceDiagram
participant S as Scenario body
participant T as terminal (cast)
participant B as browser (mp4)
participant TL as timeline.json
S->>T: chat: "add a rule"
T->>TL: markFocus(terminal) @ t0
Note over T: anchor.terminal = t0
S->>B: step: open /policies
B->>TL: enterFocus(browser) @ t1
Note over B: anchor.browser = t1
S->>T: chat: "done"
T->>TL: markFocus(terminal) @ t2
Note over TL: focus = [terminal@t0, browser@t1, terminal@t2]
A recording at machine speed flickers through states too fast to read. Two off-by-default knobs fix that,
gated on E2E_FILM=1 (or E2E_DESK=1):
The browser surface passes Playwright slowMo: 400, so each action takes 400 ms. That is the "17-second" pace you see in the demo videos. Normal CI runs pass undefined and stay fast.
A "beat" is a deliberate dwell, framework-owned, not hand-coded waitForTimeout. The surfaces beat at the end of a visible step and when focus changes ("look before you tab away"). beat() is a no-op unless filming, so nobody pays for pacing in CI.
The design rule stated in the source is worth repeating: a scenario should never hand-code a sleep to make a film readable, because that is the recording's concern. Pacing is a property of the focus transition, owned by the framework, so the test stays a clean spec.
After a successful run that has both a terminal.cast and a session.mp4,
scenario.ts shells out to e2e/scripts/film.ts (best-effort: a failure here never
fails the run; the parts simply remain). film.ts builds a cut list and concatenates:
{ source, from, to } in that recording's own clock, computed from the wall-clock focus transitions minus the recording's anchor. A run without a usable timeline falls back to a heuristic (locate the browser hop by a narrator line, or the largest idle gap in the cast).agg turns terminal.cast into a gif with idle compression disabled (so cast time equals video time), then ffmpeg makes it an even-dimensioned mp4.filter_complex trims each act out of its source, fits every act onto an identical 1280×800 letterboxed canvas (scale + pad + fps + format), and concats them in order into film.mp4.// per act: trim the source window, normalize onto one canvas [0:v]trim=from:to,setpts=PTS-STARTPTS,<FIT>[act0] // terminal [1:v]trim=from:to,setpts=PTS-STARTPTS,<FIT>[act1] // browser [act0][act1]concat=n=2:v=1:a=0[out] // FIT = scale=1280:800:force_original_aspect_ratio=decrease, // pad=1280:800:(ow-iw)/2:(oh-ih)/2:color=0x0b0b10, // setsar=1,fps=24,format=yuv420p
The only editorial act is the cut; both segments are the genuine recordings, so the film is honest:
it plays like a screen recording of someone tabbing between a full-screen terminal and a full-screen
browser. film.mp4 is registered back into result.json's artifact list so the viewer prefers
it over the bare parts.
There is a heavier alternative for when you want a true single-camera take. e2e/desk/ runs the
scenario inside a Docker image with a virtual X display and films it with one screen grab:
Xvfb :99 -screen 0 1440x900x24 & # virtual display
openbox & xsetroot -solid "#0b0b10" # a window manager + backdrop
ffmpeg -f x11grab -framerate 24 -video_size 1440x900 -i :99 \
-c:v libx264 -preset veryfast -crf 24 -pix_fmt yuv420p desk.mp4 &
(cd e2e && E2E_DESK=1 vitest run --project cloud "$SCENARIO")
Under E2E_DESK=1 the browser launches headed at a fixed window position, and
chat theater renders into a visible xterm (events over a FIFO instead of a recorded PTY). One
x11grab films both windows on the same screen as the developer "tabs" between them; the
resulting desk.mp4 replaces session.mp4 in the run dir. Same scenario file, no
changes: the surfaces just switch transports on the env var. The splice is the cheap default; the desk is
the cinematic one.
e2e/scripts/pr-media.ts closes the loop to GitHub. Given a run dir it picks the best recording
(film.mp4 > session.mp4 > terminal.cast), converts it to a gif
(ffmpeg two-pass palette for the browser mp4, agg for a cast), commits the gif to an orphan
e2e-media branch through the git database API (never touching the worktree), and prints
ready-to-paste markdown. The gif route exists because GitHub renders gifs inline from a raw repo URL and
has no API for the drag-and-drop asset upload.
| Artifact | Producer | What it is |
|---|---|---|
session.mp4 | browser surface + ffmpeg | The browser video, transcoded webm → mp4 for portability. |
trace.zip | Playwright tracing | Time-travel DOM, network, console; opens in trace.playwright.dev. |
NN-<step>.png | step() | One screenshot per named step; failure.png on error. |
terminal.cast | CLI surface | asciicast v2 of the PTY/TUI session. |
timeline.json | all surfaces | Recording anchors, focus transitions, navigations. |
traces.json | browser + MCP | Per-request distributed-trace ledger for the run. |
film.mp4 | scripts/film.ts | The spliced single-camera cut, when both tracks exist. |
result.json | scenario.ts | Pass/fail, timing, and the artifact inventory for the viewer. |
slowMo and beat() live in the surfaces and only fire when filming, so CI stays fast and tests stay free of presentation sleeps.acquireUseRelease guarantees the browser flushes its video and the PTY writes its cast even on timeout: exactly the runs whose footage you need.+faststart for inline playback, and a gif fallback for GitHub, are chosen for where the evidence will be watched.film.ts or a true single-screen take via the desk, by switching transports on an env var, not by forking the test.