OpenHands / verification as infrastructure 01 / 12

Give agents a way
to prove it.

A map of the product. A way to run it. A standard of proof.

We taught agents how to use OpenHands itself: open the app, take the paths a user takes, check what actually happened, and leave evidence someone else can replay.

The verify-openhands story · Engel Nyst
Updated 9 October 2026 · implementation and evidence snapshot

See how the pieces fit ↓

● ● ●Agent Canvas · real verification run

1. 01 Find the behavior
2. 02 Drive the product
3. 03 Observe the result
4. 04 Keep the proof

The design 02 / 12

“Test it like a user”
needs an operating manual.

The repository now carries the knowledge an agent needs to verify its own work.

Merged in #17961; maintained in #18085. The existing CLI deep dive covers the command surface in more detail.

A readable inventory 03 / 12

The unit is a behavior.

“A secret is still there after a reload” gives an agent something concrete to prove.

# F14 — Secrets
Source: src/routes/secrets-settings.tsx, …

## Sub-features
F14.create: adding a secret persists it;
it is listed after a reload.

## How to get to it (user POV)
Settings → Secrets · direct URL · command menu

## Driving it with control-openhands
Preconditions → commands → expected observation

## Gotchas
Real traps and linked known failures

Mapped does not mean passed. The map keeps failures, blocked prerequisites, and gaps visible. Cloud accounts, external integrations and platform-specific behavior need their own available environment.

Counts at merged maintenance revision 8793c11, checked 9 October. The original 716 behaviors have grown to 743 across the same 27 families. The groups above are presentation groupings; the linked index owns the inventory.

The lever 04 / 12

One checkout. One isolated run.
One browser that remembers.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)

control-openhands doctor
control-openhands onboard --skip
control-openhands browser goto /settings/secrets
control-openhands browser snapshot
control-openhands browser errors

# After verification:
control-openhands stop

The CLI returns structured results with actionable failures. Read the launch, doctor, drive and cleanup contract.

A recipe you can replay 05 / 12

Example: save it, reload it,
find it again.

F14.create verifies persistence through the UI. Opening the form is only the beginning.

control-openhands browser goto /settings/secrets
control-openhands browser click 'testid=add-secret-button'
control-openhands browser fill 'testid=add-secret-form >> testid=name-input' QA_TMP_SECRET
control-openhands browser fill 'testid=add-secret-form >> testid=value-input' dummy-value-123
control-openhands browser click 'testid=add-secret-form >> testid=submit-button'

control-openhands browser reload
control-openhands browser count 'testid=secret-item >> has-text=QA_TMP_SECRET'

# Expected observation: count 1 after reload.
# If that is not what happened, do not record a pass.

control-openhands browser screenshot --feature F14.create --name after-reload

control-openhands evidence add --feature F14.create --result pass \
  --entry "Settings > Secrets > Add" \
  --expected "row survives reload" --actual "count 1 after reload" \
  --artifact evidence/F14.create/after-reload.png
control-openhands evidence report
control-openhands stop

1. Setup is not proof.

   API writes may arrange a test. A UI claim still needs UI evidence.

2. One pass has a precise scope.

   The ledger records an ID and entry point: pass, fail, blocked or not-run.

3. The result must be observable.

   Inspect saved state, real files or tool observations. Model wording cannot stand in for those checks.

Adapted from the actual F14 Secrets recipe and evidence contract. Run in order on a fresh isolated stack.

Evidence / 01 06 / 12

Example: the Close button
was off the phone.

A perfectly ordinary user action exposed a real layout failure.

Open Settings on a 390px viewport. Open the Agent Canvas update dialog. The 520px dialog spills past both edges; its title is cut off and its Close button is outside the screen.

The agent reproduced it, captured the before state, applied the shared viewport cap, and checked the fixed build at phone, narrow and desktop widths.

Merged · 5 Oct PR #18008 · fixes #17903

With the update dialog open

control-openhands browser viewport phone
control-openhands browser bbox \
  'testid=agent-canvas-update-modal'
control-openhands browser click \
  'testid=close-agent-canvas-update-modal'
control-openhands browser count \
  'testid=agent-canvas-update-modal'
# Expected: 0 after Close.

The PR reports x = −65 / width 520 before, x = 19.5 / width 351 after. The screenshot is paired with geometry and dismissal checks.

Before · 390 × 844

After · 390 × 844

Original, unmodified captures from #18008. The initial verification effort and its 6 October refresh filed 68 issues. This was one of dozens of fixes that landed while an agent kept updating the still-open feature-map PR.

Evidence / 02 07 / 12

Example: some failures disappear
before the next screenshot.

One failed “Run now” action showed two error toasts.

The recipe creates an inert automation, loads its card, then deletes it through the API to make the UI stale. Clicking Run now takes the real error path.

Both the caller and a global error handler displayed a toast. browser toasts --history preserved the evidence. The fix leaves one useful error message.

#18043 · merged 6 Oct · fixes #17906

Example: keyboard behavior needs keyboard proof.

#18011 fixed Delete automation: Escape did nothing, focus stayed outside, and assistive technology had no named dialog. The verification used key presses and the accessibility tree.

Example: visible content needs a fresh read.

#18032 fixed a stale Files preview after an agent edited a file. Even Refresh and a page reload had continued to show old contents.

Before · two toastsAfter · one toast

Original captures from separate before/after stacks; fixture lists differ. The comparison is the number and content of error toasts. Click the image to inspect it.

These PRs explicitly connect their fixes to the full agentic run. Screenshots support the claim; the reproduction and observation steps make it checkable.

Why the structure matters 08 / 12

Every new agent inherits
what the last one learned.

It was used beyond the first sweep. The 5 October fix-fleet report records 36 Canvas fixers using the CLI; across the wider fix/review fleet, 170 isolated launches and 261 screenshots. These are usage counts, not a measured speedup or a correctness guarantee.

The OpenHands adaptation explains the persistent browser and isolation. Unit tests and CI remain part of verification; this adds the live user experience.

The part that keeps it useful 09 / 12

The map changes with the product.

Maintenance asks three separate questions: is the feature present, does it work and look right, and was the change intended?

1. 01

   Freeze revisions

   Pin BASE and TARGET. Comparable runs need known code and separate state.

2. 02

   Read the changes

   Map changed paths to behavior IDs. Read PR and issue intent. Shared code widens the scope.

3. 03

   Drive it live

   Replay recipes and known failures. Compare BASE and TARGET before attributing a regression.

4. 04

   Reconcile

   Keep evidence, fix the right layer, and propose a reviewed baseline only after a completed pass.

Intent and runtime are independent. “Documented change” does not mean “works.” A merged fix still needs to be exercised; known failures get replayed.

The merged maintenance rules define the comparison, intent ledger, live pass and baseline policy.

A living system 10 / 12

The follow-ups close the loop.

Credit where it belongs 11 / 12

The idea came from pstack.

The OpenHands attribution and adaptation notes record what was borrowed and what was built for this product. See also our earlier pstack decision note.