Automation detail, runs and editing

F23 · 41 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F23 Automation detail, runs and editing

41 behaviors · 34 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

The detail page (/automations/<id>) shows one automation: a Back to Automations link, a header with the name, an Active/Inactive badge, Run now, an on/off switch and a kebab (Export, Tarball, Edit, Turn on/off, Delete); a Why this is paused banner when it was turned off; the Prompt (or, for a script automation, the Script bundle); Configuration (repositories, trigger, schedule or event source/type/filter, LLM profile); a Plugins card when it has plugins; an Activity card (Created, Last run); and the Activity Log of runs with status, summary, cost, a logs dialog, a link to each run's conversation and JSON/CSV export. Kebab Edit opens the setup page in edit mode (/automations/setup?automationId=<id>, since #17688), which changes name, prompt, agent and LLM profile, timeout and the trigger. A DRAFT-state test automation (materialized by a draft Test, F21.saved-drafts) shows a Draft badge and no on/off switch.

How a user gets here

Dashboard (/automations, F21): click a card's heading, a list row link, or the kebab Automation actions → View; a sparkline bar opens /automations/<id>?run=<runId> (F21.sparkline-deeplink).
Direct URL /automations/<id> and /automations/<id>?run=<runId>. Get <id> from control-openhands browser url after opening a card, or from control-openhands api GET /api/automation/v1; get <runId> from control-openhands api GET /api/automation/v1/<id>/runs.
Edit: the detail kebab Edit (exercised here) and the dashboard kebab Edit (F21.edit) both open /automations/setup?automationId=<id>; a cold load of that URL opens it too.
Browser Back from a run's conversation returns to the detail page.
The detail page has no command-menu entry and no keyboard shortcut.

Given

Preconditions:

Baseline state (launched, doctored, onboard --skip done) and an active LLM profile for the runs: control-openhands llm preset deepseek (--api-key-file <file> when the key is in a file); the edit recipes also expect deepseek-pro in the LLM profile list.
No QA_F23 automations and no Daily news digest exist (control-openhands api GET /api/automation/v1).
Arrange three fixtures through the API (arrange only; F22 owns creating automations in the UI). Each prints the new automation in body (body.id is <id>): control-openhands api POST /api/automation/v1/preset/prompt --write --data '{"name":"QA_F23 Pong","prompt":"Reply with the single word: pong. Do not run any tools.","trigger":{"type":"cron","schedule":"0 9 * * *","timezone":"UTC"},"enabled":false}', control-openhands api POST /api/automation/v1/preset/prompt --write --data '{"name":"QA_F23 Fail","prompt":"Reply with the single word: pong.","trigger":{"type":"cron","schedule":"0 0 1 1 *","timezone":"UTC"},"enabled":true}', control-openhands api POST /api/automation/v1/preset/plugin --write --data '{"name":"QA_F23 Event","prompt":"QA line 1: reply pong.\nQA line 2: reply pong.\nQA line 3: reply pong.\nQA line 4: reply pong.\nQA line 5: reply pong.\nQA line 6: reply pong.\nQA line 7: reply pong.\nQA line 8: reply pong.\nQA line 9: reply pong.\nQA line 10: reply pong.\nQA line 11: reply pong.\nQA line 12: reply pong.\nQA line 13: reply pong.\nQA line 14: reply pong.\nQA line 15: reply pong.","plugins":[{"source":"github:qa-example/qa-plugin"}],"repos":[{"url":"https://github.com/qa-example/qa-repo","ref":"main"}],"trigger":{"type":"event","source":"github","on":"pull_request.opened","filter":"pull_request.draft == `false` && pull_request.base.ref == repository.default_branch"},"enabled":false}'. Pong is a daily preset (0 9 * * *) created off; Fail is a yearly custom cron (0 0 1 1 *) created on, so it never fires by itself; Event never fires (no GitHub webhooks).
Desktop viewport. Open a fixture's page the way a user does: control-openhands browser goto /automations, then control-openhands browser click '[data-testid^="automation-card-"] >> has-text=<name> >> role=heading' --expect-url '/automations/[0-9a-f-]+'. Below this is called "open <name>".

Open from the dashboardF23.open-detail · F23.headerMapped behavior

Scenario F23--open-from-the-dashboard · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation actions”.
Do
Press Escape.

Expect

a dashboard card, a list row, the kebab View and a direct URL open /automations/<id>; browser Back from a run's conversation returns to it.

the header shows the name, the Active/Inactive badge, Run now (disabled while inactive), the switch (Turn on/Turn off) and the Automation actions kebab with Export, Tarball, Edit, Turn on/Turn off and Delete. A DRAFT-state test automation shows a Draft badge (active-status-badge-draft), no switch, no paused banner and a kebab without Turn on/off.

Exact CLI recipe & bug checks

Open QA_F23 Pong, then control-openhands browser url (/automations/<id>), control-openhands browser snapshot and control-openhands browser screenshot --feature F23.header --name inactive. The snapshot shows link Back to Automations, heading QA_F23 Pong (level 1), Inactive, button "Run now" [disabled], switch "Turn on" and button "Automation actions", then the Prompt, Configuration, Activity (Last run Never) and Activity Log (No activity yet, Export JSON [disabled], Export CSV [disabled]) sections. Run control-openhands browser click 'role=button[name="Automation actions"]' and control-openhands browser snapshot 'role=list >> has-text=Tarball': buttons Export, Tarball, Edit, Turn on, Delete. control-openhands browser press Escape closes it (browser count 'role=list >> has-text=Tarball' is 0).

Other ways inF23.open-detail · F23.back-linkMapped behavior

Scenario F23--other-ways-in · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Back to Automations”.
Do
Choose the element identified in the CLI recipe.
Do
Choose the list / containing “Export” / “View”.
Do
Open /automations.
Do
Choose “Automations view toggle”.
Do
Choose “Automations view toggle list”.
Do
Choose the element identified in the CLI recipe.
Do
Open /automations/<id>.

Expect

a dashboard card, a list row, the kebab View and a direct URL open /automations/<id>; browser Back from a run's conversation returns to it.

Back to Automations (on the page and on the not-found state) returns to /automations.

Exact CLI recipe & bug checks

control-openhands browser click 'role=link[name="Back to Automations"]' --expect-url '/automations(\?|$)'; then control-openhands browser click '[data-testid^="automation-card-"] >> has-text=QA_F23 Pong >> role=button[name="Automation actions"]' and control-openhands browser click 'role=list >> has-text=Export >> role=button[name="View"]' --expect-url '/automations/[0-9a-f-]+' land on the same /automations/<id>. List view too: control-openhands browser goto /automations, control-openhands browser click 'testid=automations-view-toggle', control-openhands browser click 'testid=automations-view-toggle-list', control-openhands browser click '[data-testid^="automation-list-row-"] >> has-text=QA_F23 Pong >> role=link >> nth=0' --expect-url '/automations/[0-9a-f-]+'; then restore grid (goto /automations, the toggle, testid=automations-view-toggle-grid; browser eval "localStorage.getItem('openhands-automations-view')" is "grid"). control-openhands browser goto /automations/<id> (direct URL) shows the same page.

States: loading and not foundF23.loading · F23.not-foundMapped behavior

Scenario F23--states-loading-and-not-found · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/00000000-0000-0000-0000-000000000000.
Wait
Wait for the element identified in the CLI recipe to appear.
Do
Choose “Back to Automations”.

Expect

detail-skeleton stands in while the automation loads.

an unknown id shows Automation not found with "The automation you're looking for doesn't exist or has been deleted." and the back link.

Exact CLI recipe & bug checks

control-openhands browser goto /automations/00000000-0000-0000-0000-000000000000 and at once control-openhands browser count 'testid=detail-skeleton' (1). control-openhands browser wait-text 'Automation not found' succeeds after about 7 s, detail-skeleton is then 0, and control-openhands browser snapshot 'role=paragraph >> text=Automation not found >> xpath=..' lists both paragraphs and link Back to Automations. Screenshot with --feature F23.not-found --name not-found; control-openhands browser click 'role=link[name="Back to Automations"]' --expect-url '/automations(\?|$)'. browser errors --app-only lists the four induced 404s on /api/automation/v1/00000000-....

Backend unavailableF23.backend-unavailableMapped behavior

Scenario F23--backend-unavailable · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Change the run-owned service state: stop automation.
Do
Open /automations/<id>.
Wait
Wait for the element identified in the CLI recipe to appear.
Do
Restart the run-owned stack as specified.
Do
Choose “Retry”.
Wait
Wait for “QA_F23 Pong” to appear.

Expect

with the automation service down, the page shows Automations Unavailable and Retry; Retry recovers once the service is back.

Exact CLI recipe & bug checks

On QA_F23 Pong: control-openhands service stop automation, control-openhands browser goto /automations/<id>, control-openhands browser wait-text 'Automations Unavailable' and control-openhands browser text 'role=button[name="Retry"] >> xpath=..': Automations Unavailable, The automations backend is not available right now. Please try again later or check that the automation service is running., Retry. Then control-openhands restart, control-openhands browser click 'role=button[name="Retry"]' and control-openhands browser wait 'role=heading[name="QA_F23 Pong"]': the page recovers without a reload.

Load errorF23.load-errorMapped behavior

Scenario F23--load-error · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations.
Setup
Change the run-owned service state: stop automation.
Do
Choose the element identified in the CLI recipe.
Wait
Wait for the element identified in the CLI recipe to appear.
Do
Restart the run-owned stack as specified.

Expect

when the health check is still good but the automation request fails, the page shows Failed to load automations and Retry.

Exact CLI recipe & bug checks

control-openhands browser goto /automations (a fresh load, so the detail is not cached), wait for the card, control-openhands service stop automation, then within 30 s control-openhands browser click '[data-testid^="automation-card-"] >> has-text=QA_F23 Pong >> role=heading' --expect-url '/automations/[0-9a-f-]+' and control-openhands browser wait-text 'Failed to load automations' (after the retries, about 7 s). control-openhands browser text 'role=button[name="Retry"] >> xpath=..' is Failed to load automations / Retry. control-openhands restart, click Retry, and the detail page returns.

Turn on with the switchF23.toggleMapped behavior

Scenario F23--turn-on-with-the-switch · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Turn on”.
Wait
Wait for “Active status badge active” to appear.
Do
Reload the page.

Expect

the header switch flips the enabled flag at once (no confirmation); the badge, switch and Run now follow and survive a reload.

Exact CLI recipe & bug checks

On Pong run control-openhands browser click 'role=switch[name="Turn on"]', control-openhands browser wait 'testid=active-status-badge-active' --timeout 5000 and control-openhands browser count 'testid=confirmation-modal' (0: no confirmation). After control-openhands browser reload, control-openhands browser text 'testid=active-status-badge-active' is Active, control-openhands browser attr 'role=switch[name="Turn off"]' aria-checked is true and control-openhands browser enabled 'role=button[name="Run now"]' is true.

Run now and live statusF23.run-now · F23.run-status-polling · F23.activity-logKnown failureKnown issue

Scenario F23--run-now-and-live-status · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Run now”.

Expect

Run now dispatches a run with the toast Automation dispatched and a Pending row appears in the log (the button reads Starting… only for the few ms of the request).

while a run is Pending or Running the log refetches every 3 s, so Pending → Running → Successful/Failed, the summary, the cost and the conversation link appear without a reload; the Activity card's Last run updates too.

Activity Log shows No activity yet with Export disabled before the first run, then one row per run (newest first): start time, summary, cost ($0.0000 format, run-cost), a logs icon, status badge (run-status-icon-<status>).

Linked known issues

Known issue #15465 (F23.configuration-event F23.activity-log): An automation configured for Agent Canvas release events never runs automatically, although Run now works. Recheck a real signed GitHub/Forgejo release webhook through an externally reachable responder into a disposable automation and inspect its activity log; local manual dispatch cannot verify event delivery. No external repository webhook credentials or controlled release-event fixture are available.

Known issue OpenHands/software-agent-sdk#4705 (F23.run-status-polling, F23.run-logs-modal): GitHub MCP tool events can omit inputSchema, causing an automation's RemoteConversation decoder/run log to report failure while the agent continues, so task and automation statuses disagree. Recheck a real GitHub-MCP-enabled run using the affected client/server versions and its retained tool-definition event; the issue does not give those versions. A generic MCP fixture does not establish the reported GitHub/version branch, and genuine run comparison needs a model key.

Known issue OpenHands/automation#286 (F23.run-status-polling): Multi-worker/replica automation deployments can run duplicate watchdogs that query the same sandbox and hit rate limits, falsely failing runs. Recheck the real deployment's worker/replica configuration with an owned long-running Cloud run and sandbox verification traffic; this single-worker local stack cannot prove distributed scheduler/watchdog ownership.

Known issue OpenHands/automation#219 (F23.run-status-polling): Starting a Cloud automation at the user's sandbox cap can pause an existing interactive session instead of reaching the graceful Skipped–Limit Reached path. Recheck only on an owned disposable Cloud account at its recorded cap, with a live interactive fixture and a new automation run, then observe both run and conversation states. This run has no Cloud account/cap fixture; do not evict a production session as a test.

Known failure OpenHands/automation#514 spans Home cancellation (F03.cancel-in-flight-run) and F23.activity-log/F23.run-logs-modal: on 2026-10-12 a genuine Home Cancel run changed the owned bounded local worker's row to Cancelled, while its Python entrypoint remained alive in the owned launcher ancestry. The F03 driving bullet owns upload/create/cancel steps and retained evidence. The worker later reached its 180 s bound and exited; a cancelled status alone does not prove process cleanup.

Known issue OpenHands/software-agent-sdk#4388 (F23.run-status-polling, F23.run-logs-modal): incremental SDK Bash output polling can return empty pages before newer chunks when order__gt is applied after the file-page boundary. Canvas's frontend log reader does not send that filter, but automation launch preparation and prompt/plugin preset repository clones use (Async)RemoteWorkspace.execute_command, whose output poller does. Recheck a bounded genuine preset/setup or repository-clone command that emits enough chunks to cross the 100-event boundary, correlating its final BashOutput and real run result; a frontend Logs read or a one-output command does not cover the incremental branch.

Known issue OpenHands/software-agent-sdk#5937 (F23.run-status-polling, F23.run-logs-modal): prompt/plugin preset workers attach AUTOMATION_CONVERSATION_ID for continued conversations and then call RemoteConversation.run(). If that same conversation is already running a tool for more than 30 s, the join request can time out and fail the automation worker while the agent continues. Recheck an owned continued-conversation automation with a genuine already-running long tool and correlate worker error, agent status and final run result; a fresh conversation or one normal first run() does not exercise the join. No model execution is authorized in this pass.

Known issue OpenHands/software-agent-sdk#5938 (F23.run-status-polling): prompt/plugin preset workers register an event callback that forwards ActionEvent.summary as the live phase. After an events-socket reconnect, REST-reconciled events reach SDK state but are not replayed to that callback, so summaries emitted in the gap can be missed. Recheck an owned real preset worker through a bounded reconnect window while a genuine action is recorded; compare callback-driven phase with stored events. Current Canvas phase rendering has its separate #17121 gap, and the worker's completion metadata is not claimed lost merely because this callback missed an event.

Exact CLI recipe & bug checks

Run control-openhands browser click 'role=button[name="Run now"]', then at once control-openhands browser toasts (Automation dispatched) and control-openhands browser testids 'testid=automation-activity-log' (run-status-icon-pending). Without reloading, repeat control-openhands browser text 'testid=automation-activity-log' every few seconds: Pending, then Running (run-status-icon-running) after about 5-20 s, then about 30-60 s after dispatch <weekday, month day, year at h:mm AM>, the agent's summary (for example Replied with the single word "pong" as requested, without running any tools.), a cost such as $0.0018 and Successful (run-status-icon-completed, run-cost with title Cost). control-openhands browser text 'role=heading[name="Activity"][exact] >> xpath=ancestor::div[2]' (without [exact] it also matches Activity Log and returns two texts) reads Last run / Just now. Screenshot with control-openhands browser screenshot --feature F23.run-status-polling --name completed.

Linked conversationF23.run-linked-conversationKnown issueBlocked

Scenario F23--linked-conversation · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation activity log” / the link.
Do
Go back in browser history.
Wait
Wait for “Automation activity log” to appear.

Expect

a run row with a conversation is a link labelled View conversation for run at <time> to /conversations/<conversation_id>, whose title is <automation name> — <UTC timestamp>.

Linked known issues

Known issue OpenHands/software-agent-sdk#5279 (F23.run-linked-conversation): An automation that attaches to an existing remote conversation can complete with no conversation link in its run, because the workspace callback does not register the attached id. Recheck a Cloud automation script that calls RemoteConversation.attach on an owned stable conversation and exits through the real workspace completion callback, then inspect the Activity Log link. Blocked without that authenticated Cloud workspace/callback context; a plain local GET is not callback proof.

Known issue OpenHands/software-agent-sdk#4260 (F23.run-linked-conversation, F23.run-logs-modal): Cloud PR-review automation can remain at review in progress and time out without posting a review; its event webhook also fails to serialize SecretStr, breaking the UI stream even on successful reviews. Keep the stream-serialization failure distinct from the empty-response/600-second run-budget failure. Recheck with an owned Cloud reviewer/webhook deployment, authorized disposable review target and real model credentials; a local script run proves neither production path.

Known issue OpenHands/automation#113 (F23.run-linked-conversation): Scheduled plugin jobs can start conversations visible in the archived list while the automation run UI says no conversation started. Recheck the affected automation/run ids, archived conversation ids and deployment version, using the real callback relationship; these identifiers and version are not provided. This is a reported missing-link state, not proof that no agent ran.

Exact CLI recipe & bug checks

Run control-openhands browser attr 'testid=automation-activity-log >> role=link' href (/conversations/<conversation_id>) and ... aria-label (View conversation for run at <time>), then control-openhands browser click 'testid=automation-activity-log >> role=link' --expect-url '/conversations/' and control-openhands browser text 'testid=conversation-name': QA_F23 Pong — <YYYY-MM-DD HH:MM:SS> UTC. control-openhands browser back returns to the detail page (browser wait 'testid=automation-activity-log').

Run deep linkF23.run-deeplinkMapped behavior

Scenario F23--run-deep-link · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/<id>?run=<runId>.
Wait
Wait for “Automation run highlight <runid>” to appear.

Expect

?run=<runId> highlights that run (automation-run-highlight-<runId>, class bg-focus/10) and scrolls to it; an unknown run id highlights nothing.

Exact CLI recipe & bug checks

Get <runId> from control-openhands api GET /api/automation/v1/<id>/runs (body.runs[0].id). control-openhands browser goto '/automations/<id>?run=<runId>', control-openhands browser wait 'testid=automation-run-highlight-<runId>' and control-openhands browser attr 'testid=automation-run-highlight-<runId>' class (bg-focus/10). With ?run=00000000-0000-0000-0000-000000000000, control-openhands browser testids --filter highlight lists nothing.

Logs dialogF23.run-logs-modalKnown failureKnown issue

Scenario F23--logs-dialog · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose the element identified in the CLI recipe.
Do
Choose the dialog / “Error”.
Do
Press Escape.
Do
Reload the page.

Expect

the row's terminal icon (View logs) opens the Logs dialog with Run, Task (status, summary, automation-task-metadata) and System sections and Output/Error tabs showing the run command's stdout/stderr; Escape and the X close it without navigating.

Linked known issues

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=automation-activity-log >> title=View logs', control-openhands browser url (unchanged: the click does not follow the row link) and control-openhands browser snapshot 'role=dialog': dialog "Logs" with Run COMPLETED, Task Successful plus the summary and Task metadata (automation-task-metadata: the finish tool's extra fields, for example { "message": "pong", "summary": "Replied with the single word pong as requested" }; the keys vary with what the agent sent), System No system issues reported., and tabs Output [selected] / Error. control-openhands browser text 'testid=run-logs-output-stdout' is the run command's stdout: it starts [setup] Fetching SDK version from automation service and contains === EXECUTION MODE === and mode: LOCAL (the same text control-openhands api GET '/api/bash/bash_events/search?kind__eq=BashOutput&command_id__eq=<bash_command_id>' returns; bash_command_id from the runs API). Screenshot with --feature F23.run-logs-modal --name output. control-openhands browser click 'role=dialog >> role=tab[name="Error"]' selects Error (aria-selected true); control-openhands browser press Escape makes browser count 'role=dialog' 0. Request scope: control-openhands browser reload (a reopened dialog reuses its cached logs), control-openhands browser network --clear, click testid=automation-activity-log >> title=View logs again and run control-openhands browser network --filter bash: only GET /api/bash/bash_events/search (server level, 200), no /api/conversations/<id>/bash/... although this run has a conversation. Escape.

Export the logF23.activity-log-exportMapped behavior

Scenario F23--export-the-log · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Activity log export json”.
Do
Choose “Activity log export csv”.

Expect

Export JSON / Export CSV download <name-slug>.activity-log.json|csv with every run, including conversation_url.

Exact CLI recipe & bug checks

control-openhands browser click 'testid=activity-log-export-json' then control-openhands browser downloads --last 1 --inspect: qa-f23-pong.activity-log.json with runs[] holding run_id, automation_name, trigger, start_time, end_time, duration_seconds, status COMPLETED, conversation_id and conversation_url (http://127.0.0.1:<port>/conversations/<id>). control-openhands browser click 'testid=activity-log-export-csv' and the same downloads call: qa-f23-pong.activity-log.csv with header run_id,automation_id,automation_name,trigger,start_time,end_time,duration_seconds,status,conversation_id,conversation_url,error,cost,phase.

Export and TarballF23.export · F23.tarballMapped behavior

Scenario F23--export-and-tarball · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation actions”.
Do
Choose the list / containing “Tarball” / “Export”.

Expect

kebab Export downloads <name-slug>.automation.json (version, kind: "automation", spec); an automation's first repository and its plugins go into spec as repository, branch and plugins.

kebab Tarball downloads the automation's code bundle.

Exact CLI recipe & bug checks

control-openhands browser click 'role=button[name="Automation actions"]', control-openhands browser click 'role=list >> has-text=Tarball >> role=button[name="Export"]', control-openhands browser downloads --last 1 --inspect: qa-f23-pong.automation.json with "version": 1, "kind": "automation" and spec (name, trigger, enabled, prompt, model null, timeout 600, timezone). Reopen the kebab and click role=list >> has-text=Tarball >> role=button[name="Tarball"]; control-openhands browser downloads --last 1 lists QA_F23_Pong.tar (about 9 KB) and no toast appears. The file is gzip data despite the name: tar -tzf <path> lists main.py, finish_tool_hook.py, prompt.txt, setup.sh.

Turn off and the paused bannerF23.disabled-reason · F23.toggleMapped behavior

Scenario F23--turn-off-and-the-paused-banner · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose the list / containing “Tarball” / “Turn off”.
Do
Reload the page.

Expect

after Turn off, the banner automation-disabled-reason-banner reads Why this is paused / Turned off manually / Paused <time>; it is gone while active.

the header switch flips the enabled flag at once (no confirmation); the badge, switch and Run now follow and survive a reload.

Exact CLI recipe & bug checks

While Pong is active control-openhands browser count 'testid=automation-disabled-reason-banner' is 0. Open the kebab, control-openhands browser click 'role=list >> has-text=Tarball >> role=button[name="Turn off"]', control-openhands browser count 'testid=confirmation-modal' (0), control-openhands browser reload, control-openhands browser text 'testid=automation-disabled-reason-banner': Why this is paused / Turned off manually / Paused Just now (automation-disabled-reason-text, automation-disabled-reason-timestamp). The badge reads Inactive and Run now is disabled. Screenshot with --feature F23.disabled-reason --name manual.

Edit fields and closingF23.edit-open · F23.edit-closeMapped behavior

Scenario F23--edit-fields-and-closing · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation actions”.
Do
Choose the list / containing “Tarball” / “Edit”.
Do
Press Escape.
Do
Choose “Automation setup close”.
Do
Open /automations/setup?automationId=<id>.

Expect

kebab Edit opens the setup page /automations/setup?automationId=<id> (#17688; the Edit automation modal is gone) prefilled with Name, Prompt, agent profile and LLM profile pickers, the trigger (Schedule with Frequency, or Event), timezone and Timeout seconds, with Test, Save and Close in the toolbar and a docked composer; no conversation exists until a prompt is sent.

Close leaves without saving and lands on /automations (the dashboard, not the detail page); Escape does nothing on the page.

Exact CLI recipe & bug checks

On Pong run control-openhands browser click 'role=button[name="Automation actions"]' and control-openhands browser click 'role=list >> has-text=Tarball >> role=button[name="Edit"]' --expect-url '/automations/setup': the URL is /automations/setup?automationId=<id>, browser count 'testid=edit-automation-save' is 0 (no modal), and browser value reads automation-setup-name (QA_F23 Pong), automation-setup-prompt, the stored schedule (a custom cron shows Frequency Custom with automation-setup-custom-schedule) and automation-setup-timeout (600); browser text 'testid=automation-setup-agent-profile' is default and browser text 'testid=automation-setup-model' names the active profile (the stored model stays null). browser testids 'testid=automation-setup-toolbar' lists automation-setup-test, automation-setup-save and automation-setup-close; control-openhands conversation list is unchanged. control-openhands browser press Escape leaves the URL unchanged; control-openhands browser click 'testid=automation-setup-close' --expect-url '/automations$' lands on the dashboard without a PATCH (the dashboard's own GETs follow). A cold control-openhands browser goto '/automations/setup?automationId=<id>' opens the same form.

ValidationF23.edit-validationMapped behavior

Scenario F23--validation · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Enter “” in “Automation setup prompt”.
Do
Choose “Automation setup save”.
Do
Hover over “Automation setup name”.

Expect

an empty prompt is refused with the toast Add a prompt before continuing. (no request); a bad cron, a cron that never runs or a timeout above 1800 is sent to the API, refused with 422, and its message is the single error toast (Value error, Invalid cron expression: not a cron, Value error, Cron expression cannot produce any future fire times: 0 0 31 2 *, Value error, timeout must not exceed 1800 seconds (30 minutes)); nothing is saved.

Exact CLI recipe & bug checks

On the edit page run control-openhands browser fill 'testid=automation-setup-prompt' '', control-openhands browser network --clear and control-openhands browser click 'testid=automation-setup-save': control-openhands browser toasts --history includes the toast Add a prompt before continuing. and browser network --filter 'automation/v1' lists no request. Refill the prompt. On a Custom-cron automation (QA_F23 Fail) fill testid=automation-setup-custom-schedule with not a cron (then 0 0 31 2 *, then a valid cron with testid=automation-setup-timeout 1801), click Save and read control-openhands browser toasts --history after browser toasts --history --clear. Each sends a PATCH that returns 422 (browser network --filter 'automation/v1'), and the history holds exactly one toast with the API message: Value error, Invalid cron expression: not a cron, Value error, Cron expression cannot produce any future fire times: 0 0 31 2 * and Value error, timeout must not exceed 1800 seconds (30 minutes); api GET /api/automation/v1/<Fail id> keeps 0 0 1 1 * and timeout 600. Before each new click move the pointer off the toast (browser hover 'testid=automation-setup-name', about 5 s; see Gotchas).

No-change SaveF23.edit-saveMapped behavior

Scenario F23--no-change-save · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open Edit, control-openhands browser network --clear, click testid=automation-setup-save, then control-openhands browser network --filter 'automation/v1': one PATCH with the unchanged values and the toast Automation updated (the old modal sent nothing).

Expect

Save sends one PATCH (also when nothing changed), shows Saving… then Saved just now, the toast Automation updated, and the detail page and API reflect it; an Active profile model stays null.

Exact CLI recipe & bug checks

Open Edit, control-openhands browser network --clear, click testid=automation-setup-save, then control-openhands browser network --filter 'automation/v1': one PATCH with the unchanged values and the toast Automation updated (the old modal sent nothing).

Weekly schedule, model and timeoutF23.edit-schedule · F23.edit-saveMapped behavior

Scenario F23--weekly-schedule-model-and-timeout · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation setup frequency weekly”.
Do
Use browser select as specified in the exact recipe.
Do
Enter “10:30” in “Automation setup time”.
Do
Choose “Automation setup model”.
Do
Choose “Automation setup model option deepseek pro”.
Do
Enter “300” in “Automation setup timeout”.
Do
Choose “Automation setup save”.
Do
Reload the page.

Expect

Frequency offers Once, Hourly, Daily, Weekdays (Mon–Fri), Weekly and Custom; Weekly adds Day of week and At; they are saved as a cron expression and read back after a reload.

Exact CLI recipe & bug checks

On the edit page run control-openhands browser click 'testid=automation-setup-frequency-weekly', control-openhands browser select 'testid=automation-setup-weekday' 3 (Wednesday; options 0:Sunday … 6:Saturday), control-openhands browser fill 'testid=automation-setup-time' 10:30, control-openhands browser click 'testid=automation-setup-model', control-openhands browser click 'testid=automation-setup-model-option-deepseek-pro', control-openhands browser fill 'testid=automation-setup-timeout' 300 and control-openhands browser screenshot --feature F23.edit-schedule --name weekly-form. Then control-openhands browser network --clear, control-openhands browser click 'testid=automation-setup-save' --observe 'testid=automation-setup-save-state' --observe-ms 2000 (Saving… → Saved just now) and control-openhands browser toasts --history (Automation updated). browser network --bodies --filter 'automation/v1' shows one PATCH /api/automation/v1/<id> with schedule 30 10 * * 3, model deepseek-pro and timeout 300. After control-openhands browser reload the weekday is 3, At 10:30, the model deepseek-pro and Timeout 300; the detail page Configuration reads Schedule 30 10 * * 3 and LLM profile deepseek-pro.

Cleared timeF23.edit-time-clearedKnown failure

Scenario F23--cleared-time · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Enter “” in “Automation setup time”.

Expect

clearing At on a preset schedule and saving should be refused with a message instead of being dropped or replaced.

Known bug observations

Known failure (reproduced 2026-10-13, #17948 item, new symptom since #17688): browser network --bodies --filter 'automation/v1' shows a PATCH with schedule 0 0 * * 3, the toast reads Automation updated and the automation now runs at midnight. Save draft in the create flow (/automations/setup) stores 0 0 * * * the same way; open PR #18260 targets both paths in automation-setup-panel.tsx at head 0f812ac68902d2c71896568311bffb8aa6f216a6 (checked 2026-10-15), but is not merged.

Known failure (reproduced 2026-10-13, #17948 item, new symptom since #17688): Save silently stores midnight (0 0 * * 3) and toasts Automation updated; Save draft in the create flow stores 0 0 * * * the same way. Open PR #18260 (head 0f812ac68902d2c71896568311bffb8aa6f216a6, checked 2026-10-15) targets the live setup panel: it rejects an invalid or cleared time before Save and Save draft, with an edit-mode regression test. It is not merged; the known failure above remains the observed behavior on this map’s baseline.

Exact CLI recipe & bug checks

On the weekly edit page control-openhands browser fill 'testid=automation-setup-time' '', control-openhands browser network --clear, click Save. Expected: an inline message and nothing saved. Known failure (reproduced 2026-10-13, #17948 item, new symptom since #17688): browser network --bodies --filter 'automation/v1' shows a PATCH with schedule 0 0 * * 3, the toast reads Automation updated and the automation now runs at midnight. Save draft in the create flow (/automations/setup) stores 0 0 * * * the same way; open PR #18260 targets both paths in automation-setup-panel.tsx at head 0f812ac68902d2c71896568311bffb8aa6f216a6 (checked 2026-10-15), but is not merged.

Custom cronF23.edit-cronMapped behavior

Scenario F23--custom-cron · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open QA_F23 Fail and Edit: Frequency Custom is selected and control-openhands browser value 'testid=automation-setup-custom-schedule' is 0 0 1 1 * beside the timezone select (UTC). Fill 0 0 2 1 * and testid=automation-setup-timeout with 1 (this makes the next run fail), Save and browser toasts --history (Automation updated); after a reload of the detail page the Configuration shows 0 0 2 1 *. Invalid crons: see Validation.

Expect

a custom cron schedule shows Frequency Custom with an editable Custom cron field and a timezone select.

Exact CLI recipe & bug checks

Open QA_F23 Fail and Edit: Frequency Custom is selected and control-openhands browser value 'testid=automation-setup-custom-schedule' is 0 0 1 1 * beside the timezone select (UTC). Fill 0 0 2 1 * and testid=automation-setup-timeout with 1 (this makes the next run fail), Save and browser toasts --history (Automation updated); after a reload of the detail page the Configuration shows 0 0 2 1 *. Invalid crons: see Validation.

Failed run, its row and Debug with OpenHandsF23.run-status-polling · F23.run-no-conversation · F23.run-logs-modal · F23.debug-with-openhandsKnown failureKnown issue

Scenario F23--failed-run-its-row-and-debug-with-openhands · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Wait
Wait for the element identified in the CLI recipe to appear.
Do
Open /automations.
Wait
Wait for “Automation activity log” / “Run status icon failed” to appear.
Do
Choose the element identified in the CLI recipe.
Do
Choose “Debug automation”.

Expect

a finished run without a conversation reads No Conversation; only a script run (no agent) gets "No conversation — this run executed a script. Use View logs for its output."

in the Logs dialog of a Failed run, Debug with OpenHands starts a conversation seeded with the automation, the error and the run id, and opens it.

Linked known issues

Exact CLI recipe & bug checks

On QA_F23 Fail (timeout 1 from the previous bullet) click role=button[name="Run now"] and control-openhands browser wait-text 'Automation dispatched'. The run stays Running for about 3-4 minutes and ends FAILED (poll control-openhands api GET /api/automation/v1/<id>/runs; drive the event automation bullets meanwhile). Then control-openhands browser goto /automations, open QA_F23 Fail, control-openhands browser wait 'testid=automation-activity-log >> testid=run-status-icon-failed' and control-openhands browser text 'testid=automation-activity-log': the newest row reads <time>, Timed out: command timed out or was killed, Failed. Expected beside the time: No Conversation (this is a prompt automation); today it reads No conversation — this run executed a script. Use View logs for its output. Run control-openhands browser click 'testid=automation-activity-log >> title=View logs >> nth=0' and control-openhands browser snapshot 'role=dialog': Run FAILED, Task No task outcome reported., System Error: Timed out: command timed out or was killed and Status detail: ..., the Output tab with the [setup] Fetching SDK version ... lines, and button Debug with OpenHands. Screenshot with --feature F23.debug-with-openhands --name failed-logs. control-openhands browser click 'testid=debug-automation-button' --expect-url '/conversations/' opens a new conversation; control-openhands conversation events <conversation id from browser url> --kinds MessageEvent starts The scheduled automation "QA_F23 Fail" failed during a run. Please investigate the error and fix the root cause., and control-openhands browser eval "document.body.innerText.includes('Timed out: command timed out or was killed')" is true (the page also shows the run id). The agent then works on that prompt with the active profile.

Task outcome badgeF23.run-task-outcomeMapped behavior

Scenario F23--task-outcome-badge · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Arrange the prerequisite backend state through the API.
Do
Choose the element identified in the CLI recipe.

Expect

a Completed run whose agent finished with a task outcome other than success shows that outcome instead of Successful: blocked and partial_success read Blocked / Partial with run-status-icon-warning, unknown and any unrecognised status string read Needs review (run-status-icon-needs-review), and a finish response without a string status (summary only, custom fields only, a non-string status, or not an object) is expected to read Successful, and an outcome_summary, when it has one, is still shown (#17682); the row summary is the agent's outcome_summary, and the Logs dialog's Task section repeats it.

Exact CLI recipe & bug checks

Arrange (API, arrange only) control-openhands api POST /api/automation/v1/preset/prompt --write --data '{"name":"QA_F23 Blocked","prompt":"Do not run any tools. Call the finish tool right away with status \"blocked\" and outcome_summary \"QA blocked on purpose\".","trigger":{"type":"cron","schedule":"0 0 1 1 *","timezone":"UTC"},"enabled":true}' (a good thing to do while the Fail run is still running). Open QA_F23 Blocked, click role=button[name="Run now"], and poll control-openhands browser text 'testid=automation-activity-log' until it leaves Running (about 35 s): <time>, QA blocked on purpose, a cost such as $0.0016, Blocked; control-openhands browser testids 'testid=automation-activity-log' lists run-status-icon-warning (not -completed) although api GET /api/automation/v1/<id>/runs has status COMPLETED and run_metadata.finish_tool_response.status blocked. control-openhands browser click 'testid=automation-activity-log >> title=View logs' and browser snapshot 'role=dialog': Run COMPLETED, Task Blocked / QA blocked on purpose / Task metadata { "message": ... }. Screenshot with --feature F23.run-task-outcome --name blocked, Escape. partial_success (Partial, run-status-icon-warning) and unknown (Needs review, run-status-icon-needs-review) follow the same path with the same prompt shape. The normal automation 1.19.0 prompt/plugin presets configure SDK 1.53.0 FinishTool with the required TaskOutcome.status, limited to success, partial_success, blocked, failed and unknown; SDK custom response schemas are a separate producer boundary. Its normal prompt path therefore does not establish the new fallback for a finish response without a string status. That positive boundary needs a genuine completed run from a legacy or custom-schema producer whose real finish arguments have that shape; a run with no finish metadata checks only the ordinary Completed fallback. Arbitrary unrecognised strings and summary retention on a failed run need the corresponding genuine producer too. Until available, record those checks as blocked and do not treat injected run metadata or mocked LLM output as live proof (#17682).

Event automationF23.prompt-collapse · F23.configuration-event · F23.plugins-repo · F23.exportKnown issue

Scenario F23--event-automation · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Show more”.
Do
Choose “Automation actions”.
Do
Choose the list / containing “Tarball” / “Export”.

Expect

a prompt taller than about 10 lines is clipped to 240 px with a fade (automation-prompt-fade) and View More; View Less collapses it again.

an event automation shows Trigger Event, Event Source, Event Type and Filter, truncated at 60 characters with Show more/Show less.

an automation created with repositories and plugins shows Repositories (one owner/repo row each, with a branch badge for its ref) and a Plugins card with one chip per plugin.

Linked known issues

Exact CLI recipe & bug checks

Open QA_F23 Event. control-openhands browser bbox 'testid=automation-prompt-content' is 240 px high, testid=automation-prompt-fade is present and control-openhands browser text 'testid=automation-prompt-toggle' is View More; screenshot with --feature F23.prompt-collapse --name collapsed. Click the toggle: height 360, no fade, text View Less; click again to collapse. control-openhands browser text 'role=heading[name="Configuration"] >> xpath=ancestor::div[2]' reads Repositories qa-example/qa-repo main, Trigger Event, Event Source github, Event Type pull_request.opened, Filter pull_request.draft == `false` && pull_request.base.ref == re…, Show more, LLM profile Active profile. control-openhands browser click 'role=button[name="Show more"]' shows the full filter and control-openhands browser attr 'role=button[name="Show less"]' aria-expanded is true. control-openhands browser text 'testid=automation-repository' is qa-example/qa-repo / main (the fixture's https://github.com/qa-example/qa-repo as owner/repo, with its branch badge) and control-openhands browser text 'role=heading[name="Plugins"] >> xpath=ancestor::div[2]' is Plugins / github:qa-example/qa-plugin; screenshot with --feature F23.plugins-repo --name event-repo-plugins. Export keeps them: control-openhands browser click 'role=button[name="Automation actions"]', control-openhands browser click 'role=list >> has-text=Tarball >> role=button[name="Export"]' and control-openhands browser downloads --last 1 --contains '"branch": "main"' (qa-f23-event.automation.json, contains true); --contains '"repository": "https://github.com/qa-example/qa-repo"' and --contains 'github:qa-example/qa-plugin' are true too (the --inspect head ends inside the long prompt).

Event trigger in EditF23.edit-event-readonlyMapped behavior

Scenario F23--event-trigger-in-edit · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
On QA_F23 Event open Edit: Event is selected, browser testids 'testid=automation-setup-form' --filter event lists automation-setup-event-source (GitHub), automation-setup-event-source-toggle, automation-setup-event-public-url-notice, automation-setup-event-key, automation-setup-event-filter and automation-setup-event-filter-remove (the fixture has a filter), and there are no Frequency controls for a schedule. The trigger is editable here (it was read-only text in the old modal). Screenshot with --feature F23.edit-event-readonly --name event, then Close.

Expect

for an event automation the edit page shows Event selected with Event source, event key and filter controls (editable now, no longer read-only text) and no schedule fields.

Exact CLI recipe & bug checks

On QA_F23 Event open Edit: Event is selected, browser testids 'testid=automation-setup-form' --filter event lists automation-setup-event-source (GitHub), automation-setup-event-source-toggle, automation-setup-event-public-url-notice, automation-setup-event-key, automation-setup-event-filter and automation-setup-event-filter-remove (the fixture has a filter), and there are no Frequency controls for a schedule. The trigger is editable here (it was read-only text in the old modal). Screenshot with --feature F23.edit-event-readonly --name event, then Close.

PhoneF23.phoneMapped behavior

Scenario F23--phone · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Use the phone viewport.
Setup
Use the desktop viewport.

Expect

at 390×844 the detail page has no horizontal overflow, and the Edit page fits with its title, Test, Save and Close reachable.

Exact CLI recipe & bug checks

Run control-openhands browser viewport phone, open QA_F23 Pong, control-openhands browser bbox 'role=heading[name="QA_F23 Pong"] >> xpath=ancestor::div[3]' (insideViewport true, pageHorizontalOverflow false) and control-openhands browser screenshot --feature F23.phone --name detail (the name and Run now wrap). Open Edit (kebab, --expect-url '/automations/setup'), then control-openhands browser bbox 'testid=automation-setup-save' and ... 'testid=automation-setup-close' (both insideViewport true, Save at y about 65) and control-openhands browser bbox 'testid=automation-setup-form' (pageHorizontalOverflow false); screenshot with --feature F23.phone --name edit-setup-page. Close and control-openhands browser viewport desktop.

Run now and save errorsF23.run-now-error · F23.edit-save-errorKnown failure

Scenario F23--run-now-and-save-errors · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Arrange the prerequisite backend state through the API.
Wait
Wait for “QA_F23 Fail” to disappear.
Setup
Arrange the prerequisite backend state through the API.
Setup
Arrange the prerequisite backend state through the API.
Do
Hover over “Automation setup name”.

Expect

a failed dispatch shows exactly one error toast with the API message; when the automation is gone (Automation not found), the page then shows the Automation not found state.

a failed save shows exactly one error toast with the API message (Automation not found for an automation deleted meanwhile), the save state reads Save failed and the edits stay on the page; the toolbar Test on that page also toasts Automation not found.

Known bug observations

Known failure: the wait times out and the page keeps heading QA_F23 Fail, Active and an enabled Run now (control-openhands browser enabled 'role=button[name="Run now"]' is true) until a reload (#18062). The save error does not reuse that page (it is gone once #18062 is fixed): arrange a second fixture, control-openhands api POST /api/automation/v1/preset/prompt --write --data '{"name":"QA_F23 Stale","prompt":"Reply with the single word: pong.","trigger":{"type":"cron","schedule":"0 0 1 1 *","timezone":"UTC"},"enabled":false}' (body.id is <stale-id>), open QA_F23 Stale, its kebab and role=list >> has-text=Tarball >> role=button[name="Edit"] (--expect-url '/automations/setup'). With the edit page open, control-openhands api DELETE /api/automation/v1/<stale-id> --write and control-openhands browser toasts --history --clear. Fill testid=automation-setup-name with QA_F23 Stale renamed and click testid=automation-setup-save: the page keeps the edit (browser value 'testid=automation-setup-name' still QA_F23 Stale renamed) and browser toasts --history holds exactly one toast, Automation not found. Screenshot with --feature F23.edit-save-error --name toast. Move the pointer off the toast (browser hover 'testid=automation-setup-name', about 5 s), browser toasts --history --clear and click testid=automation-setup-test: again exactly one toast, Automation not found. Close. browser errors --app-only lists the induced POST /api/automation/v1/<Fail id>/dispatch, PATCH /api/automation/v1/<stale-id> and POST /api/automation/v1/<stale-id>/dispatch 404s; both fixtures are already deleted.

Exact CLI recipe & bug checks

Arrange a stale page: open QA_F23 Fail, then control-openhands api DELETE /api/automation/v1/<Fail id> --write. Run control-openhands browser toasts --history --clear (empties the toast history), click role=button[name="Run now"], then control-openhands browser toasts --history and control-openhands browser screenshot --feature F23.run-now-error --name toasts: exactly one toast, Automation not found, and no Request failed with status code 404. Expected: the page then shows Automation not found: control-openhands browser wait 'role=heading[name="QA_F23 Fail"]' --state detached --timeout 15000 succeeds (the not-found state takes about 7 s, as in States) and control-openhands browser snapshot 'role=paragraph >> text=Automation not found >> xpath=..' lists link Back to Automations (do not use wait-text 'Automation not found' here: the toast has the same text). Once the heading is gone, run control-openhands browser toasts --history again: it still holds only Automation not found (a late second toast, Request failed with status code 404, would come from the page's own refetch of the deleted automation). Known failure: the wait times out and the page keeps heading QA_F23 Fail, Active and an enabled Run now (control-openhands browser enabled 'role=button[name="Run now"]' is true) until a reload (#18062). The save error does not reuse that page (it is gone once #18062 is fixed): arrange a second fixture, control-openhands api POST /api/automation/v1/preset/prompt --write --data '{"name":"QA_F23 Stale","prompt":"Reply with the single word: pong.","trigger":{"type":"cron","schedule":"0 0 1 1 *","timezone":"UTC"},"enabled":false}' (body.id is <stale-id>), open QA_F23 Stale, its kebab and role=list >> has-text=Tarball >> role=button[name="Edit"] (--expect-url '/automations/setup'). With the edit page open, control-openhands api DELETE /api/automation/v1/<stale-id> --write and control-openhands browser toasts --history --clear. Fill testid=automation-setup-name with QA_F23 Stale renamed and click testid=automation-setup-save: the page keeps the edit (browser value 'testid=automation-setup-name' still QA_F23 Stale renamed) and browser toasts --history holds exactly one toast, Automation not found. Screenshot with --feature F23.edit-save-error --name toast. Move the pointer off the toast (browser hover 'testid=automation-setup-name', about 5 s), browser toasts --history --clear and click testid=automation-setup-test: again exactly one toast, Automation not found. Close. browser errors --app-only lists the induced POST /api/automation/v1/<Fail id>/dispatch, PATCH /api/automation/v1/<stale-id> and POST /api/automation/v1/<stale-id>/dispatch 404s; both fixtures are already deleted.

Load more runsF23.activity-log-moreNot run

Scenario F23--load-more-runs · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation activity log” / “Load more”.

Expect

with more than 20 runs a Load more button grows the page by 20.

Exact CLI recipe & bug checks

Not run: it needs more than 20 runs of one automation (about 35 s and one model call each). With them, control-openhands browser click 'testid=automation-activity-log >> role=button[name="Load more"]' grows the list from 20 to 40 rows.

Script automationF23.script-sectionMapped behavior

Scenario F23--script-automation · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/templates.
Do
Choose “Recommended automation card news digest”.
Do
Enter “0 0 1 1 *” in “Setup field schedule”.
Do
Choose “Setup continue”.
Wait
Wait for “Setup review” to appear.
Do
Choose “Setup continue”.

Expect

a script (tarball) automation shows Script with the entrypoint and the bundle's files in place of the Prompt.

Exact CLI recipe & bug checks

Arrange a script automation through F22's bundle entry (UI arrange; F22 owns the creation proof): control-openhands browser goto /automations/templates, control-openhands browser click 'testid=recommended-automation-card-news-digest' --expect-url '/automations/new/news-digest', control-openhands browser fill 'testid=setup-field-schedule' '0 0 1 1 *', control-openhands browser click 'testid=setup-continue-button', control-openhands browser wait 'testid=setup-review' --timeout 10000, control-openhands browser click 'testid=setup-continue-button' --expect-url '/automations/[0-9a-f-]{8,}' --timeout 60000. The detail page of Daily news digest shows role=heading[name="Script"] and no Prompt (browser count 'role=heading[name="Prompt"]' is 0); control-openhands browser text 'testid=automation-script-entrypoint' is python3 main.py, and browser eval "Array.from(document.querySelectorAll('[data-testid=automation-script-file]')).map(e=>e.innerText.split('\n').slice(0,2).join(' | '))" lists main.py | Entrypoint first, then config.json (both with line-numbered source). Screenshot with --feature F23.script-section --name news-digest. automation-script-unavailable (bundle unreadable) was not reached. Do not Run now: the script fetches public feeds and starts a model-backed conversation. Delete it in the Delete bullet.

Edit-mode Test and unsaved editsF23.edit-test · F23.edit-dirty-stateMapped behavior

Scenario F23--edit-mode-test-and-unsaved-edits · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Hover over “Automation setup prompt”.

Expect

on the edit page Test dispatches the automation as last saved (POST /api/automation/v1/<id>/dispatch, no PATCH first) and shows Test run started. on the form; unsaved edits are not part of that run. Whether it should test the current form is a product decision (Gotchas).

the edit page shows no save state for unsaved edits (no Unsaved changes, which create mode shows, F28); the label appears only while saving (Saving…) and after (Saved just now or Save failed). Close discards the edits.

Exact CLI recipe & bug checks

Open QA_F23 Event and its kebab Edit (--expect-url 'automationId='). Fill testid=automation-setup-name with QA_F23 Unsaved: control-openhands browser count 'testid=automation-setup-save-state' is 0. control-openhands browser network --clear, click testid=automation-setup-test; control-openhands browser network --filter 'automation/v1' shows POST …/<id>/dispatch 201 and no PATCH, and the form shows Test run started. Move the pointer off the toast (control-openhands browser hover 'testid=automation-setup-prompt') and click testid=automation-setup-close (--expect-url '/automations$'): the card still reads QA_F23 Event. The dispatched run is not checked here; Delete below removes it with the automation.

DeleteF23.deleteMapped behavior

Scenario F23--delete · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Delete automation” / “Cancel”.
Do
Choose “Delete automation” / “Delete”.

Expect

kebab Delete asks Delete "<name>"? This cannot be undone.; Cancel keeps it, Delete removes it and returns to /automations.

Exact CLI recipe & bug checks

Open QA_F23 Event, click the kebab and role=list >> has-text=Tarball >> role=button[name="Delete"]. control-openhands browser snapshot 'role=dialog[name="Delete automation"]' shows Delete "QA_F23 Event"? This cannot be undone., Cancel and Delete (the same dialog as F21.delete). control-openhands browser click 'role=dialog[name="Delete automation"] >> role=button[name="Cancel"]' keeps the page (control-openhands browser count 'role=dialog[name="Delete automation"]' is 0). Reopen and control-openhands browser click 'role=dialog[name="Delete automation"] >> role=button[name="Delete"]' --expect-url '/automations(\?|$)'; control-openhands api GET /api/automation/v1 no longer lists it. Delete QA_F23 Pong, QA_F23 Blocked and Daily news digest the same way (and QA_F23 Fail if the error bullet was skipped); api GET /api/automation/v1 shows "total": 0.

ErrorsF23 · setup / cleanupSupport step

Scenario F23--errors · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
After each group control-openhands browser errors --app-only shows no page errors; the only app errors are the induced ones (404s for the unknown id and the deleted fixture, 502s while the service is stopped).

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

After each group control-openhands browser errors --app-only shows no page errors; the only app errors are the induced ones (404s for the unknown id and the deleted fixture, 502s while the service is stopped).

UTF-8 script memberF23.script-sectionKnown failure

Scenario F23--utf-8-script-member · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Arrange an inactive annual-cron script automation through the real upload/create APIs with a two-file UTF-8 USTAR bundle (tarfile.USTAR_FORMAT, names café.py and main.py) and entrypoint python3 café.py; do not use a PAX-only member name. Open /automations/<id> and inspect automation-script-entrypoint, both automation-script-file headings, and the screenshot. Expected: café.py keeps its name, receives the Entrypoint badge and sorts first. Known failure, reproduced 2026-10-12 on fa1f724: the command is correct, but the file heading is cafÃ©.py without the badge; the ASCII control reads normally (#17704). No run or model is needed.

Expect

a script (tarball) automation shows Script with the entrypoint and the bundle's files in place of the Prompt.

Known bug observations

Known failure, reproduced 2026-10-12 on fa1f724: the command is correct, but the file heading is cafÃ©.py without the badge; the ASCII control reads normally (#17704). No run or model is needed.

Exact CLI recipe & bug checks

Arrange an inactive annual-cron script automation through the real upload/create APIs with a two-file UTF-8 USTAR bundle (tarfile.USTAR_FORMAT, names café.py and main.py) and entrypoint python3 café.py; do not use a PAX-only member name. Open /automations/<id> and inspect automation-script-entrypoint, both automation-script-file headings, and the screenshot. Expected: café.py keeps its name, receives the Entrypoint badge and sorts first. Known failure, reproduced 2026-10-12 on fa1f724: the command is correct, but the file heading is cafÃ©.py without the badge; the ASCII control reads normally (#17704). No run or model is needed.

Live free-form phaseF23.run-status-polling · F23.activity-logKnown failureKnown issue

Scenario F23--live-free-form-phase · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Upload an owned bounded script, create it active with the inert yearly cron, open its detail page, and click Run now. While genuinely running, compare api GET /api/automation/v1/<id>/runs with browser snapshot 'testid=automation-activity-log'. Expected: a run's current_phase renders as its live/last phase when structured fields are absent. Known failure, reproduced 2026-10-12 against automation 1.19.3: the API says Starting automation with no phase_code/phase_label, but Activity Log shows only Running (#17121). Keep the worker bounded and clean up only its owned command; the script needs no model.

Expect

Known bug observations

Known failure, reproduced 2026-10-12 against automation 1.19.3: the API says Starting automation with no phase_code/phase_label, but Activity Log shows only Running (#17121). Keep the worker bounded and clean up only its owned command; the script needs no model.

Linked known issues

Exact CLI recipe & bug checks

Upload an owned bounded script, create it active with the inert yearly cron, open its detail page, and click Run now. While genuinely running, compare api GET /api/automation/v1/<id>/runs with browser snapshot 'testid=automation-activity-log'. Expected: a run's current_phase renders as its live/last phase when structured fields are absent. Known failure, reproduced 2026-10-12 against automation 1.19.3: the API says Starting automation with no phase_code/phase_label, but Activity Log shows only Running (#17121). Keep the worker bounded and clean up only its owned command; the script needs no model.

Custom nested setup (`F23.script-section`, `F23.run-logs-modal`; creation in F22)F23.script-section · F23.run-logs-modalKnown failureKnown issue

Scenario F23--custom-nested-setup-f23-script-section-f23-run-logs-modal-creation-in-f22 · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Arrange the prerequisite backend state through the API.

Expect

a script (tarball) automation shows Script with the entrypoint and the bundle's files in place of the Prompt.

Known bug observations

Known failure, reproduced 2026-10-12 against automation 1.19.3: the real BashOutput exits 1 with the marker assertion, proving the setup step was skipped (OpenHands/automation#343). Only an owned marker is written; no external service or model is used.

Linked known issues

Exact CLI recipe & bug checks

Arrange a reviewed tarball containing scripts/setup.sh that writes .qa-setup-marker and main.py that asserts that marker exists. Upload with api POST /api/automation/v1/uploads?name=qa-nested-setup --write --data-file <tarball> --content-type application/gzip; create with entrypoint: python3 main.py, setup_script_path: scripts/setup.sh, active state and yearly cron. Open the detail page and click Run now; correlate its bash_command_id with api GET '/api/bash/bash_events/search?command_id__eq=<id>&limit=100'. Expected: the configured nested setup runs before the entrypoint. Known failure, reproduced 2026-10-12 against automation 1.19.3: the real BashOutput exits 1 with the marker assertion, proving the setup step was skipped (OpenHands/automation#343). Only an owned marker is written; no external service or model is used.

Selected profile on a prompt presetF23.configuration · F23.run-linked-conversationKnown failureKnown issueBlocked

Scenario F23--selected-profile-on-a-prompt-preset · Permalink · Source

Given

Use F23’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Arrange a bare saved Agent Profile (tools: [], unique persona) referencing an offline LLM profile whose base URL is the closed loopback http://127.0.0.1:9; set zero retries and disable title generation. Create an owned active annual-cron prompt preset, PATCH its agent_profile_id to the bare profile, then click Run now in Canvas. Inspect the linked conversation's launched_agent_profile, tool names and persona without revealing secrets. Expected: the chosen profile governs that conversation. Known failure, reproduced 2026-10-12 against automation 1.19.3/Agent Server 1.54.0: provenance is null, the selected persona is absent, and default terminal/file-editor/task-tracker/finish tools replace the bare selection (OpenHands/automation#557). The deliberate loopback connection failure is incidental to this metadata check; no provider is reachable.

Expect

a cron automation shows Trigger Schedule, the Schedule and LLM profile (Active profile or the profile name).

a run row with a conversation is a link labelled View conversation for run at <time> to /conversations/<conversation_id>, whose title is <automation name> — <UTC timestamp>.

Known bug observations

Known failure, reproduced 2026-10-12 against automation 1.19.3/Agent Server 1.54.0: provenance is null, the selected persona is absent, and default terminal/file-editor/task-tracker/finish tools replace the bare selection (OpenHands/automation#557). The deliberate loopback connection failure is incidental to this metadata check; no provider is reachable.

Linked known issues

Exact CLI recipe & bug checks

Arrange a bare saved Agent Profile (tools: [], unique persona) referencing an offline LLM profile whose base URL is the closed loopback http://127.0.0.1:9; set zero retries and disable title generation. Create an owned active annual-cron prompt preset, PATCH its agent_profile_id to the bare profile, then click Run now in Canvas. Inspect the linked conversation's launched_agent_profile, tool names and persona without revealing secrets. Expected: the chosen profile governs that conversation. Known failure, reproduced 2026-10-12 against automation 1.19.3/Agent Server 1.54.0: provenance is null, the selected persona is absent, and default terminal/file-editor/task-tracker/finish tools replace the bare selection (OpenHands/automation#557). The deliberate loopback connection failure is incidental to this metadata check; no provider is reachable.

Family notes, prerequisites for known issues & limits Known failureKnown issueBlockedNot run

The automations QA checklist (F28) expects edit-mode Test to run the current form state; the panel calls dispatchAutomation(<id>), which runs the saved automation. F23.edit-test records current behavior; whether it should change is a product decision, not a map correction.
The Logs dialog's panel keeps long content reachable (F23.run-logs-modal, #18175 / #18173). Re-verified 2026-10-13 (revision 2bdc0e9d, SDK 1.54.0, automation 1.19.3) with a 30-line outcome_summary at 1440×1000, 390×844 and 390×500: the panel is testid=run-logs-body (max-h-[80vh], overflow-y: auto) with a sticky header (Logs + X stay visible); the Output box no longer scrolls on its own, the whole panel does, and its end (=== RESULT === ALL_OK) is reachable at the bottom. Verified 2026-10-10 on an independent fresh Linux Local stack at product revision 5a8c3832154b14d2dad651801c0ab3f74c96f02e, SDK 1.54.0 and automation 1.19.3: a genuine DeepSeek finish produced a 4,596-character outcome_summary and 1,079-character metadata message; summary/metadata endings, Output/Error tabs and the true stdout end were reachable at 1440×1000, 390×844, 1440×500 and 390×500. The header/X remained visible, closing did not navigate, and phone had no horizontal overflow. Preserve those long-content and short-height checks when retesting. The verified scope is pinned Linux Local; macOS, Cloud and legacy/custom finish schemas need their own checks. browser bbox 'role=dialog' measures the full-screen wrapper, not the panel: measure the panel with browser eval.
The detail page has no main landmark: browser snapshot 'role=main' times out. Snapshot the page, or scope by section headings (role=heading[name="Configuration"] >> xpath=ancestor::div[2]).
The kebab menu has no test ids; role=list >> has-text=Tarball scopes the detail kebab (the dashboard kebab has no Tarball, so F21 uses has-text=Export).
Kebab Edit (detail and dashboard) opens /automations/setup?automationId=<id>. The edit page's Save and Test report API failures with the API's message (see F23.edit-validation, F23.edit-save-error), and its Automation updated toast covers Test/Save/Close until the pointer moves away.
Known defect: clearing At on a preset schedule and saving stores midnight with no message (#17948 item; with the old modal it was silently ignored).
On a local backend the Logs dialog reads every run's output from the server-level /api/bash/bash_events/search, with or without a conversation. It caches what it fetched: reload before checking its requests with browser network.
Known defect: prompt runs also have a bash_command_id, so every row shows the logs icon, and a prompt run that ended without a conversation is labelled as a script run (F23.run-no-conversation) (#17948).
The automation service (1.19.0 here) keeps repos and plugins only under preset_metadata; the UI reads them from there. Repositories show owner/repo without the host (a GitLab URL reads like a GitHub one), a repo without a ref gets no badge, and Export keeps only the first repository (all plugins). The Notification field reads a top-level notification that the service has no field for, so it never renders on a local stack (#16644 tracks removing or mapping it).
The Schedule field shows the raw cron (0 9 * * *) and no timezone: the service returns no schedule_human and no top-level timezone, so the Edit dialog also omits its Timezone: hint.
run-phase never appeared in the log here (phase_code stays null on the local service), so the phase chip is not reachable on this stack.
An LLM profile that does not exist (model: "qa-no-such-profile") does not fail the run: it ran and succeeded with the active profile. Use timeout 1 to get a failed run; it takes several minutes to fail.
Load more in the log (F23.activity-log-more) needs more than 20 runs of one automation; at about 35 s and one model call each it is left out of a normal pass (not-run).
The script section needs a tarball automation. The account-free way to get one is F22's Daily news digest bundle entry (it uploads its bundle and creates the automation); the custom-automation Upload tarball action takes control-openhands fixture tarball. Its kebab Edit shows generated starter Python instead of the saved bundle (#18341, see F21.edit); a cold load of /automations/setup?automationId=<id> shows the Existing code bundle summary.
Starting… on Run now lasts only for the dispatch request (under 50 ms locally); --observe does not catch it.
The run row is a plain <a href>, so opening a run's conversation reloads the whole app (a document request in browser network), unlike sidebar navigation.
A 404 detail request is retried three times before Automation not found shows (about 7 s of skeleton) (#17948); wait with wait-text, not count.
browser fill on the Timeout spinbutton with letters fails in Playwright and, after its 30 s retry, the dialog was found closed; use numeric values (1.5, 0, 1801) to reach the timeout errors.
The Tarball download is named <name>.tar (spaces become _) but is gzip-compressed (#17948); browser downloads --inspect does not list tar entries, so inspect it with tar -tzf.
Debug with OpenHands starts a real agent with the openhands-automation skill and the automation API key. It acts on live automations: here it looked up QA_F23 Fail by name (a newer fixture with the same name), PATCHed its timeout back to 600 and dispatched it, which adds runs and conversations. Stop it once it has started: on its page control-openhands browser click 'testid=stop-button', then control-openhands conversation wait <id> --until paused,finished,idle --timeout 45 (paused).
A successful run leaves a conversation <name> — <UTC timestamp> that stays after the automation is deleted (conversations family owns deleting it).
Known issue #15465 (F23.configuration-event F23.activity-log): An automation configured for Agent Canvas release events never runs automatically, although Run now works. Recheck a real signed GitHub/Forgejo release webhook through an externally reachable responder into a disposable automation and inspect its activity log; local manual dispatch cannot verify event delivery. No external repository webhook credentials or controlled release-event fixture are available.
Known issue OpenHands/software-agent-sdk#5279 (F23.run-linked-conversation): An automation that attaches to an existing remote conversation can complete with no conversation link in its run, because the workspace callback does not register the attached id. Recheck a Cloud automation script that calls RemoteConversation.attach on an owned stable conversation and exits through the real workspace completion callback, then inspect the Activity Log link. Blocked without that authenticated Cloud workspace/callback context; a plain local GET is not callback proof.
Known issue OpenHands/software-agent-sdk#4705 (F23.run-status-polling, F23.run-logs-modal): GitHub MCP tool events can omit inputSchema, causing an automation's RemoteConversation decoder/run log to report failure while the agent continues, so task and automation statuses disagree. Recheck a real GitHub-MCP-enabled run using the affected client/server versions and its retained tool-definition event; the issue does not give those versions. A generic MCP fixture does not establish the reported GitHub/version branch, and genuine run comparison needs a model key.
Known issue OpenHands/software-agent-sdk#4260 (F23.run-linked-conversation, F23.run-logs-modal): Cloud PR-review automation can remain at review in progress and time out without posting a review; its event webhook also fails to serialize SecretStr, breaking the UI stream even on successful reviews. Keep the stream-serialization failure distinct from the empty-response/600-second run-budget failure. Recheck with an owned Cloud reviewer/webhook deployment, authorized disposable review target and real model credentials; a local script run proves neither production path.
Known issue OpenHands/automation#286 (F23.run-status-polling): Multi-worker/replica automation deployments can run duplicate watchdogs that query the same sandbox and hit rate limits, falsely failing runs. Recheck the real deployment's worker/replica configuration with an owned long-running Cloud run and sandbox verification traffic; this single-worker local stack cannot prove distributed scheduler/watchdog ownership.
Known issue OpenHands/automation#219 (F23.run-status-polling): Starting a Cloud automation at the user's sandbox cap can pause an existing interactive session instead of reaching the graceful Skipped–Limit Reached path. Recheck only on an owned disposable Cloud account at its recorded cap, with a live interactive fixture and a new automation run, then observe both run and conversation states. This run has no Cloud account/cap fixture; do not evict a production session as a test.
Known issue OpenHands/automation#113 (F23.run-linked-conversation): Scheduled plugin jobs can start conversations visible in the archived list while the automation run UI says no conversation started. Recheck the affected automation/run ids, archived conversation ids and deployment version, using the real callback relationship; these identifiers and version are not provided. This is a reported missing-link state, not proof that no agent ran.
Known failure OpenHands/automation#514 spans Home cancellation (F03.cancel-in-flight-run) and F23.activity-log/F23.run-logs-modal: on 2026-10-12 a genuine Home Cancel run changed the owned bounded local worker's row to Cancelled, while its Python entrypoint remained alive in the owned launcher ancestry. The F03 driving bullet owns upload/create/cancel steps and retained evidence. The worker later reached its 180 s bound and exited; a cancelled status alone does not prove process cleanup.
Known issue OpenHands/software-agent-sdk#4388 (F23.run-status-polling, F23.run-logs-modal): incremental SDK Bash output polling can return empty pages before newer chunks when order__gt is applied after the file-page boundary. Canvas's frontend log reader does not send that filter, but automation launch preparation and prompt/plugin preset repository clones use (Async)RemoteWorkspace.execute_command, whose output poller does. Recheck a bounded genuine preset/setup or repository-clone command that emits enough chunks to cross the 100-event boundary, correlating its final BashOutput and real run result; a frontend Logs read or a one-output command does not cover the incremental branch.
Known issue OpenHands/software-agent-sdk#5937 (F23.run-status-polling, F23.run-logs-modal): prompt/plugin preset workers attach AUTOMATION_CONVERSATION_ID for continued conversations and then call RemoteConversation.run(). If that same conversation is already running a tool for more than 30 s, the join request can time out and fail the automation worker while the agent continues. Recheck an owned continued-conversation automation with a genuine already-running long tool and correlate worker error, agent status and final run result; a fresh conversation or one normal first run() does not exercise the join. No model execution is authorized in this pass.
Known issue OpenHands/software-agent-sdk#5938 (F23.run-status-polling): prompt/plugin preset workers register an event callback that forwards ActionEvent.summary as the live phase. After an events-socket reconnect, REST-reconciled events reach SDK state but are not replayed to that callback, so summaries emitted in the gap can be missed. Recheck an owned real preset worker through a bounded reconnect window while a genuine action is recorded; compare callback-driven phase with stored events. Current Canvas phase rendering has its separate #17121 gap, and the worker's completion metadata is not claimed lost merely because this callback missed an event.

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.