Creating automations: templates, setup and import

F22 · 29 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F22 Creating automations: templates, setup and import

29 behaviors · 39 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

A user adds an automation in three ways. The Templates sub-page (/automations/templates) lists the prebuilt catalog from @openhands/extensions (6 featured, 17 Beta) with search; picking a card either opens a manifest-driven setup dialog at /automations/new/<id> (prerequisites → form → review → Confirm and create), or first asks where a GitHub/Slack responder should run, or first asks the user to install missing MCP integrations, or seeds a conversation with the entry's launch prompt. Add Automation → Create Automation opens the agent-assisted setup page at /automations/setup, where the form opens before a conversation exists. Saving a server-backed draft and testing its generated custom script need no LLM; sending a prompt starts genuine model assistance through automation_form_update. The setup page's own fields, draft validation, Test and Create each have their own IDs in F28. Add Automation → Import automation reads an exported JSON file, previews it and creates it disabled.

How a user gets here

Sidebar Automate (sidebar-automations-link), then Templates in the Automate sub-page nav (automations-navbar-desktop >> automations-navigation-templates; at phone width automations-navbar-mobile). The command menu has no Templates entry (searching template gives No commands found).
Direct URLs /automations/templates and /automations/new/<catalog-id> (for example /automations/new/custom-automation, /automations/new/news-digest, /automations/new/qa-changes). A deep link opens the setup dialog over the page; it bypasses the responder and install-queue dialogs that a card click shows first.
Template cards elsewhere use the same launcher: the Recommended automations rail on home/New Chat (F03) and on the empty dashboard (F21.empty-rail), and the onboarding say-hello step (F01).
Dashboard header Add Automation (automations-add-automation) → Create Automation (automations-add-automation-create) or Import automation (automations-import-automation). The empty dashboard also shows the Create Automation instructions inline (automations-empty >> automations-create-automation).

Given

Preconditions:

Baseline state (launched, doctored, onboard --skip done) and control-openhands llm preset deepseek for model/profile-dependent checks. Agent assistance, prompt/plugin Test and Run now use genuine deepseek-flash within the run's authorized model budget. The Open and save a draft, Credential-free script test and Resume at phone width bullets may start on an unconfigured fresh stack without this preset.
No automation named QA_F22 … or Daily news digest (control-openhands api GET '/api/automation/v1?limit=100'), no OPENHANDS_URL secret (control-openhands api GET /api/settings/secrets) and no MCP server installed (control-openhands api GET /api/settings, mcp_config empty). An existing OPENHANDS_URL skips the secret creation in the responder recipe; an installed Slack/Linear/Tavily server skips its install dialog.
Import fixtures (arrange only; each command prints the path, $OH_VERIFY_RUN/evidence/_fixtures/<name>): control-openhands fixture file --name qa-f22-import.automation.json --content '{"version":1,"kind":"automation","spec":{"name":"QA_F22 Imported","trigger":{"type":"cron","schedule":"0 0 1 1 *","timezone":"UTC"},"enabled":true,"prompt":"Reply with the single word: pong. Do not run any tools.","plugins":["github:OpenHands/extensions/plugins/qa-none"]}}', control-openhands fixture file --name qa-f22-view.automation.json --content '{"version":1,"kind":"automation","spec":{"name":"QA_F22 View","trigger":{"type":"cron","schedule":"0 0 1 1 *"},"enabled":true,"prompt":"Reply with the single word: pong. Do not run any tools."}}', control-openhands fixture file --name qa-f22-broken.json --content '{"version": 1, "kind": ' and control-openhands fixture file --name qa-f22-schema.json --content '{"version":2,"kind":"automation","spec":{"name":"","trigger":{"type":"event"},"enabled":"yes","prompt":"x"}}'.
Every automation created here uses the yearly cron 0 0 1 1 *, so none fires during the run.
Tarball fixture (arrange only): control-openhands fixture tarball --name qa-f22-tarball writes $OH_VERIFY_RUN/evidence/_fixtures/qa-f22-tarball.tar.gz (one main.py that prints pong).
Blocked here: completing F22.template-mcp-install-queue (and launching a card whose only missing integration is installable, such as research-brief-writer) needs a real Slack, Linear or Tavily credential: the install dialog tests the connection before saving.

Templates pageF22.templates-pageMapped behavior

Scenario F22--templates-page · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Sidebar automations”.
Do
Choose “Automations navbar desktop” / “Automations navigation templates”.
Do
Open /automations/templates.

Expect

/automations/templates shows the h1 Templates, a description, a search box, Start from a proven workflow with a count badge (6) and a Beta section (17).

Exact CLI recipe & bug checks

From / run control-openhands browser click 'testid=sidebar-automations-link' --expect-url '/automations$', control-openhands browser click 'testid=automations-navbar-desktop >> testid=automations-navigation-templates' --expect-url '/automations/templates', control-openhands browser text 'role=main >> role=heading[level=1]' (Templates), control-openhands browser count '[data-testid^="recommended-automation-card-"]' (23) and control-openhands browser count 'testid=recommended-automations-beta-section >> [data-testid^="recommended-automation-card-"]' (17); browser text 'testid=recommended-automations-beta-heading' is Beta and 17. control-openhands browser snapshot 'role=main' --max-lines 60 shows heading "Start from a proven workflow" with the count 6. Take control-openhands browser screenshot --feature F22.templates-page --name grid. control-openhands browser goto /automations/templates renders the same page.

Cards and pillsF22.template-cardMapped behavior

Scenario F22--cards-and-pills · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Recommended automation pills research brief writer overflow”.
Do
Press Escape.

Expect

each card shows an icon or logo stack, name, category, two-line description, a plus badge and integration pills (N MCPs to connect before launch, Needs external setup for Jira/Bitbucket; Connected once installed, which needs a working integration and was not reached here); a +N overflow button opens a popover without launching the card.

Exact CLI recipe & bug checks

Run control-openhands browser testids 'testid=recommended-automation-card-research-brief-writer' --hidden: recommended-automation-icon-…, recommended-automation-plus-…, recommended-automation-pills-… and recommended-automation-pills-research-brief-writer-overflow (Show 1 more). control-openhands browser testids 'role=main' --hidden --filter integration lists automation-integration-external-jira and automation-integration-external-bitbucket (Needs external setup). control-openhands browser click 'testid=recommended-automation-pills-research-brief-writer-overflow' opens a popover (browser snapshot 'role=dialog': dialog: 2 MCPs to connect before launch) and the URL stays /automations/templates; control-openhands browser press Escape closes it (browser count 'role=dialog' 0).

SearchF22.templates-searchKnown issue

Scenario F22--search · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Enter “tavily” in the element identified in the CLI recipe.
Do
Enter “qa-no-such-template” in the element identified in the CLI recipe.
Do
Enter “” in the element identified in the CLI recipe.

Expect

the search box narrows cards by name, category, description, launch prompt and integration names; a query that matches nothing should say so.

Linked known issues

Known issue #17959: a template search with no matches leaves a blank area with no message (F22.templates-search).

Exact CLI recipe & bug checks

Run control-openhands browser fill 'role=main >> placeholder=Search automations...' tavily and control-openhands browser testids 'role=main' --filter recommended-automation-card: only recommended-automation-card-research-brief-writer (matched through its Tavily integration). Then control-openhands browser fill 'role=main >> placeholder=Search automations...' qa-no-such-template, control-openhands browser count 'testid=recommended-automations-section' and control-openhands browser screenshot --feature F22.templates-search --name no-match. Expected: a "no templates match" message. Actual: the count is 0 and browser text 'role=main' is only the title and description; the area under the search box is blank (recorded as fail). Clear with control-openhands browser fill 'role=main >> placeholder=Search automations...' '' (23 cards again).

Card opens setup, prerequisitesF22.template-launch-setup · F22.setup-prerequisitesMapped behavior

Scenario F22--card-opens-setup-prerequisites · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Recommended automation card custom automation”.
Do
Choose “Setup continue”.

Expect

a card whose entry ships a setup form (custom-automation, news-digest, github-pr-reviewer, qa-changes, …) navigates to /automations/new/<id>.

Before you start lists the entry's integrations with the reason each is needed and Manage integrations (/mcp); Continue is disabled while a required integration is missing; an entry with nothing to check skips the step.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=recommended-automation-card-custom-automation' --expect-url '/automations/new/custom-automation' and control-openhands browser snapshot 'testid=setup-dialog': heading Before you start, GitHub Optional. Used when you choose a GitHub repository…, link Manage integrations (/mcp) and Continue; control-openhands browser enabled 'testid=setup-continue-button' is true (an optional integration only warns). Run control-openhands browser click 'testid=setup-continue-button'; the heading becomes Custom automation and browser testids 'testid=setup-dialog' lists setup-action-kind, setup-field-schedule (0 9 * * *), setup-field-timezone (UTC), setup-field-name, setup-field-model, setup-field-timeout, setup-field-prompt, setup-field-repository and setup-field-ref. There is no Trigger selector here because this deployment only supports cron, and no automation-agent-profile because the default Prompt action cannot take one (F22.setup-agent-profile).

ValidationF22.setup-form-validationMapped behavior

Scenario F22--validation · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Enter “every day” in “Setup field schedule”.
Do
Press Tab.
Wait
Wait for “Setup field schedule error” to appear.
Wait
Wait for “Setup field schedule error” to disappear.

Expect

Continue checks fields locally (This field is required., Must be at least N characters., Must be at most N.) and a field blur sends the draft to the service, whose errors appear under the field (Value error, Invalid cron expression: …).

Exact CLI recipe & bug checks

Click testid=setup-continue-button with the form untouched: control-openhands browser text 'testid=setup-field-name-error' and ... 'testid=setup-field-prompt-error' are This field is required. Fill testid=setup-field-timeout with 5000, testid=setup-field-name with QA_F22 Custom, testid=setup-field-prompt with Reply with the single word: pong. Do not run any tools., click Continue: browser text 'testid=setup-field-timeout-error' is Must be at most 1800. Fill the timeout with 60. Service preflight: control-openhands browser fill 'testid=setup-field-schedule' 'every day', control-openhands browser press Tab, control-openhands browser wait 'testid=setup-field-schedule-error' --timeout 5000, then browser text 'testid=setup-field-schedule-error': Value error, Invalid cron expression: every day (from POST /api/automation/v1/validate, see browser network --last 5). Fill the schedule with 0 0 1 1 *, press Tab, and control-openhands browser wait 'testid=setup-field-schedule-error' --state detached --timeout 5000 succeeds.

Setup form colours (`F22.setup-form-validation`, #18330)F22.setup-form-validationMapped behavior

Scenario F22--setup-form-colours-f22-setup-form-validation-18330 · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
In the custom-automation setup dialog, the required and service errors read rgb(231, 106, 94) on OpenHands-Neutral and rgb(209, 52, 56) (field errors) / rgb(192, 40, 45) (prompt error) on white under Light+ (F16.color-theme); read them with control-openhands browser eval "getComputedStyle(document.querySelector('[data-testid=setup-field-schedule-error]')).color". The schedule preflight answers only once name and prompt are filled: with them empty /validate returns 200 and no schedule error appears.

Expect

Exact CLI recipe & bug checks

In the custom-automation setup dialog, the required and service errors read rgb(231, 106, 94) on OpenHands-Neutral and rgb(209, 52, 56) (field errors) / rgb(192, 40, 45) (prompt error) on white under Light+ (F16.color-theme); read them with control-openhands browser eval "getComputedStyle(document.querySelector('[data-testid=setup-field-schedule-error]')).color". The schedule preflight answers only once name and prompt are filled: with them empty /validate returns 200 and no schedule error appears.

Action kindsF22.setup-action-kindMapped behavior

Scenario F22--action-kinds · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Setup action kind”.
Do
Choose the listbox / “Plugin”.

Expect

the Action combobox (custom-automation: Prompt / Plugin / Upload tarball) swaps the action fields and keeps shared values.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=setup-action-kind', control-openhands browser snapshot 'role=listbox' (Prompt selected, Plugin, Upload tarball); the snapshot closes the listbox, so click testid=setup-action-kind again and then control-openhands browser click 'role=listbox >> role=option[name="Plugin"]': browser testids 'testid=setup-dialog' now includes setup-field-plugins. Choose Upload tarball the same way: setup-field-tarball, setup-field-entrypoint and setup-field-setupScript replace prompt, repository and ref. Choose Prompt again; control-openhands browser value 'testid=setup-field-name' is still QA_F22 Custom and setup-field-prompt still holds the prompt.

Agent profileF22.setup-agent-profileMapped behavior

Scenario F22--agent-profile · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Plugin” in “Setup action kind”.
Do
Choose “Upload tarball” in “Setup action kind”.
Do
Choose “Automation agent profile”.
Do
Press Escape.
Do
Choose “default” in “Automation agent profile”.
Do
Choose “Default” in “Automation agent profile”.
Do
Choose “Prompt” in “Setup action kind”.

Expect

the Agent profile combobox appears only for actions that create through /v1 (Upload tarball and bundle entries), not for Prompt or Plugin; it lists Default and every saved agent profile and shows Default while no profile is chosen (also after a switch through Prompt or Plugin and after picking Default, #18111), picking a profile hides the LLM profile field, switching to Prompt or Plugin clears the choice, and the automation is created with that agent_profile_id.

Exact CLI recipe & bug checks

In the same form (Prompt action, name QA_F22 Custom, prompt and schedule 0 0 1 1 * filled), control-openhands browser count 'testid=automation-agent-profile' is 0; after control-openhands browser choose 'testid=setup-action-kind' 'Plugin' it is still 0, and after control-openhands browser choose 'testid=setup-action-kind' 'Upload tarball' it is 1 (only actions that create through /v1 can carry a profile; the preset endpoints behind Prompt and Plugin reject agent_profile_id). Run control-openhands browser value 'testid=automation-agent-profile', control-openhands browser click 'testid=automation-agent-profile' and control-openhands browser snapshot 'role=listbox': the value is Default and the listbox has option "Default" [selected] and option "default" (the seeded agent profile from api GET /api/agent-profiles), the same as on a fresh Upload tarball form (F22.setup-tarball-create). control-openhands browser press Escape closes the listbox and leaves the dialog open (browser count 'testid=setup-dialog' is 1). Run control-openhands browser choose 'testid=automation-agent-profile' 'default': browser count 'testid=setup-field-model' is 0 (LLM profile hidden); browser choose 'testid=automation-agent-profile' 'Default' brings it back (1). Pick default again, then control-openhands browser choose 'testid=setup-action-kind' 'Prompt': the selector is gone and setup-field-model is back (1). Choose Upload tarball once more and run control-openhands browser value 'testid=automation-agent-profile': Default (the choice was cleared, shown as on a fresh form; choosing Default from the list also reads Default). Choose Prompt again before continuing. Creating an automation with a profile is in the Upload tarball bullet below.

Review and createF22.setup-review-confirmKnown issue

Scenario F22--review-and-create · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Setup field model”.
Do
Choose the listbox / “deepseek-flash”.
Do
Choose “Setup continue”.
Wait
Wait for “Setup review” to appear.
Do
Choose “Setup back”.
Do
Choose “Setup continue”.
Do
Reload the page.
Do
Go back in browser history.

Expect

Review lists every value (Not set for blanks); Back keeps the values; Confirm and create creates the automation (enabled) and replaces the URL with its detail page.

Linked known issues

Known issue #17436 (F22.setup-review-confirm): Prompt-preset automations created after the Canvas 1.17.0 upgrade fail on the first conversation events request while older tarballs work, so the UI shows Failed runs. Recheck with a real LLM key and the reported automation 1.11.1/Agent Server 1.46.0 deployment plus pre-upgrade and finish-tool-wired tarballs; retain this version-specific report separately from the plugin-preset Cloud failure in #16611.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=setup-field-model', control-openhands browser click 'role=listbox >> role=option[name="deepseek-flash"]', control-openhands browser click 'testid=setup-continue-button', control-openhands browser wait 'testid=setup-review' --timeout 10000 and control-openhands browser snapshot 'testid=setup-dialog': heading Review, terms Action Prompt, Schedule 0 0 1 1 *, Timezone UTC, Automation name, LLM profile deepseek-flash, Timeout 60, Prompt, Repository Not set, Branch or ref Not set, and buttons Back and Confirm and create. Take browser screenshot --feature F22.setup-review-confirm --name review. control-openhands browser click 'testid=setup-back-button' returns to the form with setup-field-name still QA_F22 Custom; click Continue, wait for setup-review again, then control-openhands browser click 'testid=setup-continue-button' --expect-url '/automations/[0-9a-f-]{8,}'. browser url is /automations/<id> (the id of the new automation); after control-openhands browser reload, control-openhands browser text 'role=heading[level=1]' is QA_F22 Custom and browser text 'testid=active-status-badge-active' is Active (the detail page has no main landmark: 'role=main >> …' times out there), and api GET '/api/automation/v1?limit=100' shows it enabled: true, trigger.schedule 0 0 1 1 *, timeout 60, model deepseek-flash. control-openhands browser back lands on /automations/templates (the setup URL was replaced).

Plugin action and timezoneF22.setup-plugin-createKnown issue

Scenario F22--plugin-action-and-timezone · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/new/custom-automation.
Do
Choose “Plugin” in “Setup action kind”.
Do
Choose “Europe/Berlin” in “Setup field timezone”.
Wait
Wait for “Setup review” to appear.
Do
Choose “Setup continue”.

Expect

the Plugin action takes plugin sources and a prompt, and a picked Timezone carries through Review into the created automation (preset_metadata.plugins, trigger.timezone).

Linked known issues

Known issue #16611 (F22.setup-plugin-create): a minimal plugin-preset Cloud automation fails on its in-sandbox conversation/events request with HTTP 500, while the identical prompt preset works. Live prerequisites: an authenticated Cloud account, a real model key, and a reviewed no-op plugin fixture for the paired cases. The reported cost-zero error is visible in run logs, but a local prompt-only check cannot establish the Cloud plugin-loading behavior.

Exact CLI recipe & bug checks

Run control-openhands browser goto /automations/new/custom-automation, click testid=setup-continue-button, then control-openhands browser choose 'testid=setup-action-kind' 'Plugin' and control-openhands browser choose 'testid=setup-field-timezone' 'Europe/Berlin'. Fill testid=setup-field-schedule with 0 0 1 1 *, testid=setup-field-name with QA_F22 Plugin, testid=setup-field-plugins with github:OpenHands/extensions/plugins/qa-none and testid=setup-field-prompt with Reply with the single word: pong. Do not run any tools.; click Continue and browser wait 'testid=setup-review' --timeout 10000. browser snapshot 'testid=setup-dialog': Action Plugin, Timezone Europe/Berlin, LLM profile Not set, Timeout Not set, Plugins github:OpenHands/extensions/plugins/qa-none. control-openhands browser click 'testid=setup-continue-button' --expect-url '/automations/[0-9a-f-]{8,}' --timeout 60000; api GET '/api/automation/v1?limit=100' shows QA_F22 Plugin enabled with trigger.timezone Europe/Berlin and preset_metadata {preset_type: plugin, plugins: [{source: github:OpenHands/extensions/plugins/qa-none}]}.

Upload tarball action, with an agent profileF22.setup-tarball-create · F22.setup-agent-profileKnown issue

Scenario F22--upload-tarball-action-with-an-agent-profile · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/new/custom-automation.
Do
Choose “Upload tarball” in “Setup action kind”.
Do
Use browser upload as specified in the exact recipe.
Do
Choose “default” in “Automation agent profile”.
Wait
Wait for “Setup review” to appear.
Do
Choose “Setup continue”.

Expect

the Upload tarball action requires a .tar/.tar.gz file, shows its file name in Review and creates an automation with the uploaded tarball and the entrypoint.

Linked known issues

Known issue #15509 (F22.setup-tarball-create): A native Windows event-triggered automation fails while uploading its workspace tarball with Path must be absolute. Recheck a Windows local stack and real event delivery, then observe its failed run/upload; a POSIX tarball upload is not a Windows path-validation check. This run has neither Windows nor the webhook fixture.

Exact CLI recipe & bug checks

Run control-openhands browser goto /automations/new/custom-automation, click testid=setup-continue-button, control-openhands browser choose 'testid=setup-action-kind' 'Upload tarball' (Entrypoint defaults to python3 main.py, and on this fresh form control-openhands browser value 'testid=automation-agent-profile' is Default), fill testid=setup-field-name with QA_F22 Tarball and testid=setup-field-schedule with 0 0 1 1 *, click Continue: browser text 'testid=setup-field-tarball-error' is This field is required. Run control-openhands browser upload 'testid=setup-field-tarball' "$OH_VERIFY_RUN/evidence/_fixtures/qa-f22-tarball.tar.gz" and control-openhands browser choose 'testid=automation-agent-profile' 'default' (browser count 'testid=setup-field-model' is 0); take control-openhands browser screenshot 'testid=setup-dialog' --feature F22.setup-agent-profile --name upload-profile. Click Continue, browser wait 'testid=setup-review' --timeout 10000: Review shows Action Upload tarball, LLM profile Not set, Tarball qa-f22-tarball.tar.gz, Entrypoint python3 main.py, Setup script Not set (Review does not list the agent profile). control-openhands browser click 'testid=setup-continue-button' --expect-url '/automations/[0-9a-f-]{8,}' --timeout 60000; browser text 'role=heading[level=1]' is QA_F22 Tarball, and control-openhands api GET /api/automation/v1/<id> (the id in the URL) shows it enabled with entrypoint python3 main.py, a tarball_path oh-internal://uploads/…, model null and agent_profile_id equal to the default profile's id from control-openhands api GET /api/agent-profiles.

CloseF22.setup-closeMapped behavior

Scenario F22--close · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/templates.
Do
Choose “Recommended automation card news digest”.
Do
Choose “Setup dialog close”.
Do
Press Escape.
Do
Open /automations/new/news-digest.
Do
Choose “Setup dialog close”.

Expect

X or Escape goes back in history; on a cold deep link it replaces the location with /automations.

Exact CLI recipe & bug checks

From /automations/templates (control-openhands browser goto /automations/templates; the previous bullet ends on the new automation's page), control-openhands browser click 'testid=recommended-automation-card-news-digest' --expect-url '/automations/new/news-digest' then control-openhands browser click 'testid=setup-dialog-close' --expect-url '/automations/templates'; the same with control-openhands browser press Escape instead of X also returns to /automations/templates. Cold deep link: control-openhands browser goto /automations/new/news-digest, control-openhands browser click 'testid=setup-dialog-close' --expect-url '/automations$' (replaced, not back).

Bundle entry, min lengthF22.setup-bundle-create · F22.setup-form-validationMapped behavior

Scenario F22--bundle-entry-min-length · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/templates.
Wait
Wait for “Setup review” to appear.
Do
Choose “Setup continue”.

Expect

a bundle entry (Daily news digest) uploads its script tarball and creates an automation with the bundle's entrypoint, without any account.

Exact CLI recipe & bug checks

From /automations/templates (control-openhands browser goto /automations/templates; Close ends on /automations) click testid=recommended-automation-card-news-digest (--expect-url '/automations/new/news-digest'): there is no prerequisites step and the form shows automation-agent-profile (a bundle creates through /v1, so it offers a profile; leave it on Default), setup-field-schedule (0 8 * * *), setup-field-timezone, setup-field-feeds (three default feeds) and setup-field-topics. Fill testid=setup-field-feeds with short and click Continue: browser text 'testid=setup-field-feeds-error' is Must be at least 8 characters. Close (testid=setup-dialog-close), click the card again (the draft is gone), fill testid=setup-field-schedule with 0 0 1 1 *, click Continue, browser wait 'testid=setup-review' --timeout 10000, then control-openhands browser click 'testid=setup-continue-button' --expect-url '/automations/[0-9a-f-]{8,}' --timeout 60000. The detail h1 is Daily news digest; api GET '/api/automation/v1?limit=100' shows it enabled with entrypoint python3 main.py, a tarball_path oh-internal://uploads/… and timeout 900.

Repository listF22.setup-repository-listKnown issue

Scenario F22--repository-list · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/new/github-pr-reviewer.
Do
Choose “Setup list repositories remove qa org/qa repo”.

Expect

a repository list field has Add (disabled while the input is empty) and a Remove per row; an empty required list is refused.

Linked known issues

Known issue OpenHands/software-agent-sdk#5729 (F22.setup-repository-list): prompt and plugin preset workers call RemoteWorkspace.clone_repos, which asks for lowercase provider-token secret names. A private repository can therefore fail to clone when only the documented uppercase GITHUB_TOKEN (or GitLab/Bitbucket equivalent) exists. Recheck an owned preset with a private repository and only its uppercase secret, using an authorized disposable repository or a reviewed token-gated local Git fixture; a public repository and the Canvas Agent Server's separate Git-provider service do not exercise that lookup.

Exact CLI recipe & bug checks

Run control-openhands browser goto /automations/new/github-pr-reviewer (no prerequisites step locally). control-openhands browser enabled 'testid=setup-list-repositories-add' is false; fill testid=setup-field-repositories with qa-org/qa-repo (enabled turns true), click testid=setup-list-repositories-add: browser text 'testid=setup-list-repositories' starts qa-org/qa-repo / Remove and the input is empty again. control-openhands browser click 'testid=setup-list-repositories-remove-qa-org/qa-repo' removes it (count 0); Continue then shows setup-field-repositories-error This field is required. Close the dialog.

Blocking prerequisiteF22.setup-prerequisitesMapped behavior

Scenario F22--blocking-prerequisite · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/new/incident-retrospective-drafter.

Expect

Exact CLI recipe & bug checks

Run control-openhands browser goto /automations/new/incident-retrospective-drafter, control-openhands browser snapshot 'testid=setup-dialog' (Slack, Linear and Notion with reasons) and control-openhands browser enabled 'testid=setup-continue-button': false while Slack and Linear are not installed.

Not available and fallbackF22.setup-unsupported · F22.setup-fallback-conversationMapped behavior

Scenario F22--not-available-and-fallback · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/new/qa-changes.
Do
Choose “Setup fallback conversation”.

Expect

an entry this deployment cannot run (qa-changes needs event triggers) shows Not available, the unmet requirements, Close and Set up in a conversation.

Set up in a conversation creates a conversation seeded with the entry's setup message.

Exact CLI recipe & bug checks

Run control-openhands browser goto /automations/new/qa-changes and control-openhands browser snapshot 'testid=setup-dialog': heading Not available, This deployment does not support everything this automation needs., setup-unmet-requirements webhookDelivery, event, buttons Close and Set up in a conversation. Run control-openhands browser click 'testid=setup-fallback-conversation' --expect-url '/conversations/[0-9a-f]' --timeout 60000 and control-openhands browser eval "document.querySelector('[data-testid=chat-input]').innerText": /qa-changes, a blank line and This deployment cannot run the webhook-driven QA automation directly. Set it up in this conversation instead: …. The message is pre-filled, not sent (no model cost until the user presses send).

Unknown idF22.setup-unknown-idMapped behavior

Scenario F22--unknown-id · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/new/qa-no-such-template.

Expect

/automations/new/<id> for an id no catalog entry claims is a 404.

Exact CLI recipe & bug checks

control-openhands browser goto /automations/new/qa-no-such-template then control-openhands browser snapshot: heading "404" and Not Found.

Responder choiceF22.responder-deployment-choiceMapped behavior

Scenario F22--responder-choice · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations/templates.
Do
Choose “Recommended automation card slack channel monitor”.
Do
Press Escape.
Do
Choose “Responder deployment modal close”.
Do
Choose “Responder deployment open openhands cloud”.
Do
Use browser tabs as specified in the exact recipe.
Do
Use browser tabs as specified in the exact recipe.
Do
Use browser close tab as specified in the exact recipe.
Do
Choose “Responder deployment continue local”.
Wait
Wait for “Responder deployment modal” to disappear.
Do
Choose “Mcp install cancel”.

Expect

on a local backend a GitHub/Slack-only responder card first asks Choose how your responder should run; Continue with local setup creates the OPENHANDS_URL secret and continues, Open OpenHands Cloud integrations opens Cloud in a new tab, X and Escape cancel.

Exact CLI recipe & bug checks

On /automations/templates (control-openhands browser goto /automations/templates; the previous bullet ends on a 404 page) run control-openhands browser click 'testid=recommended-automation-card-slack-channel-monitor' and control-openhands browser snapshot 'testid=responder-deployment-modal': Choose how your responder should run, Poll locally on your laptop / Continue with local setup, Use OpenHands Cloud / Open OpenHands Cloud integrations. control-openhands browser press Escape and, after reopening, control-openhands browser click 'testid=responder-deployment-modal-close' each make browser count 'testid=responder-deployment-modal' 0. Reopen and control-openhands browser click 'testid=responder-deployment-open-openhands-cloud': the modal closes and control-openhands browser tabs lists tab 1 at https://app.all-hands.dev/settings/integrations (it can take a second to appear; rerun browser tabs). Close it with control-openhands browser close-tab 1. Reopen and control-openhands browser click 'testid=responder-deployment-continue-local', control-openhands browser wait 'testid=responder-deployment-modal' --state detached --timeout 10000: the Slack install dialog (testid=mcp-install-modal, dialog "Slack") opens next and api GET /api/settings/secrets now lists OPENHANDS_URL. Close it with control-openhands browser click 'testid=mcp-install-cancel'.

Responder launch with Slack installedF22.responder-local-launchMapped behavior

Scenario F22--responder-launch-with-slack-installed · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Arrange the prerequisite backend state through the API.
Do
Choose “Recommended automation card slack standup digest”.
Wait
Wait for “Responder deployment modal” to appear.
Do
Choose “Responder deployment continue local”.
Wait
Wait for the element identified in the CLI recipe to appear.
Do
Choose “Submit”.
Wait
Wait for “User message” / containing “/standup-digest:setup” to appear.
Do
Pause the conversation.
Setup
Arrange the prerequisite backend state through the API.

Expect

with the responder's integration already installed (Slack for Slack standup digest), Continue with local setup skips the install dialog and opens a new conversation whose composer is pre-filled with the entry's setup command (/standup-digest:setup). Nothing is sent until the user presses send; Send delivers it as the user message.

Exact CLI recipe & bug checks

Arrange Slack as installed with its catalog command, a dummy token and offline npm, so nothing is downloaded or contacted (arrange, not proof): write qa-slack.json (outside the workspace) containing {"agent_settings_diff":{"mcp_config":{"slack":{"command":"npx","args":["-y","@zencoderai/slack-mcp-server"],"env":{"SLACK_BOT_TOKEN":"xoxb-qa-dummy","SLACK_TEAM_ID":"T0000000000","npm_config_offline":"true"}}}}} and run control-openhands api PATCH /api/settings --data @qa-slack.json --write. On /automations/templates run control-openhands browser click 'testid=recommended-automation-card-slack-standup-digest', control-openhands browser wait 'testid=responder-deployment-modal', control-openhands browser click 'testid=responder-deployment-continue-local' and control-openhands browser wait-url '/conversations/[0-9a-f-]+' --timeout 60000 (note <slack-id>). control-openhands browser count 'testid=mcp-install-modal' is 0. After sleep 5, control-openhands browser eval "document.querySelector('[data-testid=chat-input]').innerText" is /standup-digest:setup and control-openhands browser count 'testid=user-message' is 0 (control-openhands browser screenshot --feature F22.responder-local-launch --name slack-prefilled). Run control-openhands browser click 'testid=submit-button' and control-openhands browser wait 'testid=user-message >> has-text=/standup-digest:setup' --timeout 15000. Then run control-openhands conversation pause <slack-id> at once. The setup skill is open-ended. On 2026-10-08, deepseek-flash spent 25 tool calls exploring the host, including files outside the run, before asking a setup question. Restore with control-openhands api DELETE /api/settings/mcp/slack --write; control-openhands api GET /api/settings --pick agent_settings.mcp_config is {}. Run this bullet after Responder choice, which needs Slack not installed.

Install queueF22.template-mcp-install-queueBlocked

Scenario F22--install-queue · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Recommended automation card linear triage assistant”.
Do
Choose “Mcp install cancel”.

Expect

a card with missing required integrations opens each one's install dialog in turn; Cancel abandons the launch without creating anything.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=recommended-automation-card-linear-triage-assistant': dialog "Linear" (URL https://mcp.linear.app/mcp, optional API key). control-openhands browser click 'testid=mcp-install-cancel'; browser count 'role=dialog' is 0, the URL stays /automations/templates, control-openhands conversation list has no new conversation and api GET /api/settings still has an empty mcp_config. Completing the queue is blocked: Install first tests the connection (Linear without a key: Connection failed: HTTPStatusError: Client error '401 Unauthorized'…; Tavily research-brief-writer with a dummy key: Connection timed out. Check the URL and try again.), so it never launches without a working credential.

Prompt launchF22.template-launch-conversationBlocked

Scenario F22--prompt-launch · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Recommended automation pills jira issue to bitbucket pr overflow”.
Do
Press Escape.
Do
Choose “Recommended automation card jira issue to bitbucket pr”.

Expect

a card without a setup form and with no installable integration missing creates a conversation and pre-fills its composer with the entry's launch command (for example /ticket-to-code-change:setup); nothing is sent until the user presses send.

Exact CLI recipe & bug checks

Use a card whose required integrations all need external setup, so no install dialog queues: on /automations/templates run control-openhands browser click 'testid=recommended-automation-pills-jira-issue-to-bitbucket-pr-overflow' (dialog: Bitbucket Needs external setup), control-openhands browser press Escape, then control-openhands browser click 'testid=recommended-automation-card-jira-issue-to-bitbucket-pr' --expect-url '/conversations/[0-9a-f]' --timeout 60000. control-openhands browser eval "document.querySelector('[data-testid=chat-input]').innerText" is /ticket-to-code-change:setup, and control-openhands conversation events <id> --kinds MessageEvent --from-start (id from the URL) has total 0 even 20 s later: the launch command is pre-filled, not sent (the featured copy promises "a pre-filled conversation"). Do not press send (no model cost). Take browser screenshot --feature F22.template-launch-conversation --name prefilled. Cards with an installable missing integration (research-brief-writer, Tavily) go through the install queue first and are blocked here (see Preconditions).

Open and save a draft (`F22.agent-assisted-setup`, desktop and phone)F22.agent-assisted-setupMapped behavior

Scenario F22--open-and-save-a-draft-f22-agent-assisted-setup-desktop-and-phone · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automations add automation”.
Do
Choose “Automations add automation create”.
Do
Enter “QA_F22 Delta Draft” in “Automation setup name”.
Do
Enter “Reply pong. Do not run any tools.” in “Automation setup prompt”.
Do
Choose “Automation setup frequency custom”.
Do
Enter “0 0 1 1 *” in “Automation setup custom schedule”.
Do
Choose “Automation setup save draft”.
Wait
Wait for “Automation setup draft details” to appear.
Do
Reload the page.

Expect

Add Automation → Create Automation opens /automations/setup, whose setup panel collects name, action, trigger, model/profile and custom code, can ask the agent for help, saves server-backed drafts, opens test runs behind a Back button and dispatches the finished automation.

Exact CLI recipe & bug checks

This scoped recipe needs a doctored baseline stack with automation draft support (1.19.3 here); its non-dispatch steps need no LLM key or profile. From /automations run control-openhands browser click 'testid=automations-add-automation', then control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup'. The form and docked composer are visible; control-openhands conversation list is unchanged. Run control-openhands browser fill 'testid=automation-setup-name' 'QA_F22 Delta Draft', control-openhands browser fill 'testid=automation-setup-prompt' 'Reply pong. Do not run any tools.', control-openhands browser click 'testid=automation-setup-frequency-custom', control-openhands browser fill 'testid=automation-setup-custom-schedule' '0 0 1 1 *', then control-openhands browser click 'testid=automation-setup-save-draft'. control-openhands browser wait 'testid=automation-setup-draft-details' --timeout 10000 and browser snapshot 'testid=automation-setup-draft-details' show Draft saved and Ready to test. control-openhands api GET /api/automation/v1/drafts supplies <draft-id> and confirms the name, prompt and yearly cron; the URL carries draftId. After browser reload, read each field with browser value 'testid=automation-setup-name', browser value 'testid=automation-setup-prompt' and browser value 'testid=automation-setup-custom-schedule': the values match. Screenshot with --feature F22.agent-assisted-setup --name saved-draft. Saving a prompt draft does not prove model execution; use the configured flash profile and the Genuine prompt/plugin draft Test checks below for that path.

Credential-free script testF22.agent-assisted-setupMapped behavior

Scenario F22--credential-free-script-test · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation setup kind custom”.
Do
Use browser record as specified in the exact recipe.
Do
Choose “Automation setup draft test”.
Wait
Wait for “Automation setup draft runs page” to appear.
Wait
Wait for “Automation setup draft run” / “Successful” to appear.
Do
Reload the page.
Do
Choose “Automation setup draft runs back”.
Do
Use browser record as specified in the exact recipe.

Expect

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=automation-setup-kind-custom', then inspect browser value 'testid=automation-setup-custom-code': the generated Python prints its prompt and calls the real automation callback, without an agent/LLM. Keep its code, setup and entrypoint unchanged; click Save draft. Run browser record start --feature F22.agent-assisted-setup --name custom-script-test --max-seconds 180, then browser click 'testid=automation-setup-draft-test' and browser wait 'testid=automation-setup-draft-runs-page' --timeout 10000. Its snapshot initially shows Pending. Immediately run api GET /api/automation/v1/drafts/<draft-id> for materialized_automation_id and last_test_run_id, then api GET '/api/automation/v1/<materialized-id>/runs?limit=10'. Repeat the quoted runs GET after short sleep 5 intervals until that run has terminal backend status (bounded to 90 seconds; no single wait over 45 seconds). Once it is COMPLETED, browser wait 'testid=automation-setup-draft-run >> text=Successful' --timeout 45000 tests automatic UI completion. Expected: Successful without a reload. Record the actual row and a second backend receipt before any recovery. As a separate readback, run browser reload, click testid=automation-setup-view-test-runs, screenshot Successful, then browser click 'testid=automation-setup-draft-runs-back': the draft details read Latest test passed. End with browser record stop. A saved draft, Pending row or successful reload does not prove automatic completion updates.

Resume at phone widthF22.agent-assisted-setupMapped behavior

Scenario F22--resume-at-phone-width · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Use the phone viewport.
Do
Open /automations.

Expect

Exact CLI recipe & bug checks

Run browser viewport phone, browser goto /automations, then click testid=automation-setup-draft-resume-<draft-id> with --expect-url '/automations/setup'. Read Name, rename to QA_F22 Phone Draft, Save draft and reload: browser value 'testid=automation-setup-name' and api GET /api/automation/v1/drafts/<draft-id> both have the new name. browser bbox 'testid=automation-setup-form' has pageHorizontalOverflow: false; this long form scrolls vertically and need not fit wholly inside the viewport. Screenshot with --feature F22.agent-assisted-setup --name resumed-phone. Repeat initial open/save/reload on a fresh phone draft when checking that entry point; desktop creation is not phone creation. Restore desktop afterward.

Agent fills a prompt form (`F22.agent-assisted-setup`, desktop)F22.agent-assisted-setupMapped behavior

Scenario F22--agent-fills-a-prompt-form-f22-agent-assisted-setup-desktop · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations.
Do
Choose “Automations add automation”.
Do
Choose “Automations add automation create”.
Do
Start a conversation with the prompt “Call automation_form_update exactly once with fields {"name":"QA_F22 DeepSeek Desktop","kind":"prompt","prompt":"Reply pong. Do not run any tools.","frequency":"custom","customSchedule":"0 0 1 1 *","timezone":"UTC","model":"deepseek-flash"}. Do not call any other tools, run terminal commands, inspect files, or read environment variables. Briefly confirm the filled fields and stop.”.
Wait
Wait for the conversation to finish.

Expect

Exact CLI recipe & bug checks

Start a fresh form with control-openhands browser goto /automations, control-openhands browser click 'testid=automations-add-automation' and control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup'. Run control-openhands conversation start --stay --prompt 'Call automation_form_update exactly once with fields {"name":"QA_F22 DeepSeek Desktop","kind":"prompt","prompt":"Reply pong. Do not run any tools.","frequency":"custom","customSchedule":"0 0 1 1 *","timezone":"UTC","model":"deepseek-flash"}. Do not call any other tools, run terminal commands, inspect files, or read environment variables. Briefly confirm the filled fields and stop.'. --stay sends through the docked composer; omitting it would navigate away to New Chat. The returned <setup-conversation-id> opens /conversations/<id> with the form beside it. Run control-openhands conversation wait <setup-conversation-id> --until finished,idle --timeout 60, then control-openhands conversation events <setup-conversation-id> --kinds ActionEvent --full: a genuine automation_form_update supplied the exact fields. Read control-openhands browser value 'testid=automation-setup-name', control-openhands browser value 'testid=automation-setup-prompt', control-openhands browser value 'testid=automation-setup-custom-schedule' and control-openhands browser text 'testid=automation-setup-model'; they match the requested values. control-openhands browser snapshot 'testid=automation-setup-form' also shows Filled by OpenHands markers. Capture control-openhands browser screenshot --feature F22.agent-assisted-setup --name desktop-genuine-tool-filled; the UI field readbacks and real action establish the update, not the agent's confirmation alone. The sent prompt is not kept as a draft: control-openhands browser eval "sessionStorage.getItem('oh:home-prompt-draft')" is null (with no earlier Home draft), and the New Chat composer (browser eval "document.querySelector('[data-testid=chat-input]').innerText" after clicking text=New Chat) and a later /automations/setup docked composer are empty; return to /conversations/<setup-conversation-id> for the next bullet.

Desktop Hide/Show and phone viewsF22.agent-assisted-setupMapped behavior

Scenario F22--desktop-hide-show-and-phone-views · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation setup save draft”.
Wait
Wait for “Automation setup draft details” to appear.
Setup
Use the phone viewport.
Do
Choose “Automation setup mobile view form”.
Do
Choose “Automation setup mobile view conversation”.
Do
Open /automations.
Do
Open /conversations/<setup-conversation-id>.
Do
Choose “Automation setup mobile view form”.
Do
Choose “Automation setup create”.

Expect

Exact CLI recipe & bug checks

On that associated conversation, control-openhands browser attr 'testid=automation-setup-agent-toggle' aria-label reads Hide Agent. Click the toggle: the label becomes Show Agent, and control-openhands browser bbox 'testid=conversation-chat-panel' has width 0; form values remain. Click again: Hide Agent and a visible conversation panel return. Save with control-openhands browser click 'testid=automation-setup-save-draft', control-openhands browser wait 'testid=automation-setup-draft-details' --timeout 10000 and control-openhands api GET /api/automation/v1/drafts to obtain <assisted-draft-id> and the stored form. Run control-openhands browser viewport phone, control-openhands browser click 'testid=automation-setup-mobile-view-form', read Name, then control-openhands browser click 'testid=automation-setup-mobile-view-conversation'; the selected view fits without horizontal overflow. From control-openhands browser goto /automations, click testid=automation-setup-draft-resume-<assisted-draft-id> with --expect-url '/automations/setup': dashboard Resume opens /automations/setup?draftId=<id>, not the original conversation URL. Reload and read Name/prompt/schedule again. Separately control-openhands browser goto /conversations/<setup-conversation-id> and control-openhands browser click 'testid=automation-setup-mobile-view-form' restore the same draft inside its associated conversation. Capture each entry separately. control-openhands browser click 'testid=automation-setup-create' --expect-url '/automations/[0-9a-f-]{8,}' creates the final prompt automation; control-openhands api GET /api/automation/v1/<id> shows the requested fields and enabled: false.

Fresh phone prompt and genuine Run nowF22.agent-assisted-setupMapped behavior

Scenario F22--fresh-phone-prompt-and-genuine-run-now · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Turn on”.
Do
Choose “Run now”.
Wait
Wait for “Automation activity log” / “Run status icon completed” to appear.

Expect

Exact CLI recipe & bug checks

Stay at phone width, open a new form through Dashboard Add → Create, and send the exact one-tool prompt above with name QA_F22 DeepSeek Phone. Read the genuine action and fields, click testid=automation-setup-mobile-view-form, Save draft, reload and read back the saved values before clicking Create. On the resulting detail page run control-openhands browser click 'role=switch[name="Turn on"]', control-openhands browser click 'role=button[name="Run now"]', then control-openhands browser wait 'testid=automation-activity-log >> testid=run-status-icon-completed' --timeout 60000 without reloading. control-openhands api GET /api/automation/v1/<id>/runs independently reports COMPLETED with a conversation id; control-openhands conversation events <run-conversation-id> --kinds ActionEvent --full contains a real finish, and MessageEvent output is pong. The ordinary detail row becomes Successful automatically. Desktop assisted-prompt creation and fresh phone prompt execution are separate scopes. Return to desktop before the next bullet.

Agent-created plugin (`F22.agent-assisted-setup`, desktop and fresh phone)F22.agent-assisted-setupMapped behavior

Scenario F22--agent-created-plugin-f22-agent-assisted-setup-desktop-and-fresh-phone · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Arrange a public plugin fixture with mkdir -p "$OH_VERIFY_RUN/workspace" and cp -R node_modules/@openhands/extensions/plugins/magic-test "$OH_VERIFY_RUN/workspace/qa-f22-real-plugin"; <plugin-path> below is that absolute run-owned path. Open another fresh form and send the one-tool docked prompt with fields {"name":"QA_F22 DeepSeek Real Plugin","kind":"plugin","pluginSource":"<plugin-path>","prompt":"Reply pong. Do not run any tools.","frequency":"custom","customSchedule":"0 0 1 1 *","timezone":"UTC","model":"deepseek-flash"} (replace the path before sending). Read the real automation_form_update, the form and its saved values. Create, Turn on and Run now as above; the detail row becomes Successful without reload and its actual conversation finishes pong. control-openhands conversation events <run-conversation-id> --grep magic-word --from-start confirms that the public plugin's skill registered. This checks plugin loading/execution; pong does not exercise that skill's trigger, hooks, MCP or scripts. Repeat from a fresh phone form with name QA_F22 DeepSeek Plugin Phone Draft, using the Form view and the next draft-Test bullet before final Create/Turn on/Run now; desktop creation alone does not prove that phone path.

Expect

Exact CLI recipe & bug checks

Arrange a public plugin fixture with mkdir -p "$OH_VERIFY_RUN/workspace" and cp -R node_modules/@openhands/extensions/plugins/magic-test "$OH_VERIFY_RUN/workspace/qa-f22-real-plugin"; <plugin-path> below is that absolute run-owned path. Open another fresh form and send the one-tool docked prompt with fields {"name":"QA_F22 DeepSeek Real Plugin","kind":"plugin","pluginSource":"<plugin-path>","prompt":"Reply pong. Do not run any tools.","frequency":"custom","customSchedule":"0 0 1 1 *","timezone":"UTC","model":"deepseek-flash"} (replace the path before sending). Read the real automation_form_update, the form and its saved values. Create, Turn on and Run now as above; the detail row becomes Successful without reload and its actual conversation finishes pong. control-openhands conversation events <run-conversation-id> --grep magic-word --from-start confirms that the public plugin's skill registered. This checks plugin loading/execution; pong does not exercise that skill's trigger, hooks, MCP or scripts. Repeat from a fresh phone form with name QA_F22 DeepSeek Plugin Phone Draft, using the Form view and the next draft-Test bullet before final Create/Turn on/Run now; desktop creation alone does not prove that phone path.

Genuine prompt/plugin draft TestF22.agent-assisted-setupKnown failure

Scenario F22--genuine-prompt-plugin-draft-test · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automation setup draft test”.
Wait
Wait for “Automation setup draft runs page” to appear.
Wait
Wait for “Automation setup draft run” / “Successful” to appear.
Do
Reload the page.
Do
Choose “Automation setup view test runs”.
Do
Choose “Automation setup draft runs back”.

Expect

Known bug observations

Known failure #18263: the genuine phone plugin run was COMPLETED while the page remained Pending more than 201 seconds later; a separate desktop prompt draft Test reproduced the same failure through a 45-second wait. Record fail, backend receipts and the visible row before control-openhands browser reload and control-openhands browser click 'testid=automation-setup-view-test-runs'. Reload shows Successful; control-openhands browser click 'testid=automation-setup-draft-runs-back' shows Latest test passed. Then Create the phone plugin and verify its ordinary Run now completion as above. The standalone desktop prompt variant uses the name/prompt/yearly-cron fields from Open and save a draft, with kind left Prompt and the configured flash model, before the same Test/status/reload checks.

Exact CLI recipe & bug checks

On the fresh phone plugin form, Save draft and read its <model-draft-id> through control-openhands api GET /api/automation/v1/drafts, then control-openhands browser click 'testid=automation-setup-draft-test' and control-openhands browser wait 'testid=automation-setup-draft-runs-page' --timeout 10000. Use control-openhands api GET /api/automation/v1/drafts/<model-draft-id> for materialized_automation_id and last_test_run_id, then poll control-openhands api GET '/api/automation/v1/<materialized-id>/runs?limit=10' until the exact run is terminal. Inspect its real conversation's finish event and pong. As in the script-test recipe, test the open row with control-openhands browser wait 'testid=automation-setup-draft-run >> text=Successful' --timeout 45000 before recovery. Expected: automatic Successful. Known failure #18263: the genuine phone plugin run was COMPLETED while the page remained Pending more than 201 seconds later; a separate desktop prompt draft Test reproduced the same failure through a 45-second wait. Record fail, backend receipts and the visible row before control-openhands browser reload and control-openhands browser click 'testid=automation-setup-view-test-runs'. Reload shows Successful; control-openhands browser click 'testid=automation-setup-draft-runs-back' shows Latest test passed. Then Create the phone plugin and verify its ordinary Run now completion as above. The standalone desktop prompt variant uses the name/prompt/yearly-cron fields from Open and save a draft, with kind left Prompt and the configured flash model, before the same Test/status/reload checks.

Setup fixture cleanupF22.agent-assisted-setupMapped behavior

Scenario F22--setup-fixture-cleanup · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Arrange the prerequisite backend state through the API.
Setup
Arrange the prerequisite backend state through the API.

Expect

Exact CLI recipe & bug checks

Read the owned draft and final automation ids before deleting. Run control-openhands api DELETE /api/automation/v1/drafts/<draft-id> --write for each remaining draft, including the credential-free draft; confirm that its materialized test automation is absent. Delete final QA_F22 DeepSeek … automations with control-openhands api DELETE /api/automation/v1/<id> --write and confirm both lists contain no owned fixtures. Finish or pause the setup/execution conversations. This cleanup checks fixture removal; dashboard Delete is a distinct UI path.

Import pickerF22.import-pickerMapped behavior

Scenario F22--import-picker · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Automations add automation”.
Do
Choose “Automations import automation”.

Expect

Add Automation → Import automation opens a dialog that explains the format (docs link), says the import is disabled, and offers a dropzone and Choose file.

Exact CLI recipe & bug checks

On /automations run control-openhands browser click 'testid=automations-add-automation', control-openhands browser click 'testid=automations-import-automation', control-openhands browser attr 'testid=import-automation-modal' data-view (picker) and control-openhands browser snapshot 'testid=import-automation-modal': heading Import automation, the explanation with link File format documentation (https://docs.openhands.dev/openhands/usage/agent-canvas/managing-automations#exported-file-format), The imported automation will be disabled until you review and enable it., Drop a .json file here, or, Choose file.

Bad filesF22.import-validationMapped behavior

Scenario F22--bad-files · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Use browser upload as specified in the exact recipe.

Expect

a file that is not JSON shows the toast The selected file is not valid JSON.; a JSON file that breaks the format shows one toast listing every issue; the dialog stays on the picker.

Exact CLI recipe & bug checks

In the open picker run control-openhands browser upload 'testid=automations-import-file' "$OH_VERIFY_RUN/evidence/_fixtures/qa-f22-broken.json" and control-openhands browser toasts: The selected file is not valid JSON. Then upload qa-f22-schema.json the same way: one toast reads version: expected 1, spec.name: expected a non-empty string, spec.trigger.source: required for an event trigger, spec.trigger.on: required for an event trigger, spec.enabled: expected a boolean (one per line). data-view stays picker.

Preview and cancelF22.import-previewMapped behavior

Scenario F22--preview-and-cancel · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Import automation cancel”.

Expect

a valid file switches the dialog to a preview (Name, Trigger, Prompt, Plugins) with the disabled notice, Cancel and Import automation.

Exact CLI recipe & bug checks

Upload qa-f22-import.automation.json: data-view is preview and browser snapshot 'testid=import-automation-modal' shows Review the automation details before importing., Name QA_F22 Imported, Trigger 0 0 1 1 * · UTC, the Prompt, Plugins github:OpenHands/extensions/plugins/qa-none, the disabled notice, Cancel and Import automation. control-openhands browser click 'testid=import-automation-cancel' closes it (count 0); reopening shows data-view picker again.

ImportF22.import-confirmMapped behavior

Scenario F22--import · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Choose “Import automation confirm”.
Do
Choose the status / “View automation”.

Expect

Import automation reads Importing... while pending, closes the dialog, shows Imported "<name>" as disabled. View automation, and the automation is listed disabled; the toast link opens its detail page.

Exact CLI recipe & bug checks

Reopen the dialog, upload qa-f22-import.automation.json, then control-openhands browser click 'testid=import-automation-confirm' --observe 'testid=import-automation-confirm' --observe-ms 3000 (observed Importing... for about 50 ms, then <absent>) and control-openhands browser toasts: Imported "QA_F22 Imported" as disabled. View automation. The card [data-testid^="automation-card-"] >> has-text=QA_F22 Imported is under Inactive, and api GET /api/automation/v1/<id> has enabled: false, state INACTIVE and the plugin in preset_metadata.plugins. The toast lasts about 5 s, so follow its link in the very next command: import qa-f22-view.automation.json the same way, then immediately control-openhands browser click 'role=status >> role=link[name="View automation"]' --expect-url '/automations/[0-9a-f-]{8,}'; browser text 'role=heading[level=1]' is QA_F22 View.

Choose fileF22.import-choose-fileMapped behavior

Scenario F22--choose-file · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open /automations.
Do
Provide the fixture file or pasted content using “Import automation choose file”.
Do
Provide the fixture file or pasted content using “Import automation dropzone” / “Drop a .json file here”.
Do
Press Escape.

Expect

Choose file, and a click anywhere on the dropzone, open the browser's file chooser; the chosen file is previewed.

Exact CLI recipe & bug checks

On /automations (control-openhands browser goto /automations; Import ends on QA_F22 View's page) open Add Automation → Import automation and run control-openhands browser upload-via 'testid=import-automation-choose-file' "$OH_VERIFY_RUN/evidence/_fixtures/qa-f22-import.automation.json" (answers the real file chooser): browser attr 'testid=import-automation-modal' data-view is preview. Close with testid=import-automation-modal-close, reopen, and control-openhands browser upload-via 'testid=import-automation-dropzone >> text=Drop a .json file here' "$OH_VERIFY_RUN/evidence/_fixtures/qa-f22-import.automation.json": preview again. control-openhands browser press Escape closes the dialog (browser count 'testid=import-automation-modal' 0).

Drop a fileF22.import-dropzoneMapped behavior

Scenario F22--drop-a-file · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Provide the fixture file or pasted content using “Import automation dropzone”.
Do
Provide the fixture file or pasted content using “Import automation dropzone”.

Expect

dragging a file over the dropzone highlights it (data-active="true"); dropping a .json file previews it like Choose file.

Exact CLI recipe & bug checks

Open the import dialog again and run control-openhands browser drop-files 'testid=import-automation-dropzone' "$OH_VERIFY_RUN/evidence/_fixtures/qa-f22-import.automation.json" --stage over: browser attr 'testid=import-automation-dropzone' data-active is true (focus border). Close and reopen the dialog, then control-openhands browser drop-files 'testid=import-automation-dropzone' "$OH_VERIFY_RUN/evidence/_fixtures/qa-f22-import.automation.json": data-view is preview with Name QA_F22 Imported (browser text 'testid=import-automation-modal'). Close with testid=import-automation-modal-close.

PhoneF22.phoneMapped behavior

Scenario F22--phone · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Use the phone viewport.
Do
Open /automations/templates.
Do
Open /automations.
Setup
Use the desktop viewport.

Expect

the Templates grid, the setup dialog and the import dialog fit a 390 px viewport without horizontal overflow.

Exact CLI recipe & bug checks

Run control-openhands browser viewport phone, control-openhands browser goto /automations/templates, control-openhands browser bbox 'testid=recommended-automations-section' (pageHorizontalOverflow false) and browser screenshot --feature F22.phone --name templates. Click testid=recommended-automation-card-custom-automation, then testid=setup-continue-button; browser bbox 'testid=setup-dialog' is insideViewport true with no page overflow (--name setup). Close it, control-openhands browser goto /automations, open Add Automation → Import automation, upload qa-f22-import.automation.json; browser bbox 'testid=import-automation-modal' is insideViewport true (--name import-preview). Close with testid=import-automation-modal-close, then control-openhands browser viewport desktop.

CleanupF22 · setup / cleanupSupport step

Scenario F22--cleanup · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Setup
Arrange the prerequisite backend state through the API.
Do
Choose “Secret row” / containing “OPENHANDS_URL” / “Delete secret”.
Do
Open /automations.
Do
Choose “Automations empty” / “Automations create automation”.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

Deleting an automation is F21's flow; here remove the fixtures with control-openhands api DELETE /api/automation/v1/<id> --write for each QA_F22 … (Custom, Plugin, Tarball, Imported, View, Agent) and Daily news digest id from control-openhands api GET '/api/automation/v1?limit=100'. Delete OPENHANDS_URL in Settings → Secrets (control-openhands browser click 'testid=secret-item >> has-text=OPENHANDS_URL >> testid=delete-secret-button', then testid=confirmation-modal >> testid=confirm-button, as in F14.delete) unless the next recipe wants it. The fallback and setup conversations stay in the conversation list (F04 deletes them). With zero automations, control-openhands browser goto /automations and control-openhands browser click 'testid=automations-empty >> testid=automations-create-automation' --expect-url '/automations/setup' opens the agent-assisted setup page.

Malformed catalog entryF22.template-card · F22.templates-pageKnown failure

Scenario F22--malformed-catalog-entry · Permalink · Source

Given

Use F22’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

Do
Open <fixture-origin>/automations/templates.

Expect

/automations/templates shows the h1 Templates, a description, a search box, Start from a proven workflow with a count badge (6) and a Beta section (17).

Known bug observations

Known failure, reproduced 2026-10-12 on fa1f724: the whole page renders Application Error, TypeError: Cannot read properties of undefined (reading 'integrations') (#16277). Record this as catalog-data fault injection, then stop/remove only the private fixture frontend/build; never modify the shared package or build.

Exact CLI recipe & bug checks

Archive the frozen target into an isolated private checkout, link unchanged dependencies except a private copy of the pinned @openhands/extensions, and remove only requires from automations/catalog/custom-automation/manifest.json. Leave Canvas source and its lockfile unchanged; run npm run build:app only in that copy. Serve its flat build/ directory against an owned healthy backend, then open /automations/templates through browser goto <fixture-origin>/automations/templates --allow-external. Expected: the malformed entry degrades without crashing the valid catalog. Known failure, reproduced 2026-10-12 on fa1f724: the whole page renders Application Error, TypeError: Cannot read properties of undefined (reading 'integrations') (#16277). Record this as catalog-data fault injection, then stop/remove only the private fixture frontend/build; never modify the shared package or build.

Family notes, prerequisites for known issues & limits Known failureKnown issue

Expected: a completed draft test updates the open Test runs row to Successful automatically. Known failure (reproduced again 2026-10-13 on a desktop prompt draft: run COMPLETED at 08:13:39, row still Pending after a further 45 s wait; verified 2026-10-10, fresh Local stacks, Agent Server 1.54.0/automation 1.19.3): callback-only script tests and genuine DeepSeek prompt/plugin tests report COMPLETED while the open row stays Pending. The desktop prompt stayed Pending through a 45-second wait; the phone plugin capture was more than 201 seconds after completion. Reload and View test runs recover Successful (#18263). Ordinary final-automation Run now polling passes; reload recovery does not waive the draft-Test failure.
The Save toast can cover the setup toolbar Test button; use the lower automation-setup-draft-test and verify the actual run outcome. The toast does not time out while the pointer rests on it, so move the pointer away (browser hover on a form field) before clicking the toolbar again.
The setup page's docked composer keeps no draft: it opens empty even when the Home composer has an unsent draft (sessionStorage oh:home-prompt-draft), a prompt sent from it never becomes the Home draft, and leaving it keeps an existing Home draft. Unsent text typed there is lost when the user leaves the page.
In the desktop split view (form beside the conversation) the Frequency strip scrolls horizontally inside the narrower form and the selected Custom button sits off-screen; scroll it into view before reading it.
Dashboard draft Delete asks in the named dialog role=dialog[name="Delete draft?"] (F21.saved-drafts). Deleting a draft also removes its materialized test automation.
A card click and a deep link are different paths: from the grid, GitHub/Slack-only responders (qa-changes, github-agents-md-maintainer, slack-channel-monitor, …) first show the responder choice and then the GitHub/Slack install dialog, so their setup form is reachable locally only by URL (/automations/new/<id>).
A card's N MCPs to connect before launch pill counts optional integrations too: Custom automation shows 1 MCPs to connect before launch for GitHub, yet opens its setup at once because GitHub is optional there (#17959).
The featured-section copy says picking a card opens "a pre-filled conversation"; most featured cards open a setup form instead.
The setup dialog's comboboxes (Action, Agent profile, Timezone, LLM profile) are autocomplete inputs: click the field and pick the option in the same breath, scoped as role=listbox >> role=option[name="…"]. A browser snapshot between the two closes the listbox (reproduced twice), and the option click then times out after 30 s; click the field again.
Service preflight runs 400 ms after a blur, not on every keystroke; browser press Tab then browser wait for the -error element.
Confirm and create creates the automation enabled; only imports arrive disabled. Use a yearly cron (0 0 1 1 *) so nothing fires during the run.
The Daily news digest entry names its automation Daily news digest (no QA_ prefix); remember it in cleanup.
The Agent profile combobox exists only while the chosen action creates through /v1 (Upload tarball, bundle entries): the automation service rejects agent_profile_id on /v1/preset/prompt and /v1/preset/plugin, so the Prompt and Plugin actions hide it and switching to them clears a chosen profile. Leave it on Default except in the Upload tarball bullet. A cleared choice is stored as an empty id and shown as Default (#18111).
The automation detail page has no main landmark: read its title with browser text 'role=heading[level=1]', not 'role=main >> …'.
The import success toast is the only path to View automation and disappears after about 5 s.
Continue with local setup silently creates the OPENHANDS_URL secret (value: the app origin) the first time; F14's list then has an extra row.
The Install button in a template's install dialog verifies the MCP connection before saving, so dummy credentials cannot complete the queue; nothing is saved on failure.
Creating the first automation ticks Schedule a task in the sidebar Getting-started checklist (F02); a run that checks the checklist should do so before this file.
Known issue #17959: a template search with no matches leaves a blank area with no message (F22.templates-search).
Known issue #17436 (F22.setup-review-confirm): Prompt-preset automations created after the Canvas 1.17.0 upgrade fail on the first conversation events request while older tarballs work, so the UI shows Failed runs. Recheck with a real LLM key and the reported automation 1.11.1/Agent Server 1.46.0 deployment plus pre-upgrade and finish-tool-wired tarballs; retain this version-specific report separately from the plugin-preset Cloud failure in #16611.
Known issue #15509 (F22.setup-tarball-create): A native Windows event-triggered automation fails while uploading its workspace tarball with Path must be absolute. Recheck a Windows local stack and real event delivery, then observe its failed run/upload; a POSIX tarball upload is not a Windows path-validation check. This run has neither Windows nor the webhook fixture.
Known issue #16611 (F22.setup-plugin-create): a minimal plugin-preset Cloud automation fails on its in-sandbox conversation/events request with HTTP 500, while the identical prompt preset works. Live prerequisites: an authenticated Cloud account, a real model key, and a reviewed no-op plugin fixture for the paired cases. The reported cost-zero error is visible in run logs, but a local prompt-only check cannot establish the Cloud plugin-loading behavior.
Known issue OpenHands/software-agent-sdk#5729 (F22.setup-repository-list): prompt and plugin preset workers call RemoteWorkspace.clone_repos, which asks for lowercase provider-token secret names. A private repository can therefore fail to clone when only the documented uppercase GITHUB_TOKEN (or GitLab/Bitbucket equivalent) exists. Recheck an owned preset with a private repository and only its uppercase secret, using an authorized disposable repository or a reviewed token-gated local Git fixture; a public repository and the Canvas Agent Server's separate Git-provider service do not exercise that lookup.

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.