Engel’s notebook

Automations dashboard and actions

F21 · 35 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F21 Automations dashboard and actions

35 behaviors · 36 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

The Automate dashboard (/automations) is where a user sees every automation at a glance and acts on one without opening it. It shows four overview tiles, a search box, a Filters popover (status, trigger, sort), a grid/list view toggle, a Drafts group of saved setup drafts (Resume, Edit, Test, Delete), and Active and Inactive groups of cards or rows with run health, a recent-runs sparkline and stats. Each automation has a Play (Run now) button and a kebab menu with Run now, View, Export, Edit, Turn on/off and Delete; Edit opens the setup page (/automations/setup?automationId=<id>, F22) in edit mode. The page sits in a Dashboard/Templates sub-page shell. It is reached from the sidebar, the command menu or a pinned home route, and it only exists when the automation interface manifest from @openhands/extensions is admitted.

How a user gets here

  • Sidebar Automate (sidebar-automations-link); hovering it reveals the pin (sidebar-pin-home-toggle-automations).
  • Direct URL /automations; the Templates sub-page is /automations/templates (F22).
  • Command menu (Control+k/Meta+k or command-menu-trigger): type Automations or a keyword such as cron, press Enter.
  • With Automate pinned, / (logo, a bare visit) lands on /automations.
  • Sub-page navigation: automations-navbar-desktop (aside) or automations-navbar-mobile (phone strip), links automations-navigation-list and automations-navigation-templates.
  • Header Add Automation (automations-add-automation) → Create Automation (automations-add-automation-create, /automations/setup, F22) or Import automation (automations-import-automation, F22); on the empty dashboard, the Recommended automations rail cards (recommended-automation-rail-card-<id>).
  • Getting-started checklist item Schedule a task links to /automations (onboarding checklist family).
  • Saved drafts: the Drafts group rows (automation-setup-draft-resume-<draft-id>, -edit-, -test-, -delete-).
  • Per automation: Play (automation-run-now-<id>, name "Run now") and kebab (name "Automation actions") on each card or row; the card itself and the row link open the detail page (F23).

Given

Preconditions:

  • Baseline state (launched, doctored, onboard --skip done) with an active LLM profile for Run now (control-openhands llm preset deepseek, or --api-key-file <file> when the key is in a file).
  • No automations exist (control-openhands api GET /api/automation/v1 shows "total": 0); the recipes start from the empty state.
  • Fixtures are imported through Add Automation → Import automation (F22 owns that flow; here it only arranges). Create the files first; each command prints the path to upload ($OH_VERIFY_RUN/evidence/_fixtures/<name>): control-openhands fixture file --name qa-f21-alpha.automation.json --content '{"version":1,"kind":"automation","spec":{"name":"QA_F21 Alpha","trigger":{"type":"cron","schedule":"0 0 1 1 *","timezone":"UTC"},"enabled":false,"prompt":"Reply with the single word: pong. Do not run any tools."}}', control-openhands fixture file --name qa-f21-beta.automation.json --content '{"version":1,"kind":"automation","spec":{"name":"QA_F21 Beta","trigger":{"type":"event","source":"github","on":"pull_request.opened"},"enabled":false,"prompt":"Beta event-driven QA fixture; never runs."}}', control-openhands fixture file --name qa-f21-fail.automation.json --content '{"version":1,"kind":"automation","spec":{"name":"QA_F21 Fail","trigger":{"type":"cron","schedule":"0 0 1 1 *","timezone":"UTC"},"enabled":false,"prompt":"Reply with the single word: pong.","timeout":1}}'. The yearly cron (0 0 1 1 *) keeps a turned-on fixture from firing on its own.
  • Selectors below avoid run-specific ids: [data-testid^="automation-card-"] >> has-text=<name> (grid) and [data-testid^="automation-list-row-"] >> has-text=<name> (list). The kebab menu renders in a portal with no test ids; scope its items as role=list >> has-text=Export >> role=button[name="..."].
Sidebar entryF21.sidebar-entryMapped behavior

Scenario F21--sidebar-entry · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sidebar automations”.

Expect

the sidebar Automate item opens /automations with the heading Dashboard and the Automate sub-page aside.

Exact CLI recipe & bug checks

From / run control-openhands browser click 'testid=sidebar-automations-link', control-openhands browser url and control-openhands browser testids --filter automations-navigation. The URL is /automations, the main heading is Dashboard, and automations-navigation-list and automations-navigation-templates are listed.

Command menuF21.command-menuMapped behavior

Scenario F21--command-menu · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /.
  2. Do
    Press Control+k.
  3. Do
    Enter “cron” in “Command menu” / the combobox.
  4. Do
    Press Enter.

Expect

the command menu offers Automations ("Review scheduled and webhook automations."), also found by the keywords automate, cron, schedule, webhook and jobs; choosing it opens /automations.

Exact CLI recipe & bug checks

Run control-openhands browser goto / (Sidebar entry ends on /automations, where the URL check would pass at once), control-openhands browser press Control+k, control-openhands browser type 'testid=command-menu >> role=combobox' cron, control-openhands browser snapshot 'testid=command-menu', control-openhands browser press Enter, then control-openhands browser url. The snapshot lists exactly one option, Automations Review scheduled and webhook automations. Go ([selected], under Navigation), and the URL ends in /automations.

Pin as homeF21.pin-homeMapped behavior

Scenario F21--pin-as-home · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Hover over “Sidebar automations”.
  2. Do
    Choose “Sidebar pin home toggle automations”.
  3. Do
    Open /.
  4. Do
    Open /.

Expect

the pin on the sidebar Automate item makes / redirect to /automations; unpinning restores the normal home.

Exact CLI recipe & bug checks

On any page run control-openhands browser hover 'testid=sidebar-automations-link', control-openhands browser click 'testid=sidebar-pin-home-toggle-automations', control-openhands browser attr 'testid=sidebar-pin-home-toggle-automations' aria-pressed (true; aria-label becomes Unpin as home page), then control-openhands browser goto / and control-openhands browser url: the URL is /automations. Unpin with the same hover and click (aria-pressed false); control-openhands browser goto / then stays on /.

Sub-page navigationF21.subpage-navMapped behavior

Scenario F21--sub-page-navigation · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /automations.
  2. Do
    Choose “Automations navbar desktop” / “Automations navigation templates”.
  3. Do
    Choose “Automations navbar desktop” / “Automations navigation list”.
  4. Setup
    Use the tablet viewport.
  5. Setup
    Use the desktop viewport.

Expect

Dashboard and Templates sub-pages appear in a left aside from 1024 px up and in a horizontal strip below that (phones and tablets); the current page has aria-current="page".

Exact CLI recipe & bug checks

On /automations (control-openhands browser goto /automations; Pin as home ends on /) run control-openhands browser click 'testid=automations-navbar-desktop >> testid=automations-navigation-templates' (URL /automations/templates, h1 Templates), control-openhands browser click 'testid=automations-navbar-desktop >> testid=automations-navigation-list' and control-openhands browser attr 'testid=automations-navbar-desktop >> testid=automations-navigation-list' aria-current (page). Below 1024 px the same links live under testid=automations-navbar-mobile (see Phone), tablets included: run control-openhands browser viewport tablet (820 px), control-openhands browser visible 'testid=automations-navbar-mobile' (true) and control-openhands browser visible 'testid=automations-navbar-desktop' (false), then control-openhands browser viewport desktop.

Backend unavailableF21.backend-unavailableKnown issue

Scenario F21--backend-unavailable · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Change the run-owned service state: stop automation.
  2. Do
    Open /automations.
  3. Wait
    Wait for the element identified in the CLI recipe to appear.
  4. Do
    Restart the run-owned stack as specified.
  5. Do
    Choose “Retry”.
  6. Wait
    Wait for “Automations empty” to appear.

Expect

when the health check fails, the page shows Automations Unavailable with a Retry button.

Linked known issues

Known issue OpenHands/automation#246 (F21.backend-unavailable, F21.list-error): The report says only that local automation returns 404. It omits the URL, UI action, release and launcher/service setup, so a normal successful dashboard visit would not reproduce or resolve its unknown branch. Ask for those exact prerequisites before writing a failing recipe.

Exact CLI recipe & bug checks

Run control-openhands service stop automation, control-openhands browser goto /automations, control-openhands browser wait-text 'Automations Unavailable', control-openhands browser text 'role=main' and control-openhands browser screenshot --feature F21.backend-unavailable --name unavailable. The main pane reads Dashboard, the subtitle, Automations Unavailable, The automations backend is not available right now. ... and Retry; the sidebar Automate item stays. Then control-openhands restart (the page stays put), control-openhands browser click 'role=button[name="Retry"]' and control-openhands browser wait 'testid=automations-empty': the dashboard returns without a reload.

List errorF21.list-errorKnown issue

Scenario F21--list-error · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/secrets.
  2. Setup
    Change the run-owned service state: stop automation.
  3. Do
    Choose “Sidebar automations”.
  4. Wait
    Wait for the element identified in the CLI recipe to appear.
  5. Do
    Restart the run-owned stack as specified.

Expect

when the list request fails with nothing cached, the page shows Failed to load automations with Retry.

Linked known issues

Known issue OpenHands/automation#246 (F21.backend-unavailable, F21.list-error): The report says only that local automation returns 404. It omits the URL, UI action, release and launcher/service setup, so a normal successful dashboard visit would not reproduce or resolve its unknown branch. Ask for those exact prerequisites before writing a failing recipe.

Exact CLI recipe & bug checks

The health result is cached for 30 s, and every page's sidebar checklist fetches it but not the 50-row list. Run control-openhands browser goto /settings/secrets, then within 30 s control-openhands service stop automation, control-openhands browser click 'testid=sidebar-automations-link' --expect-url '/automations' and control-openhands browser wait-text 'Failed to load automations'. The header, tiles (zeros) and Filters render above Failed to load automations and Retry; screenshot with --feature F21.list-error --name list-error. Then control-openhands restart, click role=button[name="Retry"] and wait for testid=automations-empty. Starting from / does not work: home loads the same list, so the cached copy is shown instead of the error.

List loadingF21.list-loadingMapped behavior

Scenario F21--list-loading · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/secrets.
  2. Do
    Choose “Sidebar automations”.

Expect

once the backend is healthy, the header, tiles and controls render at once while three card skeletons stand in for the list until the first page arrives.

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/secrets, then control-openhands browser click 'testid=sidebar-automations-link' --observe '[data-testid="automation-card-skeleton"], [data-testid="automations-add-automation"]' --observe-ms 2000. One observed state is Add Automation plus three empty skeleton entries (about 50 ms), followed by Add Automation alone.

Health loadingF21.health-loadingMapped behavior

Scenario F21--health-loading · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Blocked on this run. The health-only skeleton (title and three automation-card-skeleton, no header buttons) shows only on the first health fetch, which the sidebar checklist always makes before a user can reach the dashboard, and stopping the service makes the fetch fail at once rather than hang.

Expect

while the health check runs, the page shows the title and three card skeletons.

Exact CLI recipe & bug checks

Blocked on this run. The health-only skeleton (title and three automation-card-skeleton, no header buttons) shows only on the first health fetch, which the sidebar checklist always makes before a user can reach the dashboard, and stopping the service makes the fetch fail at once rather than hang.

Missing manifestF21.manifest-404Known failure

Scenario F21--missing-manifest · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Use an isolated worktree at the frozen target and a private copy of its pinned @openhands/extensions package. Remove only the AUTOMATION_INTERFACE export from the copied automations/index.js before npm run build:app; leave Canvas source and the lockfile unchanged. Serve that frontend against an owned healthy Local backend. Open / and /automations: the sidebar Automate link should be absent, /automations should render 404, and Home should have no Schedule a task link, recommendation section, or no-manifest error toast. On main@3500d9e5 the route/sidebar checks pass but Home shows the dead link and two No automation interface manifest is admitted toasts (#18210; draft fix PR #18212). Record the alternate build as fault injection; the unmodified pinned package does include the manifest. Stop the private frontend and remove only this test worktree/build afterward.

Expect

without an admitted automation interface manifest, /automations is a 404 from the layout error boundary, and all Automate entry points, including the Home checklist and recommendations, are hidden without error toasts.

Known bug observations

Known failure: Canvas hides the sidebar entry but leaves a dead Home checklist link and raises two manifest errors (#18210; draft fix PR #18212).

Exact CLI recipe & bug checks

Use an isolated worktree at the frozen target and a private copy of its pinned @openhands/extensions package. Remove only the AUTOMATION_INTERFACE export from the copied automations/index.js before npm run build:app; leave Canvas source and the lockfile unchanged. Serve that frontend against an owned healthy Local backend. Open / and /automations: the sidebar Automate link should be absent, /automations should render 404, and Home should have no Schedule a task link, recommendation section, or no-manifest error toast. On main@3500d9e5 the route/sidebar checks pass but Home shows the dead link and two No automation interface manifest is admitted toasts (#18210; draft fix PR #18212). Record the alternate build as fault injection; the unmodified pinned package does include the manifest. Stop the private frontend and remove only this test worktree/build afterward.

Empty stateF21.empty-stateMapped behavior

Scenario F21--empty-state · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /automations.
  2. Do
    Choose “Automations empty” / “Automations create automation”.

Expect

with zero automations and zero saved drafts the page shows No automations configured, the "Create an automation" instructions, a docs link, Create Automation (opens /automations/setup, F22, without creating a conversation) and the Recommended automations rail; the view toggle is disabled.

Exact CLI recipe & bug checks

With no automations run control-openhands browser goto /automations, control-openhands browser text 'testid=automations-empty', control-openhands browser enabled 'testid=automations-view-toggle' and control-openhands browser screenshot --feature F21.empty-state --name empty. The text starts No automations configured, contains Create an automation, Learn more in the documentation and Create Automation, and lists the Recommended automations; enabled is false; the tiles read Automations 0 / 0 active and Needs attention 0 / No latest-run failures. control-openhands browser click 'testid=automations-empty >> testid=automations-create-automation' --expect-url '/automations/setup' opens the setup page (form plus docked composer, F22) and control-openhands conversation list is unchanged (no conversation until the user sends a prompt). The empty-state copy still says "Start a new conversation and tell OpenHands to Create an automation".

Add Automation menuF21.add-menuMapped behavior

Scenario F21--add-automation-menu · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations add automation”.
  2. Do
    Press Escape.
  3. Do
    Choose “Automations add automation create”.
  4. Do
    Choose “Automations import automation”.

Expect

the header Add Automation menu offers Create Automation and Import automation; Create Automation opens the guided setup page at /automations/setup (F22).

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=automations-add-automation' and control-openhands browser snapshot 'testid=automations-add-automation-menu': buttons Create Automation and Import automation. control-openhands browser press Escape (or a click on role=heading[name="Dashboard"]) makes browser count 'testid=automations-add-automation-menu' 0. Reopen it and control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup': the agent-assisted setup page opens. Go back to /automations, reopen the menu, and control-openhands browser click 'testid=automations-import-automation': the import modal is covered by F22.

Recommended rail on the empty dashboardF21.empty-railMapped behavior

Scenario F21--recommended-rail-on-the-empty-dashboard · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations empty” / “Recommended automation rail card custom automation”.
  2. Do
    Go back in browser history.

Expect

a Recommended automations card on the empty dashboard opens its template setup at /automations/new/<template-id> (F22 owns the setup flow).

Exact CLI recipe & bug checks

With no automations, control-openhands browser click 'testid=automations-empty >> testid=recommended-automation-rail-card-custom-automation' --expect-url 'automations/new' lands on /automations/new/custom-automation with the Before you start prerequisites dialog (F22). control-openhands browser back returns to /automations; there browser attr 'testid=recommended-automations-rail-fade-left' data-visible is false and ...-fade-right is true (scrolling is F03.recommended-automations-rail).

Saved draftsF21.saved-drafts · F21.searchMapped behavior

Scenario F21--saved-drafts · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /automations.
  2. Do
    Use browser tooltip as specified in the exact recipe.
  3. Do
    Reload the page.
  4. Do
    Press Tab.
  5. Do
    Press Escape.
  6. Do
    Use browser mouse click as specified in the exact recipe.
  7. Do
    Choose “Automation setup draft delete confirm”.

Expect

saved setup drafts (#17242) appear in a Drafts group above Active/Inactive, one row each with the name (Resume), a pencil (Edit), Play (Test draft, tooltip Test Run; disabled for event-triggered or non-dispatchable drafts) and a trash (Delete draft). Resume and Edit open /automations/setup?draftId=<id>; Test dispatches a test run (toast Test run started.) on a materialized DRAFT-state automation that the groups do not list; Delete asks Delete draft? (Delete “<name>”? This removes the saved setup draft, but does not delete any final automations.) with Cancel / Delete draft and toasts Draft deleted., also removing the draft's materialized test automation. The confirmation is the named modal dialog Delete draft? (the automation delete dialog with draft copy): it opens with focus on Cancel, keeps Tab inside, closes on Escape, X, Cancel or a backdrop click and returns focus to the trash button.

the search box narrows the visible automations client-side by name, prompt, repository or model, and the saved drafts by name, prompt, kind or trigger; the tiles do not change.

Exact CLI recipe & bug checks

Arrange one draft through the setup page (F22's Open and save a draft: name QA_F22 Delta Draft, prompt Reply pong. Do not run any tools., custom cron 0 0 1 1 *, Save draft), then control-openhands browser goto /automations. control-openhands browser text 'role=main' lists Drafts 1 / QA_F22 Delta Draft above the groups, and control-openhands browser testids 'role=main' --filter draft lists automation-setup-draft-<draft-id>, -resume-, -open-, -edit- (Edit), -test- (Test draft) and -delete- (Delete draft). control-openhands browser tooltip 'testid=automation-setup-draft-test-<draft-id>' is Test Run; clicking it toasts Test run started. and the draft (api GET /api/automation/v1/drafts/<draft-id>) gains materialized_automation_id and last_test_run_id; that automation has state DRAFT, is not listed in any group, and its detail page shows the Draft badge (F23.header). Once its run is COMPLETED (poll api GET '/api/automation/v1/<materialized-id>/runs?limit=10'), control-openhands browser reload leaves the tiles at Automations 0 / 0 active, Total runs 0 and Average duration — (browser text 'testid=overview-tile-total-runs'): the test automation is not counted. testid=automation-setup-draft-resume-<draft-id> and testid=automation-setup-draft-edit-<draft-id> each open /automations/setup?draftId=<draft-id> (--expect-url '/automations/setup') with the saved values. Search narrows drafts too: filling Delta leaves 0 rows and 1 draft; zzqa-nomatch shows the filtered empty state with no draft. Delete: click testid=automation-setup-draft-delete-<draft-id>; control-openhands browser snapshot 'role=dialog[name="Delete draft?"]' shows button "Close", heading Delete draft?, Delete “QA_F22 Delta Draft”? This removes the saved setup draft, but does not delete any final automations., Cancel and Delete draft, and control-openhands browser eval "document.activeElement.textContent" is Cancel; browser press Tab (repeated) cycles Delete draft, Close and Cancel without leaving the dialog. control-openhands browser press Escape makes browser count 'role=dialog[name="Delete draft?"]' 0 with the draft kept and focus back on automation-setup-draft-delete-<draft-id>; reopening and closing with 'role=dialog[name="Delete draft?"] >> role=button[name="Close"]', ... role=button[name="Cancel"] and a backdrop click (control-openhands browser mouse-click 1400 950) each give 0 too. At phone width the confirm button is insideViewport (browser bbox 'testid=automation-setup-draft-delete-confirm'). control-openhands browser click 'testid=automation-setup-draft-delete-confirm' toasts Draft deleted.; drafts total is 0 and the materialized test automation answers 404.

Arrange two automations (import, F22)F21 · setup / cleanupSupport step

Scenario F21--arrange-two-automations-import-f22 · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations add automation”.
  2. Do
    Choose “Automations import automation”.
  3. Do
    Use browser upload as specified in the exact recipe.
  4. Do
    Choose “Import automation confirm”.
  5. Wait
    Wait for “Import automation modal” to disappear.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=automations-add-automation', control-openhands browser click 'testid=automations-import-automation', control-openhands browser upload 'testid=automations-import-file' "$OH_VERIFY_RUN/evidence/_fixtures/qa-f21-alpha.automation.json", control-openhands browser click 'testid=import-automation-confirm', control-openhands browser wait 'testid=import-automation-modal' --state detached. Repeat with qa-f21-beta.automation.json. Both arrive disabled, under Inactive 2.

Grid cardsF21.card-gridKnown issue

Scenario F21--grid-cards · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Reload the page.

Expect

a grid card shows name, prompt preview, trigger pills, sparkline, a status strip (latest run status, summary, relative time, or No activity yet) and Runs / Recent success / Avg. duration stats.

Linked known issues

Known issue OpenHands/automation#347 (F21.card-grid, F21.overview-tiles): per-automation run-summary reads lack an aggregate contract, and the report describes a 30 s shared-resource stall with HTTP 500 plus SQLite pool settings that do not apply. A failing recipe needs the original database/lock workload and redacted service-log/HAR timing evidence, which the report withholds. Successful summary reads under the Canvas request throttle do not establish those backend contracts.

Exact CLI recipe & bug checks

Run control-openhands browser reload, control-openhands browser snapshot 'role=main' and control-openhands browser screenshot --feature F21.card-grid --name grid. Each card shows its name, prompt, pills (pull_request.opened + GitHub logo for Beta, cron for Alpha), No activity yet, and Runs 0 / Recent success — / Avg. duration —; both Play buttons are [disabled].

Kebab menuF21.kebab-menuMapped behavior

Scenario F21--kebab-menu · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Press Escape.

Expect

the Automation actions menu lists Run now, View, Export, Edit, Turn on/Turn off and Delete, closes on Escape or an outside click, and flips above the trigger near the viewport bottom.

Exact CLI recipe & bug checks

Run control-openhands browser click '[data-testid^="automation-card-"] >> has-text=QA_F21 Alpha >> role=button[name="Automation actions"]' and control-openhands browser snapshot 'role=list >> has-text=Export': buttons Run now [disabled], View, Export, Edit, Turn on, Delete. control-openhands browser press Escape, then control-openhands browser count 'role=list >> has-text=Export' is 0. Reopen it and click role=heading[name="Dashboard"]; the count is 0 again.

Turn onF21.toggleMapped behavior

Scenario F21--turn-on · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the list / containing “Export” / “Turn on”.
  2. Do
    Reload the page.

Expect

kebab Turn on / Turn off flips the enabled flag immediately (no confirmation on the dashboard) and moves the automation between the Active and Inactive groups.

Exact CLI recipe & bug checks

Open Alpha's kebab as above, run control-openhands browser click 'role=list >> has-text=Export >> role=button[name="Turn on"]', then control-openhands browser reload and control-openhands browser snapshot 'role=main'. No dialog appears; an Active 1 group holds QA_F21 Alpha, the Automations tile reads 2 / 1 active, and Alpha's Play is enabled (control-openhands browser enabled '[data-testid^="automation-card-"] >> has-text=QA_F21 Alpha >> role=button[name="Run now"]' is true).

Run now from PlayF21.run-nowMapped behavior

Scenario F21--run-now-from-play · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

Play or kebab Run now on an enabled automation dispatches a run with the toast Automation dispatched; the card moves through Pending/Running to the result. Both are disabled on a disabled automation.

Exact CLI recipe & bug checks

Run control-openhands browser click '[data-testid^="automation-card-"] >> has-text=QA_F21 Alpha >> role=button[name="Run now"]', control-openhands browser wait-text 'Automation dispatched' and control-openhands browser screenshot --feature F21.run-now --name toast. The URL stays /automations, the status strip reads Pending, then Running, and Total runs becomes 1. About 40 s later, after a reload, the strip reads Successful with the agent's summary and the sidebar lists a conversation titled QA_F21 Alpha — <UTC timestamp> (control-openhands conversation list).

List viewF21.view-toggle · F21.list-rowMapped behavior

Scenario F21--list-view · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations view toggle”.
  2. Do
    Choose “Automations view toggle list”.
  3. Do
    Reload the page.

Expect

the view-mode menu switches Grid and List; the choice survives a reload.

a list row shows a health dot, name, schedule or event, relative last run, run badge, sparkline, Play and kebab; its link is labelled <name> <status>.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=automations-view-toggle', control-openhands browser click 'testid=automations-view-toggle-list', control-openhands browser reload, control-openhands browser count '[data-testid^="automation-list-row-"]' (2) and control-openhands browser eval "localStorage.getItem('openhands-automations-view')" ("list"). Then control-openhands browser text '[data-testid^="automation-list-row-"] >> has-text=QA_F21 Alpha' reads QA_F21 Alpha / 0 0 1 1 * / <relative time> / <status>, and control-openhands browser attr '[data-testid^="automation-list-row-"] >> has-text=QA_F21 Alpha >> role=link >> nth=0' aria-label is QA_F21 Alpha <status> (for example QA_F21 Alpha Running). Screenshot with control-openhands browser screenshot --feature F21.list-row --name list. Escape closes the open view menu (control-openhands browser count 'testid=automations-view-toggle-list' is 0).

Hover previewsF21.row-tooltip · F21.summary-hovercardMapped behavior

Scenario F21--hover-previews · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Use browser tooltip as specified in the exact recipe.
  2. Do
    Use browser tooltip as specified in the exact recipe.
  3. Do
    Choose “Automations view toggle”.
  4. Do
    Choose “Automations view toggle grid”.
  5. Do
    Use browser tooltip as specified in the exact recipe.

Expect

hovering a list row shows a preview card with the name, Trigger, Status and, once it has run, Last run and Task (the run summary); the Play button's tooltip reads Run now.

hovering a grid card's truncated run summary shows the full summary in a hovercard.

Exact CLI recipe & bug checks

After Alpha's run has succeeded, in list view run control-openhands browser tooltip '[data-testid^="automation-list-row-"] >> has-text=QA_F21 Alpha >> role=link >> nth=0': QA_F21 Alpha / Trigger / 0 0 1 1 * / Status / Successful / Last run / <relative time> / Task / <the run summary> (a row without runs shows only Trigger and Status / No activity yet). control-openhands browser tooltip '[data-testid^="automation-list-row-"] >> has-text=QA_F21 Alpha >> role=button[name="Run now"]' is Run now. In grid view (control-openhands browser click 'testid=automations-view-toggle', control-openhands browser click 'testid=automations-view-toggle-grid'), control-openhands browser tooltip '[data-testid^="automation-card-"] >> has-text=QA_F21 Alpha >> text=Replied' (the start of the truncated summary) returns the full summary, for example Replied with the single word "pong" as requested, without running any tools. Switch back with testid=automations-view-toggle and testid=automations-view-toggle-list: the next bullets count list rows.

Sparkline deep linkF21.sparkline-deeplinkMapped behavior

Scenario F21--sparkline-deep-link · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Open /automations.

Expect

a sparkline bar links to /automations/<id>?run=<runId>, which opens the detail page with that run highlighted.

Exact CLI recipe & bug checks

After Alpha has a run, run control-openhands browser attr '[data-testid^="automation-activity-"] >> role=link >> nth=0' href, control-openhands browser click '[data-testid^="automation-activity-"] >> role=link >> nth=0', control-openhands browser url and control-openhands browser testids --filter run. The href and URL are /automations/<id>?run=<runId>, and automation-run-highlight-<runId> is visible. Return with control-openhands browser goto /automations.

SearchF21.searchMapped behavior

Scenario F21--search · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “pong” in “Search automations...”.

Expect

the search box narrows the visible automations client-side by name, prompt, repository or model, and the saved drafts by name, prompt, kind or trigger; the tiles do not change.

Exact CLI recipe & bug checks

Run control-openhands browser fill 'role=textbox[name="Search automations..."]' pong and control-openhands browser count '[data-testid^="automation-list-row-"]'. The count is 1 (Alpha matches by prompt only). Then fill zzqa-nomatch; control-openhands browser text 'testid=overview-tile-automations' still reads Automations 2 / 1 active.

Filtered empty and Clear filtersF21.filtered-emptyMapped behavior

Scenario F21--filtered-empty-and-clear-filters · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations clear filters”.

Expect

when nothing matches, No automations match these filters and Clear filters appear; Clear filters resets search, status and trigger but keeps the sort.

Exact CLI recipe & bug checks

With zzqa-nomatch in the search box run control-openhands browser text 'testid=automations-filtered-empty' (No automations match these filters / Clear filters), control-openhands browser click 'testid=automations-clear-filters', control-openhands browser value 'role=textbox[name="Search automations..."]' (empty) and the row count (2).

FiltersF21.filtersMapped behavior

Scenario F21--filters · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations filters” / “Dropdown trigger” / match 1.
  2. Do
    Choose “Automations filter status” / “Dropdown trigger”.
  3. Do
    Choose “Automations filter status disabled”.
  4. Do
    Choose “Automations filter trigger” / “Dropdown trigger”.
  5. Do
    Choose “Automations filter trigger schedule”.
  6. Do
    Choose “Automations filters reset”.
  7. Do
    Choose “Dashboard”.
  8. Do
    Reload the page.

Expect

the Filters popover filters by status (All / Active / Needs attention / Disabled) and trigger (All / Scheduled / Event-driven), counts non-default choices in a badge and offers Reset all; it stays open while the user changes several choices.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=automations-filters >> testid=dropdown-trigger >> nth=0', control-openhands browser snapshot 'testid=automations-filters-menu' (Filter by status All statuses, Filter by trigger All triggers, Sort automations Latest run), control-openhands browser click 'testid=automations-filter-status >> testid=dropdown-trigger', control-openhands browser click 'testid=automations-filter-status-disabled', then control-openhands browser text 'testid=automations-filters >> testid=dropdown-trigger >> nth=0' (Filters 1) and the row text (only QA_F21 Beta). The popover stays open: control-openhands browser count 'testid=automations-filters-menu' is 1 and control-openhands browser attr 'testid=automations-filters >> testid=dropdown-trigger >> nth=0' aria-expanded is true. In the same popover run control-openhands browser click 'testid=automations-filter-trigger >> testid=dropdown-trigger' and control-openhands browser click 'testid=automations-filter-trigger-schedule': the badge is Filters 2, testid=automations-filtered-empty appears and the menu count is still 1 (control-openhands browser screenshot --feature F21.filters --name two-filters). control-openhands browser click 'testid=automations-filters-reset': both rows return, the button reads Filters and the popover stays open. Trigger Event-driven (automations-filter-trigger-event, chosen in the same open popover) alone leaves only QA_F21 Beta. An outside click closes the popover: control-openhands browser click 'role=heading[name="Dashboard"]', then the menu count is 0 and aria-expanded is false. Run control-openhands browser reload to clear the filter (filters are page state): the next bullets need Alpha visible.

Turn offF21.toggleMapped behavior

Scenario F21--turn-off · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Choose the list / containing “Export” / “Turn off”.
  3. Do
    Reload the page.

Expect

kebab Turn on / Turn off flips the enabled flag immediately (no confirmation on the dashboard) and moves the automation between the Active and Inactive groups.

Exact CLI recipe & bug checks

In list view run control-openhands browser click '[data-testid^="automation-list-row-"] >> has-text=QA_F21 Alpha >> role=button[name="Automation actions"]', control-openhands browser click 'role=list >> has-text=Export >> role=button[name="Turn off"]', control-openhands browser count 'testid=confirmation-modal' (0, no confirmation), control-openhands browser reload. The page shows Inactive 2 with the tile at 0 active, and Play reports enabled false.

Edit from the dashboardF21.editKnown failure

Scenario F21--edit-from-the-dashboard · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the list / containing “Export” / “Edit”.
  2. Do
    Enter “QA_F21 Zulu” in “Automation setup name”.
  3. Do
    Choose “Automation setup save”.
  4. Do
    Hover over “Automation setup name”.
  5. Do
    Choose “Automation setup close”.
  6. Do
    Open /automations/setup?automationId=<id>.

Expect

kebab Edit opens the setup page in edit mode (/automations/setup?automationId=<id>) prefilled from the automation, with Test, Save and Close in its toolbar; Save sends one PATCH and toasts Automation updated, Close returns to /automations (field-level editing is F23). For a script (tarball) automation it should summarize the saved bundle (Existing code bundle).

Known bug observations

Known failure (reproduced 2026-10-13, #18341): for a tarball automation (create one with F22's Upload tarball bullet), kebab Edit shows Custom Python with generated starter code (browser value 'testid=automation-setup-custom-code' contains def fire_callback and an empty prompt) and browser count 'testid=automation-setup-existing-bundle' is 0; expected the Existing code bundle summary, which a cold browser goto '/automations/setup?automationId=<id>' does show.

Known failure (reproduced 2026-10-13, #18341): from the kebab it shows generated starter Python and a default setup script instead; a cold load of the same URL shows the bundle summary.

Exact CLI recipe & bug checks

Turn off (above) left both automations in the Inactive group, as Sort needs. Open Alpha's kebab and control-openhands browser click 'role=list >> has-text=Export >> role=button[name="Edit"]' --expect-url '/automations/setup': the URL is /automations/setup?automationId=<id>, control-openhands browser value 'testid=automation-setup-name' is QA_F21 Alpha, automation-setup-prompt holds the prompt, automation-setup-custom-schedule is 0 0 1 1 * (Frequency Custom, timezone UTC), Timeout 600, and browser testids 'testid=automation-setup-toolbar' lists automation-setup-test, automation-setup-save and automation-setup-close; control-openhands conversation list is unchanged. Run control-openhands browser fill 'testid=automation-setup-name' 'QA_F21 Zulu', control-openhands browser network --clear, control-openhands browser click 'testid=automation-setup-save' and control-openhands browser toasts --history (Automation updated); control-openhands browser network --bodies --filter 'automation/v1' shows one PATCH /api/automation/v1/<id> with name, trigger, model: null (Active profile kept), agent_profile_id: null, timeout and prompt, and control-openhands api GET /api/automation/v1 shows QA_F21 Zulu with the schedule unchanged. The toast sits over the toolbar and stays while the pointer rests on it (see Gotchas): run control-openhands browser hover 'testid=automation-setup-name' and wait about 5 s, then control-openhands browser click 'testid=automation-setup-close' --expect-url '/automations$'. At phone width the same kebab Edit on QA_F21 Beta (event trigger) shows Event selected with Event source GitHub and no horizontal overflow (browser bbox 'testid=automation-setup-form'). Script automation: Known failure (reproduced 2026-10-13, #18341): for a tarball automation (create one with F22's Upload tarball bullet), kebab Edit shows Custom Python with generated starter code (browser value 'testid=automation-setup-custom-code' contains def fire_callback and an empty prompt) and browser count 'testid=automation-setup-existing-bundle' is 0; expected the Existing code bundle summary, which a cold browser goto '/automations/setup?automationId=<id>' does show.

SortF21.sortMapped behavior

Scenario F21--sort · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations sort” / “Dropdown trigger”.
  2. Do
    Choose “Automations sort name”.
  3. Do
    Choose “Dashboard”.

Expect

Sort (Latest run / Most runs / Name) orders automations inside each group and counts as a non-default choice.

Exact CLI recipe & bug checks

With QA_F21 Zulu (1 run) and QA_F21 Beta (no runs) both inactive, open Filters, control-openhands browser click 'testid=automations-sort >> testid=dropdown-trigger', control-openhands browser click 'testid=automations-sort-name', then control-openhands browser text '[data-testid^="automation-list-row-"] >> nth=0'. Beta comes first, the badge reads Filters 1 and the popover stays open. Fill the search with zzqa-nomatch and click testid=automations-clear-filters (a click outside the popover, which closes it): the badge stays Filters 1 and Beta stays first (the sort is kept). Open Filters and the sort field again and choose automations-sort-runs; Zulu comes first. Close the popover with control-openhands browser click 'role=heading[name="Dashboard"]' (browser count 'testid=automations-filters-menu' is 0).

ExportF21.exportMapped behavior

Scenario F21--export · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the list / containing “Export” / “Export”.

Expect

kebab Export downloads <name-slug>.automation.json with version, kind: "automation" and the spec.

Exact CLI recipe & bug checks

Open Zulu's kebab, control-openhands browser click 'role=list >> has-text=Export >> role=button[name="Export"]', then control-openhands browser downloads --last 1 --inspect. The download path ends in qa-f21-zulu.automation.json (under <run>/private/downloads/); the inspected head holds "version": 1, "kind": "automation" and a spec with name, trigger, enabled, prompt, model (null for Active profile), timeout and timezone.

Open the detail pageF21.open-detailMapped behavior

Scenario F21--open-the-detail-page · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Choose the element identified in the CLI recipe.
  3. Do
    Choose “Automations view toggle”.
  4. Do
    Choose “Automations view toggle grid”.
  5. Do
    Use browser focus as specified in the exact recipe.
  6. Do
    Press Enter.
  7. Do
    Go back in browser history.

Expect

kebab View, the row link, a card click and Enter on a focused card all open /automations/<id>.

Exact CLI recipe & bug checks

Kebab role=list >> has-text=Export >> role=button[name="View"], control-openhands browser click '[data-testid^="automation-list-row-"] >> has-text=QA_F21 Zulu >> role=link >> nth=0', control-openhands browser click '[data-testid^="automation-card-"] >> has-text=QA_F21 Zulu >> role=heading' (grid: switch first with control-openhands browser click 'testid=automations-view-toggle' and control-openhands browser click 'testid=automations-view-toggle-grid'; the later bullets use cards), and control-openhands browser focus '[data-testid^="automation-card-"] >> has-text=QA_F21 Beta' + control-openhands browser press Enter each make control-openhands browser url read /automations/<that automation's id>. Go back with control-openhands browser back.

Git Sync buttonF21.git-sync-buttonMapped behavior

Scenario F21--git-sync-button · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automations git sync”.

Expect

the header Git Sync button (manage permission; always on local) opens /automations/git-sync (see F24).

Exact CLI recipe & bug checks

control-openhands browser click 'testid=automations-git-sync', then control-openhands browser url: /automations/git-sync (F24).

Needs attention, kebab Run nowF21.overview-tiles · F21.run-now · F21.filtersKnown issue

Scenario F21--needs-attention-kebab-run-now · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /automations.
  2. Do
    Choose the element identified in the CLI recipe.
  3. Wait
    Wait for the element identified in the CLI recipe to appear.
  4. Do
    Reload the page.
  5. Do
    Choose “Dashboard”.

Expect

four tiles summarize the automations the groups list from the loaded org list (hidden DRAFT test automations from a draft Test are left out): Automations (N active), Needs attention (Latest run failed or No latest-run failures; an enabled automation counts when its latest run is Failed or its task outcome is failed or blocked, while Partial and Needs review do not count), Total runs and Average duration.

Play or kebab Run now on an enabled automation dispatches a run with the toast Automation dispatched; the card moves through Pending/Running to the result. Both are disabled on a disabled automation.

the Filters popover filters by status (All / Active / Needs attention / Disabled) and trigger (All / Scheduled / Event-driven), counts non-default choices in a badge and offers Reset all; it stays open while the user changes several choices.

Linked known issues

Known issue OpenHands/automation#347 (F21.card-grid, F21.overview-tiles): per-automation run-summary reads lack an aggregate contract, and the report describes a 30 s shared-resource stall with HTTP 500 plus SQLite pool settings that do not apply. A failing recipe needs the original database/lock workload and redacted service-log/HAR timing evidence, which the report withholds. Successful summary reads under the Canvas request throttle do not establish those backend contracts.

Exact CLI recipe & bug checks

Run control-openhands browser goto /automations (the Git Sync button left /automations/git-sync), import qa-f21-fail.automation.json as above, turn it on from its kebab, then control-openhands browser click '[data-testid^="automation-card-"] >> has-text=QA_F21 Fail >> role=button[name="Automation actions"]', control-openhands browser enabled 'role=list >> has-text=Export >> role=button[name="Run now"]' (true), click it, control-openhands browser wait-text 'Automation dispatched'. The 1 s timeout run ends FAILED after about 3-4 minutes (check with control-openhands api GET /api/automation/v1/<id>/runs; the id is in api GET /api/automation/v1). Then control-openhands browser reload, control-openhands browser text 'testid=overview-tile-needs-attention' (Needs attention 1 / Latest run failed), and control-openhands browser screenshot --feature F21.overview-tiles --name needs-attention. The Fail card shows Failed with Timed out: command timed out or was killed, a red sparkline bar and Recent success 0%; Total runs is 2. Open Filters and choose status automations-filter-status-failing (testid=automations-filter-status >> testid=dropdown-trigger, then testid=automations-filter-status-failing): only QA_F21 Fail is left. In the still-open popover click testid=automations-filters-reset (all three automations return), then close it with control-openhands browser click 'role=heading[name="Dashboard"]': the next bullets need QA_F21 Fail and QA_F21 Beta visible.

Run now errorF21.run-now-errorKnown failure

Scenario F21--run-now-error · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Arrange the prerequisite backend state through the API.
  2. Wait
    Wait for the element identified in the CLI recipe to disappear.
  3. Do
    Reload the page.

Expect

a failed dispatch shows exactly one error toast with the API message; when the automation is gone (Automation not found), its card then drops out of the dashboard without a reload.

Known bug observations

Known failure: the wait times out and the card stays until control-openhands browser reload (#18062, see Gotchas).

Exact CLI recipe & bug checks

Arrange a stale card: with the dashboard open, run control-openhands api DELETE /api/automation/v1/<QA_F21 Fail id> --write and control-openhands browser toasts --history --clear (empties the toast history), then click '[data-testid^="automation-card-"] >> has-text=QA_F21 Fail >> role=button[name="Run now"]', then control-openhands browser toasts --history and control-openhands browser screenshot --feature F21.run-now-error --name error-toast. The history holds exactly one toast, Automation not found, and no Request failed with status code 404. browser errors --app-only then lists the induced 404 on /dispatch. Expected: the card then drops out without a reload: control-openhands browser wait '[data-testid^="automation-card-"] >> has-text=QA_F21 Fail' --state detached --timeout 10000 succeeds. Known failure: the wait times out and the card stays until control-openhands browser reload (#18062, see Gotchas).

Delete with confirmationF21.deleteMapped behavior

Scenario F21--delete-with-confirmation · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Press Tab.
  2. Do
    Choose “Delete automation” / “Cancel”.
  3. Do
    Choose “Delete automation” / “Close”.
  4. Do
    Press Escape.
  5. Do
    Choose “Delete automation” / “Delete”.
  6. Do
    Reload the page.

Expect

kebab Delete asks Delete "<name>"? This cannot be undone.; Cancel and the X keep it, Delete removes it for good, and Escape dismisses the dialog.

Exact CLI recipe & bug checks

Open Beta's kebab and click role=list >> has-text=Export >> role=button[name="Delete"]. control-openhands browser snapshot 'role=dialog[name="Delete automation"]' shows dialog Delete automation with Close, the heading, Delete "QA_F21 Beta"? This cannot be undone., Cancel and Delete; focus starts on Cancel (control-openhands browser eval "document.activeElement.textContent" is Cancel) and browser press Tab / Shift+Tab cycle Cancel, Delete and Close inside the dialog. Take control-openhands browser screenshot --feature F21.delete --name dialog. control-openhands browser click 'role=dialog[name="Delete automation"] >> role=button[name="Cancel"]' closes it (control-openhands browser count 'role=dialog[name="Delete automation"]' is 0) with the card kept. Reopen and close it with the X (control-openhands browser click 'role=dialog[name="Delete automation"] >> role=button[name="Close"]'), then reopen and control-openhands browser press Escape: the dialog count is 0 and the card count 1 each time. Reopen, then control-openhands browser click 'role=dialog[name="Delete automation"] >> role=button[name="Delete"]', control-openhands browser reload and control-openhands browser count '[data-testid^="automation-card-"] >> has-text=QA_F21 Beta'. The count is 0. Delete the remaining fixtures the same way (rows work too) until testid=automations-empty returns.

Load moreF21.load-moreMapped behavior

Scenario F21--load-more · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /automations.
  2. Wait
    Wait for “Load more” to appear.
  3. Do
    Choose “Load more”.
  4. Wait
    Wait for “Load more” to disappear.

Expect

past 50 automations a Load more button fetches the next page and disappears when everything is loaded.

Exact CLI recipe & bug checks

Arrange 51 inert automations (event trigger that never fires): for i in $(seq -w 1 51); do control-openhands api POST /api/automation/v1/preset/prompt --write --data "{\"name\":\"QA_F21 Page $i\",\"prompt\":\"QA paging fixture; never runs.\",\"trigger\":{\"type\":\"event\",\"source\":\"qa-f21\",\"on\":\"qa.f21.never\"}}"; done. Then control-openhands browser goto /automations, control-openhands browser wait 'role=button[name="Load more"]', row/card count 50, control-openhands browser click 'role=button[name="Load more"]', control-openhands browser wait 'role=button[name="Load more"]' --state detached, count 51. The preset fixtures arrive enabled (51 active) but never fire. Keep them for Phone (it needs at least one automation) and clean up after it.

PhoneF21.phone · F21.subpage-navMapped behavior

Scenario F21--phone · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Use the phone viewport.
  2. Do
    Choose “Automations navbar mobile” / “Automations navigation templates”.
  3. Do
    Press Escape.
  4. Setup
    Use the desktop viewport.
  5. Setup
    Arrange the prerequisite backend state through the API.

Expect

at 390 px the dashboard has no horizontal overflow, tiles wrap 2×2 and the kebab menu stays inside the viewport.

Dashboard and Templates sub-pages appear in a left aside from 1024 px up and in a horizontal strip below that (phones and tablets); the current page has aria-current="page".

Exact CLI recipe & bug checks

With at least one automation, run control-openhands browser viewport phone, control-openhands browser bbox 'role=main' (insideViewport true, pageHorizontalOverflow false), control-openhands browser screenshot --feature F21.phone --name dashboard, control-openhands browser click 'testid=automations-navbar-mobile >> testid=automations-navigation-templates' (URL /automations/templates) and back through automations-navigation-list. Open the first card's kebab and run control-openhands browser bbox 'role=list >> has-text=Export': the menu is insideViewport true and sits above the kebab (flipped). Take control-openhands browser screenshot --feature F21.phone --name kebab, then control-openhands browser press Escape and control-openhands browser viewport desktop. Now clean up the paging fixtures with control-openhands api DELETE /api/automation/v1/<id> --write for each QA_F21 Page id from control-openhands api GET '/api/automation/v1?limit=100' (UI deletion is covered above) until total is 0.

ErrorsF21 · setup / cleanupSupport step

Scenario F21--errors · Permalink · Source

Given

Use F21’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    After each group, control-openhands browser errors --app-only shows no page errors; the only app errors are the induced 404s for the deleted QA_F21 Fail: /dispatch and the stale card's /runs?limit=20&offset=0. The backend-down recipes add 502s on /api/automation/health, /sdk-version and the list while the service is stopped; clear them with browser errors --clear after restart.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

After each group, control-openhands browser errors --app-only shows no page errors; the only app errors are the induced 404s for the deleted QA_F21 Fail: /dispatch and the stale card's /runs?limit=20&offset=0. The backend-down recipes add 502s on /api/automation/health, /sdk-version and the list while the service is stopped; clear them with browser errors --clear after restart.

Family notes, prerequisites for known issues & limits Known failureKnown issue
  • Both sub-page navs are always in the DOM (one hidden by CSS). Unscoped testid=automations-navigation-templates is a strict-mode violation; scope it with testid=automations-navbar-desktop >> or testid=automations-navbar-mobile >> .
  • The pin toggle is invisible until the Automate item is hovered on a fine pointer: browser hover 'testid=sidebar-automations-link' before clicking it. The pin is stored per backend and org in localStorage (oh:pinned-home-route:<backend>:<org>); unpin before leaving the run.
  • The Filters popover's status and trigger choices do not apply to the Drafts group: with status Active and trigger Event-driven, a cron draft stays listed and the filtered empty state does not show (search does filter drafts). Intent undocumented in #17242.
  • The setup page's success toast (Automation updated) covers its toolbar's Test, Save and Close and does not time out while the pointer rests on it; a click there fails with "subtree intercepts pointer events". Move the pointer away (browser hover on a form field), wait about 5 s, then click.
  • testid=dropdown-trigger is reused by the backend selector, the Filters button and every inner dropdown. Always scope it (testid=automations-filters >> testid=dropdown-trigger >> nth=0 for the Filters button, testid=automations-filter-status >> testid=dropdown-trigger for a field).
  • The Filters popover stays open while choices change and closes only on a click outside it or on the Filters button; Escape does not close it. Open, it covers the right part of the list (in list view the first rows' Play buttons): close it with a click on role=heading[name="Dashboard"] before clicking row or card controls.
  • A card whose automation was deleted elsewhere stays on the dashboard after its Run now fails with Automation not found. Expected: the card drops out once the dispatch returns 404. Known failure: nothing refetches the list after the failed dispatch, so the card stays until a reload or another list refetch (still there 2 minutes later) (#18062).
  • The delete confirmation has no test ids; scope it as the named dialog role=dialog[name="Delete automation"]. It opens with focus on Cancel.
  • Delete closes the dialog at once and shows no success toast; confirm with a reload. Turn off on the dashboard never asks for confirmation (home cards do).
  • The kebab menu has no test ids; role=list >> has-text=Export is the reliable scope. A card's own Run now has the same accessible name as the kebab item, so scope Play to the card.
  • Imported automations are always disabled (Import automation says so), and Run now is disabled until you Turn on. Use the yearly cron 0 0 1 1 * or an event trigger so an enabled fixture never fires by itself, and turn it back off.
  • Run now uses the active LLM profile when the automation's model is Active profile. A successful run creates a conversation titled <automation name> — <UTC timestamp> that stays after the automation is deleted. Delete it separately (control-openhands conversation list, conversations family) to leave the run clean.
  • A timeout of 1 s does not fail fast: the run sat in Installing dependencies and failed with Timed out: command timed out or was killed about 3-4 minutes after dispatch. Budget for that when you need a failing run for Needs attention.
  • The Automations tile counts only loaded pages: with 51 automations it reads 50 until Load more is clicked, although the API total is 51.
  • The row health dot and label reflect the latest run only; a disabled automation with no runs reads No activity yet, not Disabled.
  • In grid view a cron automation without schedule_human shows the pill cron, while list view shows the raw expression (0 0 1 1 *).
  • The view mode persists in localStorage (openhands-automations-view), and the toggle is disabled when the org has no automations. Switch back to Grid before deleting the last automation, or the next run starts in List.
  • Filters, search and sort are page state only: a reload resets them.
  • service stop automation stops only this run's automation service; restart brings the whole stack back with the same state and the browser on the same page. Run control-openhands doctor afterwards.
  • The Automation dispatched toast lasts only a few seconds: run browser wait-text 'Automation dispatched' right after the click, not after a long --observe window.
  • browser screenshot --full-page at phone width does not capture below the fold, because the page scrolls inside the main pane; screenshot sections or use bbox.
  • Known issue OpenHands/automation#246 (F21.backend-unavailable, F21.list-error): The report says only that local automation returns 404. It omits the URL, UI action, release and launcher/service setup, so a normal successful dashboard visit would not reproduce or resolve its unknown branch. Ask for those exact prerequisites before writing a failing recipe.
  • Known issue OpenHands/automation#347 (F21.card-grid, F21.overview-tiles): per-automation run-summary reads lack an aggregate contract, and the report describes a 30 s shared-resource stall with HTTP 500 plus SQLite pool settings that do not apply. A failing recipe needs the original database/lock workload and redacted service-log/HAR timing evidence, which the report withholds. Successful summary reads under the Canvas request throttle do not establish those backend contracts.

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.