Engel’s notebook

Automation setup page

F28 · 13 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F28 Automation setup page

13 behaviors · 11 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

The setup page is the one form for making and changing automations outside the template dialog. Add Automation → Create Automation opens it blank at /automations/setup; a saved draft reopens at /automations/setup?draftId=<id>; kebab Edit opens an existing automation at /automations/setup?automationId=<id>. The form collects Name, a Prompt (or Custom Python with entrypoint and setup script), agent profile, model and repositories, a Schedule or Event trigger, and optional plugins and timeout. A docked composer under the form can ask the agent to fill it. Create mode's toolbar has Save draft, Test and Create Automation; edit mode's has Test, Save and Close. A server-backed draft carries the server's validation result and test runs; Create turns it into a real, enabled automation and removes the draft.

F22 keeps F22.agent-assisted-setup (the end-to-end agent, plugin, phone and draft-Test recipes); F23 owns edit mode (F23.edit-*, including Test and the save state on the edit page). This family gives each create-mode behavior its own ID so a pass can fail one without failing the others.

How a user gets here

  • Dashboard header Add Automation → Create Automation (automations-add-automation, then automations-add-automation-create): blank create mode. Every create recipe below starts here.
  • Empty dashboard Create Automation (automations-empty >> automations-create-automation): the same blank form (F21.empty-state).
  • Dashboard Drafts group: name or Resume (automation-setup-draft-resume-<draft-id>) and the pencil (Edit, automation-setup-draft-edit-<draft-id>) open ?draftId=<id>.
  • Kebab Edit on a dashboard card or the detail page header: edit mode (?automationId=<id>, F21.edit, F23.edit-open).
  • A direct URL with draftId or automationId (see the cold-load gotcha).

Given

Preconditions:

  • Baseline state (launch, doctor, onboard --skip). No LLM is needed: every bullet uses the Custom Python kind or never dispatches, and the agent-assisted path stays in F22.
  • Desktop viewport; the F28.error-contrast bullet switches color themes (Settings → Application, F16) and restores the default.
  • Every recipe below was run as written on 2026-10-15 on a fresh local control-openhands launch of main with Agent Server 1.54.0 and no LLM. Documented known failures remain failures.
Blank formF28.open-blankMapped behavior

Scenario F28--blank-form · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /automations.
  2. Do
    Choose “Automations add automation”.
  3. Do
    Choose “Automations add automation create”.

Expect

Create Automation opens /automations/setup with Name pre-filled New Automation (placeholder Standup digest), an empty Prompt, Schedule selected with Daily at 09:00 in America/New_York (the default, whatever the browser's time zone), the docked composer (Need help creating an automation?) and no new conversation. The save state already reads Unsaved changes and the toolbar Test is disabled.

Exact CLI recipe & bug checks

Run control-openhands conversation list, control-openhands browser goto /automations, control-openhands browser click 'testid=automations-add-automation' and control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup$'. Then browser value 'testid=automation-setup-name' is New Automation, browser attr 'testid=automation-setup-name' placeholder is Standup digest, browser value 'testid=automation-setup-prompt' is empty, browser attr 'testid=automation-setup-frequency-daily' aria-checked is true, browser value 'testid=automation-setup-time' is 09:00, browser text 'testid=automation-setup-save-state' is Unsaved changes and browser enabled 'testid=automation-setup-test' is false. conversation list is unchanged.

PresetsF28.schedule-presetsKnown failure

Scenario F28--presets · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automation setup frequency <preset>”.

Expect

in create mode the Frequency control (the same control as on the edit page, whose presets and saving F23 owns as edit-schedule and edit-cron) starts on Daily at 09:00, and switching presets fills create-mode defaults: Once a date-time at the next whole hour today, Weekly Monday, and Custom the cron of the preset it replaces (from Weekly, Monday at 10:30 it reads 30 10 * * 1), while a cron already typed in Custom stays.

Known bug observations

Known failure (reproduced 2026-10-15, #18461): it reads 0 9 * * *. browser count 'testid=automation-setup-timezone >> option' is 13. Click Daily again.

Known failure (reproduced 2026-10-15, #18461): Custom shows the last custom cron, 0 9 * * * until one is typed, whatever the preset; from Weekly on Monday it reads 0 9 * * *. Every preset keeps the 13-entry time zone select.

Exact CLI recipe & bug checks

For each of once, hourly, daily, weekdays, weekly, custom run control-openhands browser click 'testid=automation-setup-frequency-<preset>' and control-openhands browser testids 'testid=automation-setup-at-row' (browser snapshot is an ARIA tree and does not list test ids): Once has automation-setup-datetime, Hourly only automation-setup-timezone, Daily and Weekdays automation-setup-time, Weekly also automation-setup-weekday (value 1), Custom automation-setup-custom-schedule. Click Weekly, then Custom: expected browser value 'testid=automation-setup-custom-schedule' is Weekly's 0 9 * * 1. Known failure (reproduced 2026-10-15, #18461): it reads 0 9 * * *. browser count 'testid=automation-setup-timezone >> option' is 13. Click Daily again.

Event fieldsF28.event-triggerMapped behavior

Scenario F28--event-fields · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “^Event”.
  2. Do
    Enter “qa-hooks” in “Automation setup event source”.

Expect

choosing Event replaces the schedule with Event source (default GitHub, sent as github; its menu lists the webhook sources registered on this backend plus Custom), Event key (default issue_comment.created), Add Filter (one free-text filter input), Ask Agent, the notice Event sources need a public URL to deliver webhooks to this instance., and a Test event payload pre-filled with a JSON object for that source and key. A source the backend has not registered (menu Custom, then a typed name) adds Register this as a custom webhook source, checked by default.

Exact CLI recipe & bug checks

Run control-openhands browser click 'role=radio[name=/^Event/]'. browser value 'testid=automation-setup-event-key' is issue_comment.created, browser visible 'testid=automation-setup-event-public-url-notice' is true, and browser count 'testid=automation-setup-custom-webhook-enabled' is 0 for github. Click testid=automation-setup-event-source-toggle, browser snapshot 'testid=automation-setup-event-source-menu' lists the registered sources and Custom (GitHub, GitLab, Slack, Linear, Jira on a local stack). Click testid=automation-setup-event-source-menu >> role=button[name="Custom"], browser fill 'testid=automation-setup-event-source' qa-hooks: browser visible 'testid=automation-setup-custom-webhook-enabled' is true and browser eval "document.querySelector('[data-testid=automation-setup-custom-webhook-enabled]').checked" is true. Pick GitHub again from the same menu. Click testid=automation-setup-add-event-filter (automation-setup-event-filter appears) and testid=automation-setup-event-test-payload-toggle; browser value 'testid=automation-setup-event-test-payload' is a JSON object with "source": "github" and "event": "issue_comment.created". Click role=radio[name=/^Schedule/] to return.

Invalid drafts saveF28.draft-save · F28.draft-validationMapped behavior

Scenario F28--invalid-drafts-save · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

Save draft works on any form, valid or not; the first save replaces the URL with ?draftId=<id>, shows the Draft saved bar (relative time, View test runs, Test) and the toast Draft saved in this conversation., and the save state reads Saved just now. The draft appears in the dashboard Drafts group (F21).

the server validates on save, not on blur. An invalid draft is still saved (dispatchable: false) and its first error replaces Ready to test in the draft bar: an empty prompt shows Field required (also under Prompt), a bad custom cron shows Value error, Invalid cron expression: every day. A field error stays until the next save. Test stays enabled on a saved invalid draft: it saves again, opens Test runs with the same error in a red alert and dispatches nothing.

Exact CLI recipe & bug checks

Fill testid=automation-setup-name with QA_F28 Probe, leave the prompt empty, browser network --clear, click testid=automation-setup-save-draft. browser wait-url 'draftId='; browser toasts --history includes Draft saved in this conversation.; browser text 'testid=automation-setup-draft-validity' is Field required; control-openhands api GET /api/automation/v1/drafts/<draft-id> (the id in the URL) has validation_errors [{"field":"prompt","code":"missing",…}] and dispatchable: false (browser network --bodies shows request bodies, not the response); browser text 'testid=automation-setup-save-state' is Saved just now. Now fill the prompt with Reply pong. Do not run any tools., click testid=automation-setup-frequency-custom, fill testid=automation-setup-custom-schedule with every day, browser network --clear, press Tab, and browser network --filter 'automation/v1' lists no request (no blur validation). Click Save draft: the validity text is Value error, Invalid cron expression: every day. browser network --clear, click testid=automation-setup-test: browser text 'testid=automation-setup-draft-runs-page' contains the same cron error, browser network lists only the draft PATCH (200) and nothing under dispatch; click testid=automation-setup-draft-runs-back. Fill the cron with 0 0 1 1 * and Save draft: it reads Ready to test.

Required fields before TestF28.required-before-testMapped behavior

Scenario F28--required-fields-before-test · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “” in “Automation setup prompt”.

Expect

Test and Create check locally first and send nothing when a required field is empty: Add a prompt before continuing. (Prompt or Plugin kind), Add a plugin source before continuing., Add Python code before continuing., Add an entrypoint command before continuing., Add a setup script path before continuing., Add a setup script before continuing.; an event Test with a payload that is not a JSON object shows Enter a valid JSON object before testing this event automation. A blank Name is not an error: it is derived from the prompt.

Exact CLI recipe & bug checks

Run control-openhands browser fill 'testid=automation-setup-prompt' '', Save draft, browser network --clear, click testid=automation-setup-test: browser text 'testid=automation-setup-draft-runs-page' contains Add a prompt before continuing. and browser network --filter 'dispatch' lists nothing. Click testid=automation-setup-draft-runs-back, fill the prompt back in and Save draft.

TimeoutF28.timeout · F28.draft-save-errorMapped behavior

Scenario F28--timeout · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Click testid=automation-setup-add-timeout, fill testid=automation-setup-timeout with 1800, Save draft: Ready to test. Fill 1801, browser toasts --history --clear, browser network --clear, Save draft. browser network --filter 'automation/v1/drafts' shows PATCH … 422 (--bodies adds the request body, not the response) and browser text 'testid=automation-setup-save-state' is Save failed. browser toasts --history holds one toast, Value error, timeout must not exceed 1800 seconds (30 minutes), and browser count 'testid=automation-setup-draft-validity' is 0. Click testid=automation-setup-timeout-remove and Save draft.

Expect

Add Timeout adds an optional Timeout seconds number field with a remove button; 1800 saves and the draft reads Ready to test.

a draft the server refuses outright (a timeout above 1800 returns 422) sets the save state to Save failed and shows one error toast with the reason the API gives (a timeout of 1801 reads Value error, timeout must not exceed 1800 seconds (30 minutes)). The draft bar drops its validity pill (no stale Ready to test) until a save succeeds.

Exact CLI recipe & bug checks

Click testid=automation-setup-add-timeout, fill testid=automation-setup-timeout with 1800, Save draft: Ready to test. Fill 1801, browser toasts --history --clear, browser network --clear, Save draft. browser network --filter 'automation/v1/drafts' shows PATCH … 422 (--bodies adds the request body, not the response) and browser text 'testid=automation-setup-save-state' is Save failed. browser toasts --history holds one toast, Value error, timeout must not exceed 1800 seconds (30 minutes), and browser count 'testid=automation-setup-draft-validity' is 0. Click testid=automation-setup-timeout-remove and Save draft.

Test gatingF28.test-gatedMapped behavior

Scenario F28--test-gating · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    After a successful Save draft browser enabled 'testid=automation-setup-test' is true; fill testid=automation-setup-name with QA_F28 Probe 2 and it is false; Save draft and it is true.

Expect

in create mode Test stays disabled until the draft is saved and unchanged since; editing any field after a save disables it again until the next Save draft.

Exact CLI recipe & bug checks

After a successful Save draft browser enabled 'testid=automation-setup-test' is true; fill testid=automation-setup-name with QA_F28 Probe 2 and it is false; Save draft and it is true.

Custom kind and draft TestF28.kind-toggle · F28.draft-testKnown failure

Scenario F28--custom-kind-and-draft-test · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Wait
    Wait for “Automation setup custom code” to appear.
  2. Wait
    Wait for “Automation setup draft run” to appear.
  3. Wait
    Wait for “Automation setup draft run” / “Successful” to appear.
  4. Do
    Reload the page.

Expect

the button beside the Prompt label switches between Prompt and Custom (Custom Python). Custom shows a generated Python script that calls AUTOMATION_CALLBACK_URL, and Entrypoint and setup reveals Entrypoint python3 main.py, Setup script path setup.sh and a no-op setup script; switching back shows the Prompt again.

Test on a saved, valid draft opens Test runs (with Back), shows Test run started. and a Pending row; it runs a hidden disabled copy of the draft, not a dashboard automation. Expected: the row reaches Successful or Failed by itself.

Known bug observations

Known failure #18263 (reproduced 2026-10-15): the row stays Pending for 45 s with a single runs request in browser network --filter '/runs'; after browser reload, testid=automation-setup-view-test-runs shows it Successful. Click testid=automation-setup-draft-runs-back. Switching back with testid=automation-setup-kind-prompt restores the Prompt; leave the draft on Custom for the next bullet.

Known failure #18263 (reproduced again 2026-10-15 on a Custom Python draft): the row stays Pending and the page does not poll the runs after the first request.

Exact CLI recipe & bug checks

Click testid=automation-setup-kind-custom and browser wait 'testid=automation-setup-custom-code'; once the crossfade ends browser count 'testid=automation-setup-prompt' is 0 (right after the click it can still read 1, and visible true) and browser value 'testid=automation-setup-custom-code' contains AUTOMATION_CALLBACK_URL. Click testid=automation-setup-custom-details-toggle: Entrypoint python3 main.py, Setup script path setup.sh. Save draft (Ready to test), click testid=automation-setup-test and browser wait 'testid=automation-setup-draft-run': the row reads Pending and the page shows Test run started. Expected: browser wait 'testid=automation-setup-draft-run >> text=Successful' --timeout 45000. Known failure #18263 (reproduced 2026-10-15): the row stays Pending for 45 s with a single runs request in browser network --filter '/runs'; after browser reload, testid=automation-setup-view-test-runs shows it Successful. Click testid=automation-setup-draft-runs-back. Switching back with testid=automation-setup-kind-prompt restores the Prompt; leave the draft on Custom for the next bullet.

CreateF28.createMapped behavior

Scenario F28--create · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Automation setup create”.
  2. Do
    Open /automations.

Expect

Create Automation creates the automation enabled, navigates to /automations/<id> (a Custom Python automation shows Script), and removes the draft from the dashboard Drafts group.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=automation-setup-create' --expect-url '/automations/[0-9a-f-]{36}$'. The detail h1 is QA_F28 Probe 2, the badge reads Active and the page shows Script. browser goto /automations: browser count 'testid=automation-setup-draft-resume-<draft-id>' is 0.

Error contrastF28.error-contrastMapped behavior

Scenario F28--error-contrast · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Switch to Light+ (F16), open a draft whose cron is every day (as in the invalid-drafts bullet), and browser screenshot 'testid=automation-setup-draft-details' --feature F28.error-contrast --name light-plus. Read the pill's computed foreground and background and its parent surface with browser eval; composite its 5% red tint (#fb2c36) over the draft bar background. Expected contrast is at least 4.5:1. Click testid=automation-setup-test and inspect testid=automation-setup-draft-runs-error the same way. Repeat both checks in OpenHands-DeepSea, OpenHands-Neutral, OpenHands-Neo and Solarized Light, then restore OpenHands-Neutral.

Expect

the draft bar's error pill and the Test runs error alert use the theme's error color (text-feedback-error) and reach at least 4.5:1 in all five color themes. The pill uses a 5% red tint over the draft bar surface.

Exact CLI recipe & bug checks

Switch to Light+ (F16), open a draft whose cron is every day (as in the invalid-drafts bullet), and browser screenshot 'testid=automation-setup-draft-details' --feature F28.error-contrast --name light-plus. Read the pill's computed foreground and background and its parent surface with browser eval; composite its 5% red tint (#fb2c36) over the draft bar background. Expected contrast is at least 4.5:1. Click testid=automation-setup-test and inspect testid=automation-setup-draft-runs-error the same way. Repeat both checks in OpenHands-DeepSea, OpenHands-Neutral, OpenHands-Neo and Solarized Light, then restore OpenHands-Neutral.

CleanupF28 · setup / cleanupSupport step

Scenario F28--cleanup · Permalink · Source

Given

Use F28’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Delete QA_F28 Probe 2 from its kebab and any leftover QA_F28 draft with its trash icon, as in F21; api GET /api/automation/v1/drafts lists none.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

Delete QA_F28 Probe 2 from its kebab and any leftover QA_F28 draft with its trash icon, as in F21; api GET /api/automation/v1/drafts lists none.

Family notes, prerequisites for known issues & limits
  • A toast can cover the toolbar (Save draft, Test, Close) and stays while the pointer rests on it, so the next toolbar click times out ("subtree intercepts pointer events"). After each save, browser hover a form field (for example testid=automation-setup-name) before clicking the toolbar again, as F22's Gotchas say.
  • Drive inputs with browser fill: setting .value from browser eval does not reach React state for every field, and a save then sends the old value.
  • Change one field per Save draft when checking a validation message: a save shows only the first server error, so two edits in one save hide the second.
  • A cold load of /automations/setup?draftId=<id> or ?automationId=<id> (a reload or a typed URL) opens the form: re-checked 2026-10-14 on a control-openhands launch of main with this family (both read their Name back). An earlier instance of unknown build stayed on Loading saved draft… for more than 20 s; if that returns, compare with opening the same draft from the dashboard.
  • The draft bar duplicates the prompt error: Field required appears both under Prompt and in the bar.
  • Kebab Edit on a Custom Python automation shows the code editor instead of the saved bundle summary: open bug #18341 (F23), with fixes in PRs #18353 and #18444.
  • Escape on the setup page does nothing, by design (F23.edit-close).

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.