{ "published_snapshot": "2026-10-16", "source_repository": "https://github.com/OpenHands/OpenHands", "source_revision": "9a3c3a699e0057bd4c240c4d8158df23711c095b", "source_label": "PR #18334", "source_pr": 18334, "source_directory": ".agents/skills/verify-openhands/references/feature-map", "versions": { "agentServer": "1.55.0", "agentCanvas": "1.27.1", "automation": "1.20.0" }, "family_count": 28, "behavior_count": 778, "named_recipe_count": 844, "script_and_contract_count": 866, "families": [ { "id": "F01", "title": "First run, onboarding and sign-in", "file": "F01-first-run-and-sign-in.md", "sha256": "3cbb1784b818e66475fb0c4fed8a839d2990569c21f4b9f2e55d0bed23cbaa8d", "overview": "What a user sees before the normal app shell. A browser that has never finished\nor skipped onboarding gets a full-screen onboarding modal on every route: choose\nan agent, set up the LLM (or ACP credentials), then say hello, which starts the\nfirst conversation. In public mode the flow starts with an \"Add a backend\" step\nthat asks for the server's API key, and a browser with no stored key gets a\nfull-screen API-key screen instead of the app. A one-time telemetry consent\nmodal appears for each local backend. Unknown routes render an in-app \"Page not\nfound\" page inside the app shell, and the tab title follows the open\nconversation.", "entry": "- First run: open any URL of the app in a browser profile that has never onboarded. Every `control-openhands launch --new` starts with a fresh browser profile, and `control-openhands browser reset` gives the same run a fresh one again. Backend-side state (consent answer, LLM profile, agent kind) stays.\n- Public mode: `control-openhands launch --new --public` serves the UI without the injected key, so step 0 is \"Add a backend\". The API-key screen appears after skipping onboarding, after removing the last backend under backend selector → Manage backends (see F25), or when the server's key changes under a stored one.\n- Telemetry consent: appears by itself over onboarding or the app. It is changed later in Settings → Application (see F16).\n- Onboarding preview: direct URL with `?previewOnboardingStep=N`, for example `/?previewOnboardingStep=3` or `/settings/app?previewOnboardingStep=3`.\n- Last onboarding step: Close, a recommended-automation card, or the \"Skip Getting Started checklist\" box below the modal.\n- Not-found page: any unknown direct URL, such as `/this-route-does-not-exist`. Its **Home** button returns to `/`.\n- There is no command-menu entry or keyboard shortcut for onboarding.", "given": "Preconditions:\n\n- A fresh run you launched yourself: `control-openhands launch --new` (local) or `control-openhands launch --new --public`, then `export OH_VERIFY_RUN=` and `control-openhands doctor`. Do **not** run `control-openhands onboard --skip`: it consumes the first-run state this family tests.\n- The model-backed say-hello check needs a DeepSeek key in `DEEPSEEK_API_KEY`. It is typed into the onboarding form with `--value-env`, which replaces `llm preset deepseek` here.\n- Each fresh browser profile walks first run once; `control-openhands browser reset` starts another first run on the same stack. Local and public mode still need separate runs (`launch --new` and `launch --new --public`): the local walk, public run A (backend step walked) and public run B (onboarding skipped). Stop each one with `control-openhands stop` when done.\n- Desktop viewport unless stated.\n\nLocal run (`launch --new`):", "gotchas": "- Use a fresh run or `browser reset` for every first-run check, and never `onboard --skip` first. The browser profile lives in `/private/browser-profile`. `browser reset` clears only the browser, so the server keeps the consent answer (the consent modal does not return), the LLM profile and the agent kind.\n- `control-openhands login` passes both key prompts: the onboarding backend step, and the API-key screen, where it names the backend `Local` when Host Name is empty (after Skip and after `restart --rotate-key`, with no duplicate backend). Use it when the screen is only in the way. To check the screen itself (the disabled Connect, the wrong-key line, a chosen name), use the explicit fills above.\n- Inactive onboarding slides stay mounted and are translated off-screen. Plain `browser testids` lists only the active slide, but `count` and `wait` still find handles on the others. Assert the active step with `data-current-step`, for example `browser wait '[data-testid=onboarding-modal][data-current-step=\"2\"]'`. The `testid=` selector cannot carry attribute filters.\n- Slide indices renumber. After the public backend step succeeds, \"Choose your agent\" becomes index `0` and has no Back. `previewOnboardingStep` counts phases with the backend step included, so `0` and `1` look identical on a healthy backend.\n- The consent modal (z-70) sits over the onboarding modal, and clicks at the screen center land on agent tiles once it closes. A forced click on `first-run-onboarding-screen` selected Codex. Check backdrop non-dismissal with `browser mouse-click 40 40`, a corner outside the modal.\n- The say-hello default message (\"Create a basic webpage…\") makes the agent work for a while. Replace it with a tiny prompt to keep model cost down.\n- On a local backend, Next on the LLM step also points the `default` agent profile at the profile it just created. Activating another LLM profile does not move that pointer, and while it stays, deleting the onboarding profile answers `409` `LLM profile is referenced by 1 agent profile(s): default`. `control-openhands llm preset deepseek` moves it to `deepseek-flash` and reports `repointed`; `llm set` does not. In the UI, edit `default` in Settings → Agents and pick the profile in `testid=agent-profile-llm-selector`.\n- On a local backend the LLM step edits an LLM profile and never reads or writes the raw LLM settings (#18218). It takes the active profile, else the one the `default` agent profile points at (the Agent Server's backfill of an older install's raw settings), else a fresh one with the default model. It reads the profile only once the backend's health check reports it connected, so a step mounted while the Agent Server was down fills in after it comes back (verified 2026-10-12 with `control-openhands service stop agent-server`, then `restart`). If the profiles cannot be read it shows `Failed to load existing profile. Please try again.` (`testid=onboarding-llm-load-error`) with Retry (`testid=onboarding-llm-load-retry`) and Next disabled. Next without changes only sends `POST /api/profiles//activate`.\n- The tab title prefixes a status emoji to the stored title. A generated title that already starts with an emoji therefore shows twice (`✅ ✅ Reply hello without tools | OpenHands`). Right after launch the title is `Conversation ` until a reload picks up the generated one.\n- Right after Next, the ACP step reads \"Checking for an existing Claude Code login…\" with Next disabled. Wait for `testid=onboarding-acp-auth-detected` before reading the banner. The \"already signed in\" banner depends on the host's CLI login. On a clean CI machine the fields become required, and Next is blocked until they are filled.\n- Choosing an ACP agent saves `agent_kind: acp` immediately. Re-choose OpenHands before leaving, or later conversations in the run use the ACP agent.\n- In preview mode, Skip and Close are inert. The LLM slide shows `gpt-5.6-sol` even when another model is saved, so do not read it as data loss.\n- `browser toasts --history` lists every status or alert text (toasts and alert banners) seen since the page loaded, including success toasts and progress lines such as `Checking your backend connection…`. Read it on a page that has just loaded (or reloaded) to check that no error toast appeared. A page with a standing banner never reads `[]`: the app shell of a backend with no LLM lists the `Your LLM isn't set up yet` alert.\n- Known issue [#16601](https://github.com/OpenHands/OpenHands/issues/16601) (`F01.telemetry-consent`): Analytics consent reappears sporadically. Live prerequisite: The multi-day trigger and backend consent response when the modal returns; opted-in localStorage alone does not reproduce it.\n- Known issue [#16526](https://github.com/OpenHands/OpenHands/issues/16526) (`F01.onboarding-choose-agent`): 403 error when selecting an agent in the onboarding agent selection modal. Live prerequisite: An authenticated hosted Cloud account with the reported organization-membership mismatch during onboarding.", "features": [ { "id": "F01.first-run-gate", "description": "with no `openhands-onboarded` flag in localStorage, any URL (including `/conversations`) shows only `first-run-onboarding-screen` with the onboarding modal. There is no sidebar. Finishing or skipping sets the flag, and the app stays revealed after a reload." }, { "id": "F01.telemetry-consent", "description": "on a connected local backend whose `user_consents_to_analytics` is `null`, an elevated \"Help improve OpenHands\" modal names the backend and host. It has a checked-by-default \"Send anonymous usage data\" box and \"Confirm preferences\". Escape does not dismiss it. The choice is saved to backend settings, and the modal does not return after a reload." }, { "id": "F01.onboarding-modal", "description": "a segmented progress bar (3 steps, or 4 with the backend step) over a sliding rail. Neither Escape nor a click on the backdrop dismisses it. \"Skip for now\" shows on every step except the last, and Back/Next move between steps." }, { "id": "F01.onboarding-backend-step", "description": "when no reachable backend exists (public mode), step 0 \"Add a backend\" prefills the name `Local` and the page origin as host. Next stays disabled until an API key is typed. A wrong key shows \"Could not connect to \" / \"Invalid API key\" inline. The right key saves the backend, and the flow collapses to 3 steps on \"Choose your agent\" with no Back button. The first paint must not raise an error toast." }, { "id": "F01.onboarding-choose-agent", "description": "radio tiles for OpenHands (preselected), Claude Code, Codex and Gemini CLI, plus Pi and OpenCode on a local backend (#17758; a Cloud backend offers only the first four, and a hidden choice falls back to OpenHands). Next saves the agent kind with a \"Settings saved\" toast." }, { "id": "F01.onboarding-setup-llm", "description": "on a first run the embedded LLM form is prefilled with `openai/gpt-5.6-sol`. Typing a custom model and key then Next saves them. On a local backend Next saves the form as an LLM profile named after the model (`deepseek/deepseek-flash` → `deepseek-flash`) and activates it, without writing the raw LLM settings." }, { "id": "F01.onboarding-repeat-endpoint", "description": "onboarding again (a new browser, the same backend) fills the form from the active LLM profile, or from the one the `default` agent profile points at, model included. The All view prefills its Base URL and key even when the raw LLM settings hold another Base URL. Next without changes keeps that profile and saves no copy. Changing only the model creates a profile that keeps that Base URL and key, and the say-hello conversation reaches that endpoint." }, { "id": "F01.onboarding-acp-secrets", "description": "choosing an ACP agent swaps the LLM step for \"Add your API keys\" with that provider's fields. When the host CLI is already logged in, a green \"already signed in\" banner makes them optional. OpenCode's fields (`OPENCODE_AUTH_CONTENT`, `OPENCODE_API_KEY`) are always optional; Pi's two fields (`PI_AUTH_JSON`, `ANTHROPIC_API_KEY`) are each labelled Optional, but without a host login at least one of them is needed: Next stays disabled under a warning, readable in light themes, until either is filled. Back returns to the agent step." }, { "id": "F01.onboarding-say-hello", "description": "the last step has a prefilled message. An empty message disables send. Enter or the send button creates a conversation, navigates to `/conversations/` and completes onboarding, and the agent replies. A recommended-automations list sits under an \"Or\" separator, and a \"Skip Getting Started checklist\" box sits below the modal." }, { "id": "F01.onboarding-hello-close", "description": "the last step has Back and Close instead of \"Skip for now\". Close dismisses the modal without creating a conversation, sets the `openhands-onboarded` flag and leaves the app shell on `/`." }, { "id": "F01.onboarding-recommended-automation", "description": "picking a card under \"Start from a proven workflow\" on the last step completes onboarding too. A card with its own setup form (for example \"Daily news digest\") navigates to `/automations/new/` and opens that form. Nothing is created until the form is submitted." }, { "id": "F01.onboarding-skip-checklist", "description": "the \"Skip Getting Started checklist\" box under the last step hides the sidebar **Getting started** card (F02) once the app is revealed. It is the same per-browser setting as the Settings → Application switch, and it is saved as soon as you tick it, preview mode included." }, { "id": "F01.onboarding-skip", "description": "\"Skip for now\" closes the modal and sets the flag. In public mode, skipping before a key is stored lands on the API-key screen; skipping after the backend step lands in the app." }, { "id": "F01.onboarding-preview", "description": "`?previewOnboardingStep=0..3` on any route opens the modal (`data-preview=\"true\"`) on that phase, even after onboarding. Values outside 0-3 are ignored. Skip and Close do nothing, so leave by navigating to a URL without the parameter." }, { "id": "F01.onboarding-phone", "description": "every onboarding step fits a 390 px viewport without horizontal overflow." }, { "id": "F01.api-key-entry", "description": "in public mode with no usable key (onboarding skipped, the last backend removed, or a stored key the server now rejects), a full-screen \"Add a backend\" card appears. The host is read-only and set to the origin, and Connect stays disabled until the name and key are filled. A wrong key shows \"Invalid API key. Please check the key and try again.\" The right key opens the app shell, which survives a reload, and the consent modal then names the new backend." }, { "id": "F01.onboarding-cloud-login", "description": "the backend step's \"OpenHands Cloud\" column has \"Connect to OpenHands\" (device-flow login) and an \"Advanced\" toggle. The toggle reveals a Cloud Host field (placeholder `https://app.all-hands.dev`) for self-hosted Cloud deployments. The login itself is blocked: it needs an OpenHands Cloud account." }, { "id": "F01.route-error-boundary", "description": "an unknown URL renders a \"Page not found\" page inside the app shell, with the sidebar and a **Home** link back to `/`, and logs no page errors." }, { "id": "F01.document-title", "description": "the tab title is `OpenHands`. On a conversation it is ` | OpenHands`." }, { "id": "F01.bootstrap-loading", "description": "a centered spinner card shows while `/server_info` loads (transient, not-run)." }, { "id": "F01.locked-cloud-first-run", "description": "on a canvas locked to an OpenHands Cloud host, first run shows only the Cloud login (Add backend) without a progress bar or close button (blocked)." }, { "id": "F01.cookie-auth-redirect", "description": "OHE cookie-auth deployments probe the main-app session behind a spinner and redirect to its `/login` when it is missing (blocked)." } ], "scenarios": [ { "name": "F01--first-run-gate", "title": "First-run gate", "ids": [ "F01.first-run-gate" ], "body": "Run `control-openhands browser goto /conversations` and `control-openhands browser testids`. The list starts with `first-run-onboarding-screen`, `telemetry-consent-form` and `onboarding-modal`; `root-layout` is absent. Run `control-openhands browser screenshot --feature F01.first-run-gate --name fresh`. The screenshot shows the consent modal over \"Choose your agent\".", "raw": "**First-run gate (`F01.first-run-gate`).** Run `control-openhands browser goto /conversations` and `control-openhands browser testids`. The list starts with `first-run-onboarding-screen`, `telemetry-consent-form` and `onboarding-modal`; `root-layout` is absent. Run `control-openhands browser screenshot --feature F01.first-run-gate --name fresh`. The screenshot shows the consent modal over \"Choose your agent\".", "source_line": 61, "expected": [ "with no `openhands-onboarded` flag in localStorage, any URL (including `/conversations`) shows only `first-run-onboarding-screen` with the onboarding modal. There is no sidebar. Finishing or skipping sets the flag, and the app stays revealed after a reload." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--consent", "title": "Consent", "ids": [ "F01.telemetry-consent" ], "body": "Run `control-openhands browser snapshot 'testid=telemetry-consent-form'`. It reads `This preference is saved for the local backend “Local” at http://127.0.0.1:.` and the checkbox is `[checked]`. Run `control-openhands browser press Escape` and `control-openhands browser count 'testid=telemetry-consent-form'`; the count stays `1`. Run `control-openhands browser uncheck 'testid=telemetry-consent-form >> role=checkbox'`, `control-openhands browser click 'testid=confirm-telemetry-preferences'`, `control-openhands browser wait 'testid=telemetry-consent-form' --state detached`, then `control-openhands api GET /api/settings`. It shows `\"user_consents_to_analytics\": false`. Run `control-openhands browser reload` and `control-openhands browser count 'testid=telemetry-consent-form'`; the count is `0`.", "raw": "**Consent (`F01.telemetry-consent`).** Run `control-openhands browser snapshot 'testid=telemetry-consent-form'`. It reads `This preference is saved for the local backend “Local” at http://127.0.0.1:.` and the checkbox is `[checked]`. Run `control-openhands browser press Escape` and `control-openhands browser count 'testid=telemetry-consent-form'`; the count stays `1`. Run `control-openhands browser uncheck 'testid=telemetry-consent-form >> role=checkbox'`, `control-openhands browser click 'testid=confirm-telemetry-preferences'`, `control-openhands browser wait 'testid=telemetry-consent-form' --state detached`, then `control-openhands api GET /api/settings`. It shows `\"user_consents_to_analytics\": false`. Run `control-openhands browser reload` and `control-openhands browser count 'testid=telemetry-consent-form'`; the count is `0`.", "source_line": 62, "expected": [ "on a connected local backend whose `user_consents_to_analytics` is `null`, an elevated \"Help improve OpenHands\" modal names the backend and host. It has a checked-by-default \"Send anonymous usage data\" box and \"Confirm preferences\". Escape does not dismiss it. The choice is saved to backend settings, and the modal does not return after a reload." ], "notes": [ "Known issue [#16601](https://github.com/OpenHands/OpenHands/issues/16601) (`F01.telemetry-consent`): Analytics consent reappears sporadically. Live prerequisite: The multi-day trigger and backend consent response when the modal returns; opted-in localStorage alone does not reproduce it." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F01--modal-shell", "title": "Modal shell", "ids": [ "F01.onboarding-modal" ], "body": "Run `control-openhands browser press Escape`, `control-openhands browser count 'testid=onboarding-modal'` (`1`) and `control-openhands browser attr 'testid=onboarding-modal' data-current-step` (`0`). Run `control-openhands browser mouse-click 40 40` on the bare backdrop; the count stays `1` and the step is unchanged. `onboarding-progress-step-0..2` exist and `-3` does not, because the backend is healthy.", "raw": "**Modal shell (`F01.onboarding-modal`).** Run `control-openhands browser press Escape`, `control-openhands browser count 'testid=onboarding-modal'` (`1`) and `control-openhands browser attr 'testid=onboarding-modal' data-current-step` (`0`). Run `control-openhands browser mouse-click 40 40` on the bare backdrop; the count stays `1` and the step is unchanged. `onboarding-progress-step-0..2` exist and `-3` does not, because the backend is healthy.", "source_line": 63, "expected": [ "a segmented progress bar (3 steps, or 4 with the backend step) over a sliding rail. Neither Escape nor a click on the backdrop dismisses it. \"Skip for now\" shows on every step except the last, and Back/Next move between steps." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--choose-agent", "title": "Choose agent", "ids": [ "F01.onboarding-choose-agent" ], "body": "Run `control-openhands browser attr 'testid=onboarding-agent-option-openhands' aria-checked` (`true`) and `control-openhands browser count 'testid=onboarding-agent-back'` (`0`). Then run `control-openhands browser click 'testid=onboarding-agent-option-codex'` and `control-openhands browser attr 'testid=onboarding-agent-option-codex' aria-checked` (`true`), and switch back with `control-openhands browser click 'testid=onboarding-agent-option-openhands'`. `control-openhands browser snapshot 'testid=onboarding-step-choose-agent'` lists six radios: OpenHands, Claude Code, Codex, Gemini CLI, `Pi The open source Pi coding agent.` and `OpenCode The open source OpenCode coding agent.` (`onboarding-agent-option-pi`, `-opencode`). Run `control-openhands browser click 'testid=onboarding-agent-next'` and `control-openhands browser wait-text 'Settings saved'`; `data-current-step` becomes `1`. Run `control-openhands browser click 'testid=onboarding-llm-back'`; `data-current-step` is `0`. Click `testid=onboarding-agent-next` again.", "raw": "**Choose agent (`F01.onboarding-choose-agent`).** Run `control-openhands browser attr 'testid=onboarding-agent-option-openhands' aria-checked` (`true`) and `control-openhands browser count 'testid=onboarding-agent-back'` (`0`). Then run `control-openhands browser click 'testid=onboarding-agent-option-codex'` and `control-openhands browser attr 'testid=onboarding-agent-option-codex' aria-checked` (`true`), and switch back with `control-openhands browser click 'testid=onboarding-agent-option-openhands'`. `control-openhands browser snapshot 'testid=onboarding-step-choose-agent'` lists six radios: OpenHands, Claude Code, Codex, Gemini CLI, `Pi The open source Pi coding agent.` and `OpenCode The open source OpenCode coding agent.` (`onboarding-agent-option-pi`, `-opencode`). Run `control-openhands browser click 'testid=onboarding-agent-next'` and `control-openhands browser wait-text 'Settings saved'`; `data-current-step` becomes `1`. Run `control-openhands browser click 'testid=onboarding-llm-back'`; `data-current-step` is `0`. Click `testid=onboarding-agent-next` again.", "source_line": 64, "expected": [ "radio tiles for OpenHands (preselected), Claude Code, Codex and Gemini CLI, plus Pi and OpenCode on a local backend (#17758; a Cloud backend offers only the first four, and a hidden choice falls back to OpenHands). Next saves the agent kind with a \"Settings saved\" toast." ], "notes": [ "Known issue [#16526](https://github.com/OpenHands/OpenHands/issues/16526) (`F01.onboarding-choose-agent`): 403 error when selecting an agent in the onboarding agent selection modal. Live prerequisite: An authenticated hosted Cloud account with the reported organization-membership mismatch during onboarding." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F01--llm-step", "title": "LLM step", "ids": [ "F01.onboarding-setup-llm" ], "body": "Run `control-openhands browser snapshot 'testid=onboarding-step-setup-llm'`. It shows `Set up your LLM` and Basic tab comboboxes `OpenAI` / `gpt-5.6-sol`. Run `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-advanced-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input'` (`openai/gpt-5.6-sol`). Then run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' deepseek/deepseek-flash`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY` and `control-openhands browser click 'testid=onboarding-llm-next'`. Wait with `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"2\"]'`. Second view: `control-openhands llm show` lists profile `deepseek-flash` (`deepseek/deepseek-flash`, `api_key_set: true`) as `active_profile`.", "raw": "**LLM step (`F01.onboarding-setup-llm`).** Run `control-openhands browser snapshot 'testid=onboarding-step-setup-llm'`. It shows `Set up your LLM` and Basic tab comboboxes `OpenAI` / `gpt-5.6-sol`. Run `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-advanced-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input'` (`openai/gpt-5.6-sol`). Then run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' deepseek/deepseek-flash`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY` and `control-openhands browser click 'testid=onboarding-llm-next'`. Wait with `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"2\"]'`. Second view: `control-openhands llm show` lists profile `deepseek-flash` (`deepseek/deepseek-flash`, `api_key_set: true`) as `active_profile`.", "source_line": 65, "expected": [ "on a first run the embedded LLM form is prefilled with `openai/gpt-5.6-sol`. Typing a custom model and key then Next saves them. On a local backend Next saves the form as an LLM profile named after the model (`deepseek/deepseek-flash` → `deepseek-flash`) and activates it, without writing the raw LLM settings." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--say-hello", "title": "Say hello", "ids": [ "F01.onboarding-say-hello" ], "body": "Run `control-openhands browser count 'testid=onboarding-skip'` (`0` on the last step), `control-openhands browser fill 'testid=onboarding-hello-input' ''` and `control-openhands browser enabled 'testid=onboarding-hello-input-form >> testid=submit-button'` (`false`). Then run `control-openhands browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'` (`enabled` turns `true`), `control-openhands browser press Enter --selector 'testid=onboarding-hello-input'` and `control-openhands browser wait-url '/conversations/[0-9a-f-]+'`. Take `` from `control-openhands browser url`. Run `control-openhands conversation wait --timeout 180` and `control-openhands conversation events --kinds MessageEvent`. The agent message is `hello`, and `control-openhands browser count 'testid=onboarding-modal'` is `0`. `control-openhands browser eval \"localStorage.getItem('openhands-onboarded')\"` returns `\"1\"`. After `control-openhands browser reload` and `control-openhands browser goto /`, `testid=first-run-onboarding-screen` and `testid=onboarding-modal` both count `0`.", "raw": "**Say hello (`F01.onboarding-say-hello`).** Run `control-openhands browser count 'testid=onboarding-skip'` (`0` on the last step), `control-openhands browser fill 'testid=onboarding-hello-input' ''` and `control-openhands browser enabled 'testid=onboarding-hello-input-form >> testid=submit-button'` (`false`). Then run `control-openhands browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'` (`enabled` turns `true`), `control-openhands browser press Enter --selector 'testid=onboarding-hello-input'` and `control-openhands browser wait-url '/conversations/[0-9a-f-]+'`. Take `` from `control-openhands browser url`. Run `control-openhands conversation wait --timeout 180` and `control-openhands conversation events --kinds MessageEvent`. The agent message is `hello`, and `control-openhands browser count 'testid=onboarding-modal'` is `0`. `control-openhands browser eval \"localStorage.getItem('openhands-onboarded')\"` returns `\"1\"`. After `control-openhands browser reload` and `control-openhands browser goto /`, `testid=first-run-onboarding-screen` and `testid=onboarding-modal` both count `0`.", "source_line": 66, "expected": [ "the last step has a prefilled message. An empty message disables send. Enter or the send button creates a conversation, navigates to `/conversations/` and completes onboarding, and the agent replies. A recommended-automations list sits under an \"Or\" separator, and a \"Skip Getting Started checklist\" box sits below the modal." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--tab-title", "title": "Tab title", "ids": [ "F01.document-title" ], "body": "On `/` run `control-openhands browser url`; `title` is `OpenHands`. On `/conversations/` (after a reload, once the title is generated) the title is ` | OpenHands`, for example `✅ … | OpenHands` for a finished conversation.", "raw": "**Tab title (`F01.document-title`).** On `/` run `control-openhands browser url`; `title` is `OpenHands`. On `/conversations/` (after a reload, once the title is generated) the title is ` | OpenHands`, for example `✅ … | OpenHands` for a finished conversation.", "source_line": 67, "expected": [ "the tab title is `OpenHands`. On a conversation it is ` | OpenHands`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--preview", "title": "Preview", "ids": [ "F01.onboarding-preview" ], "body": "Run `control-openhands browser goto '/?previewOnboardingStep=3'`, then `control-openhands browser attr 'testid=onboarding-modal' data-preview` (`true`) and `control-openhands browser count 'testid=onboarding-step-say-hello'` (`1`). Step `2` shows the LLM slide. Steps `0` and `1` both show \"Choose your agent\" when the backend is healthy. `/?previewOnboardingStep=9` renders no modal. `control-openhands browser goto '/settings/app?previewOnboardingStep=3'` works too. Run `control-openhands browser goto '/?previewOnboardingStep=1'` and `control-openhands browser click 'testid=onboarding-skip'`; the modal stays (`count` `1`). `control-openhands browser goto /` shows no modal.", "raw": "**Preview (`F01.onboarding-preview`).** Run `control-openhands browser goto '/?previewOnboardingStep=3'`, then `control-openhands browser attr 'testid=onboarding-modal' data-preview` (`true`) and `control-openhands browser count 'testid=onboarding-step-say-hello'` (`1`). Step `2` shows the LLM slide. Steps `0` and `1` both show \"Choose your agent\" when the backend is healthy. `/?previewOnboardingStep=9` renders no modal. `control-openhands browser goto '/settings/app?previewOnboardingStep=3'` works too. Run `control-openhands browser goto '/?previewOnboardingStep=1'` and `control-openhands browser click 'testid=onboarding-skip'`; the modal stays (`count` `1`). `control-openhands browser goto /` shows no modal.", "source_line": 68, "expected": [ "`?previewOnboardingStep=0..3` on any route opens the modal (`data-preview=\"true\"`) on that phase, even after onboarding. Values outside 0-3 are ignored. Skip and Close do nothing, so leave by navigating to a URL without the parameter." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--skip-checklist-box", "title": "Skip checklist box", "ids": [ "F01.onboarding-skip-checklist" ], "body": "Run `control-openhands browser goto /conversations` and `control-openhands browser count 'testid=sidebar-onboarding-checklist'` (`1`). Run `control-openhands browser goto '/?previewOnboardingStep=3'`, then `control-openhands browser click 'text=Skip Getting Started checklist'`. `control-openhands browser eval \"document.querySelector('[data-testid=onboarding-skip-getting-started-checklist]').checked\"` is `true`. Run `control-openhands browser goto /conversations` and `control-openhands browser reload`; the checklist count is `0`. On `/settings/app`, `control-openhands browser eval \"document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked\"` is `false`. Restore it: go back to `/?previewOnboardingStep=3`, where the box now reads `true`, click the same text again, and the checklist count on `/conversations` is `1`.", "raw": "**Skip checklist box (`F01.onboarding-skip-checklist`).** Run `control-openhands browser goto /conversations` and `control-openhands browser count 'testid=sidebar-onboarding-checklist'` (`1`). Run `control-openhands browser goto '/?previewOnboardingStep=3'`, then `control-openhands browser click 'text=Skip Getting Started checklist'`. `control-openhands browser eval \"document.querySelector('[data-testid=onboarding-skip-getting-started-checklist]').checked\"` is `true`. Run `control-openhands browser goto /conversations` and `control-openhands browser reload`; the checklist count is `0`. On `/settings/app`, `control-openhands browser eval \"document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked\"` is `false`. Restore it: go back to `/?previewOnboardingStep=3`, where the box now reads `true`, click the same text again, and the checklist count on `/conversations` is `1`.", "source_line": 69, "expected": [ "the \"Skip Getting Started checklist\" box under the last step hides the sidebar **Getting started** card (F02) once the app is revealed. It is the same per-browser setting as the Settings → Application switch, and it is saved as soon as you tick it, preview mode included." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--phone", "title": "Phone", "ids": [ "F01.onboarding-phone" ], "body": "Run `control-openhands browser viewport phone`. Then for N in 1, 2 and 3 run `control-openhands browser goto '/?previewOnboardingStep=N'`, `control-openhands browser bbox 'testid=onboarding-modal'` and `control-openhands browser screenshot --feature F01.onboarding-phone --name step-N`. The modal is 351 px wide with `insideViewport` `true` and `pageHorizontalOverflow` `false`, and Back/Next or Back/Close are visible. Return with `control-openhands browser viewport desktop`.", "raw": "**Phone (`F01.onboarding-phone`).** Run `control-openhands browser viewport phone`. Then for N in 1, 2 and 3 run `control-openhands browser goto '/?previewOnboardingStep=N'`, `control-openhands browser bbox 'testid=onboarding-modal'` and `control-openhands browser screenshot --feature F01.onboarding-phone --name step-N`. The modal is 351 px wide with `insideViewport` `true` and `pageHorizontalOverflow` `false`, and Back/Next or Back/Close are visible. Return with `control-openhands browser viewport desktop`.", "source_line": 70, "expected": [ "every onboarding step fits a 390 px viewport without horizontal overflow." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--error-page", "title": "Error page", "ids": [ "F01.route-error-boundary" ], "body": "Run `control-openhands browser errors --clear`, `control-openhands browser goto /this-route-does-not-exist` and `control-openhands browser snapshot 'testid=not-found-screen'`. It shows `heading \"Page not found\"`, the paragraph `This address does not match any page. Check the URL, or go back to the home page.` and `link \"Home\"` (`/url: /`). `control-openhands browser count 'aside[data-collapsed]'` is `1`: the sidebar stays. `control-openhands browser screenshot --feature F01.route-error-boundary --name not-found` shows the message and the Home button centered beside the sidebar. `control-openhands browser errors --app-only` reports `pageErrors` `0` and `appErrors` `0`. Run `control-openhands browser click 'testid=not-found-home-link' --expect-url '/$'`; `control-openhands browser count 'testid=home-screen'` is `1`.", "raw": "**Error page (`F01.route-error-boundary`).** Run `control-openhands browser errors --clear`, `control-openhands browser goto /this-route-does-not-exist` and `control-openhands browser snapshot 'testid=not-found-screen'`. It shows `heading \"Page not found\"`, the paragraph `This address does not match any page. Check the URL, or go back to the home page.` and `link \"Home\"` (`/url: /`). `control-openhands browser count 'aside[data-collapsed]'` is `1`: the sidebar stays. `control-openhands browser screenshot --feature F01.route-error-boundary --name not-found` shows the message and the Home button centered beside the sidebar. `control-openhands browser errors --app-only` reports `pageErrors` `0` and `appErrors` `0`. Run `control-openhands browser click 'testid=not-found-home-link' --expect-url '/$'`; `control-openhands browser count 'testid=home-screen'` is `1`.", "source_line": 71, "expected": [ "an unknown URL renders a \"Page not found\" page inside the app shell, with the sidebar and a **Home** link back to `/`, and logs no page errors." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--onboarding-again-with-the-saved-endpoint", "title": "Onboarding again with the saved endpoint", "ids": [ "F01.onboarding-repeat-endpoint" ], "body": "Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"1\"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"2\"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Raw LLM settings that disagree with the active profile do not reach the form. To check it, arrange them before pass 2 with `control-openhands api PATCH /api/settings --write --data '{\"agent_settings_diff\":{\"llm\":{\"base_url\":\"http://127.0.0.1:9/stale-raw-settings\"}}}'`. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` still reads `https://api.deepseek.com/v1`. Fill only the model, run `browser network --clear` before Next and `browser network --filter '/api/(settings|profiles)'` after it: the rows are the profile `POST` and its `/activate`, with no `PATCH /api/settings`. `deepseek-v4-flash` keeps the DeepSeek Base URL, and the say-hello conversation finishes with `hello` (verified 2026-10-09). Known failure (reproduced 2026-10-12 and 2026-10-13, #18302): Next replaces a profile that already has the model-derived name. Arrange one before pass 2 with `control-openhands llm set --profile deepseek-v4-flash --model openai/deepseek-v4-flash --base-url http://127.0.0.1:9/other-endpoint --api-key-env DEEPSEEK_API_KEY --no-activate --no-validate`. After pass 2's Next, `browser network --filter '/api/profiles'` shows `POST /api/profiles/deepseek-v4-flash`, and `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`: its own Base URL is gone, and nothing warned. Expected: the existing profile is kept, under a name that does not collide or after a confirmation. Restore: run `control-openhands llm preset deepseek`. It also points the `default` agent profile back at `deepseek-flash`: its output has `repointed` from `deepseek-v4-flash` to `deepseek-flash`. Then `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.", "raw": "**Onboarding again with the saved endpoint (`F01.onboarding-repeat-endpoint`).** Two more first runs on this stack, both through the All view, which shows the Base URL field. Pass 1 saves a custom endpoint: `control-openhands browser reset`, `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-agent-next'`, `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"1\"]'`, `control-openhands browser click 'testid=onboarding-step-setup-llm >> testid=sdk-section-all-toggle'` and `control-openhands browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` (empty). Run `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-custom-model-input' openai/deepseek-chat`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=base-url-input' https://api.deepseek.com/v1`, `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"2\"]'`. `control-openhands llm show` lists `deepseek-chat` (`openai/deepseek-chat`, `base_url` `https://api.deepseek.com/v1`) as `active_profile`. Say hello as above (`browser fill 'testid=onboarding-hello-input' 'Reply with only the word hello. Do not run any tools.'`, `browser press Enter --selector 'testid=onboarding-hello-input'`, `browser wait-url '/conversations/[0-9a-f-]+'`), then `control-openhands conversation wait --timeout 180` is `finished` and the agent `MessageEvent` is `hello`. Pass 2 types the same endpoint again: repeat the steps up to the All view. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` is now `https://api.deepseek.com/v1`, the saved value. Fill the model `openai/deepseek-v4-flash` (a DeepSeek alias, so the new profile gets its own name), the same Base URL and the key, then Next and wait for step 2 (`control-openhands browser screenshot 'testid=onboarding-step-setup-llm' --feature F01.onboarding-repeat-endpoint --name second-pass-form` before Next). Expected: `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`, and the say-hello conversation finishes with `hello`. Raw LLM settings that disagree with the active profile do not reach the form. To check it, arrange them before pass 2 with `control-openhands api PATCH /api/settings --write --data '{\"agent_settings_diff\":{\"llm\":{\"base_url\":\"http://127.0.0.1:9/stale-raw-settings\"}}}'`. `browser value 'testid=onboarding-step-setup-llm >> testid=base-url-input'` still reads `https://api.deepseek.com/v1`. Fill only the model, run `browser network --clear` before Next and `browser network --filter '/api/(settings|profiles)'` after it: the rows are the profile `POST` and its `/activate`, with no `PATCH /api/settings`. `deepseek-v4-flash` keeps the DeepSeek Base URL, and the say-hello conversation finishes with `hello` (verified 2026-10-09). Known failure (reproduced 2026-10-12 and 2026-10-13, #18302): Next replaces a profile that already has the model-derived name. Arrange one before pass 2 with `control-openhands llm set --profile deepseek-v4-flash --model openai/deepseek-v4-flash --base-url http://127.0.0.1:9/other-endpoint --api-key-env DEEPSEEK_API_KEY --no-activate --no-validate`. After pass 2's Next, `browser network --filter '/api/profiles'` shows `POST /api/profiles/deepseek-v4-flash`, and `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`: its own Base URL is gone, and nothing warned. Expected: the existing profile is kept, under a name that does not collide or after a confirmation. Restore: run `control-openhands llm preset deepseek`. It also points the `default` agent profile back at `deepseek-flash`: its output has `repointed` from `deepseek-v4-flash` to `deepseek-flash`. Then `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps.", "source_line": 72, "expected": [ "onboarding again (a new browser, the same backend) fills the form from the active LLM profile, or from the one the `default` agent profile points at, model included. The All view prefills its Base URL and key even when the raw LLM settings hold another Base URL. Next without changes keeps that profile and saves no copy. Changing only the model creates a profile that keeps that Base URL and key, and the say-hello conversation reaches that endpoint." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12 and 2026-10-13, #18302): Next replaces a profile that already has the model-derived name. Arrange one before pass 2 with `control-openhands llm set --profile deepseek-v4-flash --model openai/deepseek-v4-flash --base-url http://127.0.0.1:9/other-endpoint --api-key-env DEEPSEEK_API_KEY --no-activate --no-validate`. After pass 2's Next, `browser network --filter '/api/profiles'` shows `POST /api/profiles/deepseek-v4-flash`, and `control-openhands llm show` lists `deepseek-v4-flash` with `base_url` `https://api.deepseek.com/v1`: its own Base URL is gone, and nothing warned. Expected: the existing profile is kept, under a name that does not collide or after a confirmation. Restore: run `control-openhands llm preset deepseek`. It also points the `default` agent profile back at `deepseek-flash`: its output has `repointed` from `deepseek-v4-flash` to `deepseek-flash`. Then `control-openhands api DELETE /api/profiles/deepseek-v4-flash --write` and `.../deepseek-chat --write` answer `200`. Delete the two say-hello conversations with Clean up's steps." ], "states": [ "failure" ] }, { "name": "F01--clean-up", "title": "Clean up", "ids": [], "body": "Delete the hello conversation from its header menu: `control-openhands browser goto /conversations/` (the hello conversation's `` from Say hello; the Error page bullet left `/`), `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'`, `control-openhands browser click 'role=button[name=\"Confirm Delete\"]'`. `control-openhands conversation list` then shows `count` `0`.\n\nPublic run A (`launch --new --public`), backend step walked:", "raw": "**Clean up.** Delete the hello conversation from its header menu: `control-openhands browser goto /conversations/` (the hello conversation's `` from Say hello; the Error page bullet left `/`), `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'`, `control-openhands browser click 'role=button[name=\"Confirm Delete\"]'`. `control-openhands conversation list` then shows `count` `0`.\n\nPublic run A (`launch --new --public`), backend step walked:", "source_line": 73, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F01--backend-step", "title": "Backend step", "ids": [ "F01.onboarding-backend-step" ], "body": "Run `control-openhands browser goto /`, `sleep 12` (gives a late toast time to appear), `control-openhands browser toasts --history`, `control-openhands browser screenshot --feature F01.onboarding-backend-step --name fresh` and `control-openhands browser snapshot 'testid=onboarding-step-check-backend'`. The toast history is `[]`, and the screenshot shows no toast. The snapshot shows `Add a backend`, name `Local`, host `http://127.0.0.1:`, Type `Local` checked, `button \"Next\" [disabled]`, and an \"OpenHands Cloud\" column with \"Connect to OpenHands\". Run `control-openhands browser fill 'testid=onboarding-backend-api-key' qa-wrong-key`, `control-openhands browser click 'testid=onboarding-backend-next'` and `control-openhands browser text 'testid=onboarding-backend-error'`. It reads `Could not connect to http://127.0.0.1:` / `Invalid API key`, and `data-current-step` stays `0`. Run `control-openhands browser fill 'testid=onboarding-backend-api-key' --value-file \"$OH_VERIFY_RUN/private/session-key\"`, `control-openhands browser click 'testid=onboarding-backend-next'` and `control-openhands browser wait 'testid=telemetry-consent-form'`. The modal is on \"Choose your agent\" with `data-current-step` `0`, `testid=onboarding-progress-step-3` and `testid=onboarding-agent-back` both count `0`, and the consent form names `“Local”`. Answer it with `control-openhands browser click 'testid=confirm-telemetry-preferences'`.", "raw": "**Backend step (`F01.onboarding-backend-step`).** Run `control-openhands browser goto /`, `sleep 12` (gives a late toast time to appear), `control-openhands browser toasts --history`, `control-openhands browser screenshot --feature F01.onboarding-backend-step --name fresh` and `control-openhands browser snapshot 'testid=onboarding-step-check-backend'`. The toast history is `[]`, and the screenshot shows no toast. The snapshot shows `Add a backend`, name `Local`, host `http://127.0.0.1:`, Type `Local` checked, `button \"Next\" [disabled]`, and an \"OpenHands Cloud\" column with \"Connect to OpenHands\". Run `control-openhands browser fill 'testid=onboarding-backend-api-key' qa-wrong-key`, `control-openhands browser click 'testid=onboarding-backend-next'` and `control-openhands browser text 'testid=onboarding-backend-error'`. It reads `Could not connect to http://127.0.0.1:` / `Invalid API key`, and `data-current-step` stays `0`. Run `control-openhands browser fill 'testid=onboarding-backend-api-key' --value-file \"$OH_VERIFY_RUN/private/session-key\"`, `control-openhands browser click 'testid=onboarding-backend-next'` and `control-openhands browser wait 'testid=telemetry-consent-form'`. The modal is on \"Choose your agent\" with `data-current-step` `0`, `testid=onboarding-progress-step-3` and `testid=onboarding-agent-back` both count `0`, and the consent form names `“Local”`. Answer it with `control-openhands browser click 'testid=confirm-telemetry-preferences'`.", "source_line": 77, "expected": [ "when no reachable backend exists (public mode), step 0 \"Add a backend\" prefills the name `Local` and the page origin as host. Next stays disabled until an API key is typed. A wrong key shows \"Could not connect to \" / \"Invalid API key\" inline. The right key saves the backend, and the flow collapses to 3 steps on \"Choose your agent\" with no Back button. The first paint must not raise an error toast." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--acp-credentials", "title": "ACP credentials", "ids": [ "F01.onboarding-acp-secrets" ], "body": "Run `control-openhands browser click 'testid=onboarding-agent-option-claude-code'`, `control-openhands browser click 'testid=onboarding-agent-next'` and `control-openhands browser wait-text 'Settings saved'`. Then run `control-openhands browser snapshot 'testid=onboarding-step-setup-acp-secrets'`. It shows `Add your API keys` with `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL`, and `control-openhands api GET /api/settings` shows `\"agent_kind\": \"acp\"`. Run `control-openhands browser click 'testid=onboarding-acp-secrets-back'`, `control-openhands browser click 'testid=onboarding-agent-option-openhands'` and `control-openhands browser click 'testid=onboarding-agent-next'`. `testid=onboarding-step-setup-llm` is shown and `agent_kind` is back to `openhands`.", "raw": "**ACP credentials (`F01.onboarding-acp-secrets`).** Run `control-openhands browser click 'testid=onboarding-agent-option-claude-code'`, `control-openhands browser click 'testid=onboarding-agent-next'` and `control-openhands browser wait-text 'Settings saved'`. Then run `control-openhands browser snapshot 'testid=onboarding-step-setup-acp-secrets'`. It shows `Add your API keys` with `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL`, and `control-openhands api GET /api/settings` shows `\"agent_kind\": \"acp\"`. Run `control-openhands browser click 'testid=onboarding-acp-secrets-back'`, `control-openhands browser click 'testid=onboarding-agent-option-openhands'` and `control-openhands browser click 'testid=onboarding-agent-next'`. `testid=onboarding-step-setup-llm` is shown and `agent_kind` is back to `openhands`.", "source_line": 78, "expected": [ "choosing an ACP agent swaps the LLM step for \"Add your API keys\" with that provider's fields. When the host CLI is already logged in, a green \"already signed in\" banner makes them optional. OpenCode's fields (`OPENCODE_AUTH_CONTENT`, `OPENCODE_API_KEY`) are always optional; Pi's two fields (`PI_AUTH_JSON`, `ANTHROPIC_API_KEY`) are each labelled Optional, but without a host login at least one of them is needed: Next stays disabled under a warning, readable in light themes, until either is filled. Back returns to the agent step." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--pi-and-opencode-credentials", "title": "Pi and OpenCode credentials", "ids": [ "F01.onboarding-acp-secrets" ], "body": "The previous bullet ends on the LLM step: run `control-openhands browser click 'testid=onboarding-llm-back'` to return to the agent step, then `control-openhands browser click 'testid=onboarding-agent-option-opencode'`, `control-openhands browser click 'testid=onboarding-agent-next'`, `browser wait-text 'Settings saved'` and `browser snapshot 'testid=onboarding-step-setup-acp-secrets'`: `OPENCODE_AUTH_CONTENT Optional` and `OPENCODE_API_KEY Optional`, and `control-openhands browser enabled 'testid=onboarding-acp-secrets-next'` is `true`. Back, then the same with `onboarding-agent-option-pi`: `This backend has no host login, so credentials are required to run Pi.`, `PI_AUTH_JSON Optional` and `ANTHROPIC_API_KEY Optional` (either one is enough), Next `false` while both are empty, and `browser text 'testid=onboarding-acp-secrets-blocked'` reads `Add the credentials above to continue, or sign in on the backend.` Its colour, `browser eval \"getComputedStyle(document.querySelector('[data-testid=onboarding-acp-secrets-blocked]')).color\"`, is `rgb(201, 185, 116)` in OpenHands-Neutral and `rgb(125, 93, 8)` in Light+ (arrange with `browser eval \"localStorage.setItem('openhands-color-theme','light-plus')\"` and `browser reload`, which returns to the agent step with OpenHands selected again, so click `onboarding-agent-option-pi` and `onboarding-agent-next` once more before reading the colour; #18333). Restore the theme with `localStorage.removeItem`, go Back, choose OpenHands and Next (`agent_kind` back to `openhands`).", "raw": "**Pi and OpenCode credentials (`F01.onboarding-acp-secrets`).** The previous bullet ends on the LLM step: run `control-openhands browser click 'testid=onboarding-llm-back'` to return to the agent step, then `control-openhands browser click 'testid=onboarding-agent-option-opencode'`, `control-openhands browser click 'testid=onboarding-agent-next'`, `browser wait-text 'Settings saved'` and `browser snapshot 'testid=onboarding-step-setup-acp-secrets'`: `OPENCODE_AUTH_CONTENT Optional` and `OPENCODE_API_KEY Optional`, and `control-openhands browser enabled 'testid=onboarding-acp-secrets-next'` is `true`. Back, then the same with `onboarding-agent-option-pi`: `This backend has no host login, so credentials are required to run Pi.`, `PI_AUTH_JSON Optional` and `ANTHROPIC_API_KEY Optional` (either one is enough), Next `false` while both are empty, and `browser text 'testid=onboarding-acp-secrets-blocked'` reads `Add the credentials above to continue, or sign in on the backend.` Its colour, `browser eval \"getComputedStyle(document.querySelector('[data-testid=onboarding-acp-secrets-blocked]')).color\"`, is `rgb(201, 185, 116)` in OpenHands-Neutral and `rgb(125, 93, 8)` in Light+ (arrange with `browser eval \"localStorage.setItem('openhands-color-theme','light-plus')\"` and `browser reload`, which returns to the agent step with OpenHands selected again, so click `onboarding-agent-option-pi` and `onboarding-agent-next` once more before reading the colour; #18333). Restore the theme with `localStorage.removeItem`, go Back, choose OpenHands and Next (`agent_kind` back to `openhands`).", "source_line": 79, "expected": [ "choosing an ACP agent swaps the LLM step for \"Add your API keys\" with that provider's fields. When the host CLI is already logged in, a green \"already signed in\" banner makes them optional. OpenCode's fields (`OPENCODE_AUTH_CONTENT`, `OPENCODE_API_KEY`) are always optional; Pi's two fields (`PI_AUTH_JSON`, `ANTHROPIC_API_KEY`) are each labelled Optional, but without a host login at least one of them is needed: Next stays disabled under a warning, readable in light themes, until either is filled. Back returns to the agent step." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--skip-into-the-app", "title": "Skip into the app", "ids": [ "F01.onboarding-skip" ], "body": "Run `control-openhands browser click 'testid=onboarding-skip'`, `control-openhands browser wait 'testid=root-layout'`, `control-openhands browser reload`, `control-openhands browser count 'testid=onboarding-modal'` (`0`) and `control-openhands browser count 'testid=api-key-entry-screen'` (`0`).", "raw": "**Skip into the app (`F01.onboarding-skip`).** Run `control-openhands browser click 'testid=onboarding-skip'`, `control-openhands browser wait 'testid=root-layout'`, `control-openhands browser reload`, `control-openhands browser count 'testid=onboarding-modal'` (`0`) and `control-openhands browser count 'testid=api-key-entry-screen'` (`0`).", "source_line": 80, "expected": [ "\"Skip for now\" closes the modal and sets the flag. In public mode, skipping before a key is stored lands on the API-key screen; skipping after the backend step lands in the app." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--last-backend-removed", "title": "Last backend removed", "ids": [ "F01.api-key-entry" ], "body": "Run `control-openhands browser click 'testid=backend-selector'`, `control-openhands browser click 'testid=manage-backends-menu-item'`, `control-openhands browser click 'testid=manage-backends-remove-Local'` and `control-openhands browser click 'testid=confirmation-modal >> testid=confirm-button'`. `control-openhands browser wait 'testid=api-key-entry-screen'` succeeds. Restore with the Connect recipe below.\n\nPublic run B (`launch --new --public`), onboarding skipped:", "raw": "**Last backend removed (`F01.api-key-entry`).** Run `control-openhands browser click 'testid=backend-selector'`, `control-openhands browser click 'testid=manage-backends-menu-item'`, `control-openhands browser click 'testid=manage-backends-remove-Local'` and `control-openhands browser click 'testid=confirmation-modal >> testid=confirm-button'`. `control-openhands browser wait 'testid=api-key-entry-screen'` succeeds. Restore with the Connect recipe below.\n\nPublic run B (`launch --new --public`), onboarding skipped:", "source_line": 81, "expected": [ "in public mode with no usable key (onboarding skipped, the last backend removed, or a stored key the server now rejects), a full-screen \"Add a backend\" card appears. The host is read-only and set to the origin, and Connect stays disabled until the name and key are filled. A wrong key shows \"Invalid API key. Please check the key and try again.\" The right key opens the app shell, which survives a reload, and the consent modal then names the new backend." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--skip-to-api-key-screen", "title": "Skip to API-key screen", "ids": [ "F01.onboarding-skip", "F01.api-key-entry" ], "body": "Run `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-skip'` and `control-openhands browser wait 'testid=api-key-entry-screen'`. After `sleep 12`, `control-openhands browser toasts --history` is `[]`: the screen raises no error toast. Run `control-openhands browser reload`, `control-openhands browser wait 'testid=api-key-entry-screen'`, `sleep 12` and `control-openhands browser toasts --history` again: still `[]` when the API-key screen is the first paint (the toast these checks guard against reads `No backend is configured.`; `control-openhands browser count 'text=No backend is configured'` is `0`). Run `control-openhands browser value 'testid=api-key-entry-host'` (the origin; the field is disabled) and `control-openhands browser enabled 'testid=api-key-entry-submit'` (`false`). Fill only the key with `control-openhands browser fill 'testid=api-key-entry-api-key' qa-wrong-key`; `enabled` is still `false` because Host Name is required. Run `control-openhands browser fill 'testid=api-key-entry-name' 'QA Public'` (`enabled` turns `true`), `control-openhands browser click 'testid=api-key-entry-submit'` and `control-openhands browser text 'testid=api-key-entry-status'`. It reads `Invalid API key. Please check the key and try again.`, the screen stays, and `control-openhands browser screenshot --feature F01.api-key-entry --name wrong-key` shows the red line above Connect. While the screen is still up, run `control-openhands browser viewport phone` and `control-openhands browser bbox 'testid=api-key-entry-form'`. It gives `insideViewport` `true` and `pageHorizontalOverflow` `false`. Return with `control-openhands browser viewport desktop`.", "raw": "**Skip to API-key screen (`F01.onboarding-skip`, `F01.api-key-entry`).** Run `control-openhands browser goto /`, `control-openhands browser click 'testid=onboarding-skip'` and `control-openhands browser wait 'testid=api-key-entry-screen'`. After `sleep 12`, `control-openhands browser toasts --history` is `[]`: the screen raises no error toast. Run `control-openhands browser reload`, `control-openhands browser wait 'testid=api-key-entry-screen'`, `sleep 12` and `control-openhands browser toasts --history` again: still `[]` when the API-key screen is the first paint (the toast these checks guard against reads `No backend is configured.`; `control-openhands browser count 'text=No backend is configured'` is `0`). Run `control-openhands browser value 'testid=api-key-entry-host'` (the origin; the field is disabled) and `control-openhands browser enabled 'testid=api-key-entry-submit'` (`false`). Fill only the key with `control-openhands browser fill 'testid=api-key-entry-api-key' qa-wrong-key`; `enabled` is still `false` because Host Name is required. Run `control-openhands browser fill 'testid=api-key-entry-name' 'QA Public'` (`enabled` turns `true`), `control-openhands browser click 'testid=api-key-entry-submit'` and `control-openhands browser text 'testid=api-key-entry-status'`. It reads `Invalid API key. Please check the key and try again.`, the screen stays, and `control-openhands browser screenshot --feature F01.api-key-entry --name wrong-key` shows the red line above Connect. While the screen is still up, run `control-openhands browser viewport phone` and `control-openhands browser bbox 'testid=api-key-entry-form'`. It gives `insideViewport` `true` and `pageHorizontalOverflow` `false`. Return with `control-openhands browser viewport desktop`.", "source_line": 85, "expected": [ "\"Skip for now\" closes the modal and sets the flag. In public mode, skipping before a key is stored lands on the API-key screen; skipping after the backend step lands in the app.", "in public mode with no usable key (onboarding skipped, the last backend removed, or a stored key the server now rejects), a full-screen \"Add a backend\" card appears. The host is read-only and set to the origin, and Connect stays disabled until the name and key are filled. A wrong key shows \"Invalid API key. Please check the key and try again.\" The right key opens the app shell, which survives a reload, and the consent modal then names the new backend." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--connect", "title": "Connect", "ids": [ "F01.api-key-entry" ], "body": "Run `control-openhands browser fill 'testid=api-key-entry-name' 'QA Public'`, `control-openhands browser fill 'testid=api-key-entry-api-key' --value-file \"$OH_VERIFY_RUN/private/session-key\"`, `control-openhands browser click 'testid=api-key-entry-submit'` and `control-openhands browser wait 'testid=api-key-entry-screen' --state detached`. `testid=root-layout` is shown, the consent form names `“QA Public”`, and `control-openhands browser snapshot 'testid=backend-selector'` shows `combobox \"QA Public\"`. After `control-openhands browser reload`, `testid=api-key-entry-screen` counts `0` and `testid=root-layout` counts `1`.", "raw": "**Connect (`F01.api-key-entry`).** Run `control-openhands browser fill 'testid=api-key-entry-name' 'QA Public'`, `control-openhands browser fill 'testid=api-key-entry-api-key' --value-file \"$OH_VERIFY_RUN/private/session-key\"`, `control-openhands browser click 'testid=api-key-entry-submit'` and `control-openhands browser wait 'testid=api-key-entry-screen' --state detached`. `testid=root-layout` is shown, the consent form names `“QA Public”`, and `control-openhands browser snapshot 'testid=backend-selector'` shows `combobox \"QA Public\"`. After `control-openhands browser reload`, `testid=api-key-entry-screen` counts `0` and `testid=root-layout` counts `1`.", "source_line": 86, "expected": [ "in public mode with no usable key (onboarding skipped, the last backend removed, or a stored key the server now rejects), a full-screen \"Add a backend\" card appears. The host is read-only and set to the origin, and Connect stays disabled until the name and key are filled. A wrong key shows \"Invalid API key. Please check the key and try again.\" The right key opens the app shell, which survives a reload, and the consent modal then names the new backend." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--stale-key-re-prompt", "title": "Stale key re-prompt", "ids": [ "F01.api-key-entry" ], "body": "Answer consent if it is open (`control-openhands browser click 'testid=confirm-telemetry-preferences'`). Run `control-openhands restart --rotate-key` (`rotatedKey` `true`), then `control-openhands browser reload` and `control-openhands browser wait 'testid=api-key-entry-screen'`; `testid=root-layout` counts `0`. Connect again as above with the new `$OH_VERIFY_RUN/private/session-key`. After a reload, `testid=root-layout` counts `1`, and `control-openhands browser eval \"JSON.parse(localStorage.getItem('openhands-backends')).length\"` is `1`, so no duplicate backend is added.", "raw": "**Stale key re-prompt (`F01.api-key-entry`).** Answer consent if it is open (`control-openhands browser click 'testid=confirm-telemetry-preferences'`). Run `control-openhands restart --rotate-key` (`rotatedKey` `true`), then `control-openhands browser reload` and `control-openhands browser wait 'testid=api-key-entry-screen'`; `testid=root-layout` counts `0`. Connect again as above with the new `$OH_VERIFY_RUN/private/session-key`. After a reload, `testid=root-layout` counts `1`, and `control-openhands browser eval \"JSON.parse(localStorage.getItem('openhands-backends')).length\"` is `1`, so no duplicate backend is added.", "source_line": 87, "expected": [ "in public mode with no usable key (onboarding skipped, the last backend removed, or a stored key the server now rejects), a full-screen \"Add a backend\" card appears. The host is read-only and set to the origin, and Connect stays disabled until the name and key are filled. A wrong key shows \"Invalid API key. Please check the key and try again.\" The right key opens the app shell, which survives a reload, and the consent modal then names the new backend." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--close-on-the-last-step", "title": "Close on the last step", "ids": [ "F01.onboarding-hello-close" ], "body": "Run `control-openhands browser reset` and `control-openhands browser goto /`. Walk to the last step: `control-openhands browser fill 'testid=onboarding-backend-api-key' --value-file \"$OH_VERIFY_RUN/private/session-key\"`, `control-openhands browser click 'testid=onboarding-backend-next'`, `control-openhands browser wait 'testid=onboarding-step-choose-agent'` (consent does not return: the backend already has an answer), `control-openhands browser click 'testid=onboarding-agent-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"1\"]'`. Then fill the key with `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, open `testid=sdk-section-advanced-toggle` and fill `llm-custom-model-input` with `deepseek/deepseek-flash` as in the LLM step. Run `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"2\"]'`. Check `control-openhands conversation list` (`count` `0`), then run `control-openhands browser click 'testid=onboarding-hello-close'` and `control-openhands browser wait 'testid=onboarding-modal' --state detached`. `testid=root-layout` counts `1`, the URL is `/` and `localStorage.getItem('openhands-onboarded')` is `\"1\"`. `conversation list` still shows `count` `0`, and after `browser reload` the modal count is `0`.", "raw": "**Close on the last step (`F01.onboarding-hello-close`).** Run `control-openhands browser reset` and `control-openhands browser goto /`. Walk to the last step: `control-openhands browser fill 'testid=onboarding-backend-api-key' --value-file \"$OH_VERIFY_RUN/private/session-key\"`, `control-openhands browser click 'testid=onboarding-backend-next'`, `control-openhands browser wait 'testid=onboarding-step-choose-agent'` (consent does not return: the backend already has an answer), `control-openhands browser click 'testid=onboarding-agent-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"1\"]'`. Then fill the key with `control-openhands browser fill 'testid=onboarding-step-setup-llm >> testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY`, open `testid=sdk-section-advanced-toggle` and fill `llm-custom-model-input` with `deepseek/deepseek-flash` as in the LLM step. Run `control-openhands browser click 'testid=onboarding-llm-next'` and `control-openhands browser wait '[data-testid=onboarding-modal][data-current-step=\"2\"]'`. Check `control-openhands conversation list` (`count` `0`), then run `control-openhands browser click 'testid=onboarding-hello-close'` and `control-openhands browser wait 'testid=onboarding-modal' --state detached`. `testid=root-layout` counts `1`, the URL is `/` and `localStorage.getItem('openhands-onboarded')` is `\"1\"`. `conversation list` still shows `count` `0`, and after `browser reload` the modal count is `0`.", "source_line": 88, "expected": [ "the last step has Back and Close instead of \"Skip for now\". Close dismisses the modal without creating a conversation, sets the `openhands-onboarded` flag and leaves the app shell on `/`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--recommended-automation", "title": "Recommended automation", "ids": [ "F01.onboarding-recommended-automation" ], "body": "Run `control-openhands browser reset`, `control-openhands browser goto /`, and walk to the last step as above (the LLM step's Next works without retyping the key once the profile is saved). Run `control-openhands browser click 'testid=recommended-automation-card-news-digest' --expect-url '/automations/new/news-digest(\\?|$)'`. `testid=onboarding-modal` counts `0`, `openhands-onboarded` is `\"1\"`, and the \"Daily news digest\" setup form (`testid=setup-dialog`) is open. Leave without creating anything: `control-openhands browser click 'testid=setup-dialog-close'` and `control-openhands browser wait 'testid=setup-dialog' --state detached` (the URL becomes `/`). `control-openhands api GET /api/automation/v1` shows `\"total\": 0`.", "raw": "**Recommended automation (`F01.onboarding-recommended-automation`).** Run `control-openhands browser reset`, `control-openhands browser goto /`, and walk to the last step as above (the LLM step's Next works without retyping the key once the profile is saved). Run `control-openhands browser click 'testid=recommended-automation-card-news-digest' --expect-url '/automations/new/news-digest(\\?|$)'`. `testid=onboarding-modal` counts `0`, `openhands-onboarded` is `\"1\"`, and the \"Daily news digest\" setup form (`testid=setup-dialog`) is open. Leave without creating anything: `control-openhands browser click 'testid=setup-dialog-close'` and `control-openhands browser wait 'testid=setup-dialog' --state detached` (the URL becomes `/`). `control-openhands api GET /api/automation/v1` shows `\"total\": 0`.", "source_line": 89, "expected": [ "picking a card under \"Start from a proven workflow\" on the last step completes onboarding too. A card with its own setup form (for example \"Daily news digest\") navigates to `/automations/new/` and opens that form. Nothing is created until the form is submitted." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F01--cloud-login-column", "title": "Cloud login column", "ids": [ "F01.onboarding-cloud-login" ], "body": "Run `control-openhands browser reset`, `control-openhands browser goto /` and `control-openhands browser click 'testid=onboarding-backend-advanced-toggle'`. `browser attr 'testid=onboarding-backend-advanced-toggle' aria-expanded` is `true`, and `browser attr 'testid=onboarding-backend-cloud-host' placeholder` is `https://app.all-hands.dev`. `browser text 'testid=onboarding-backend-advanced-panel'` mentions self-hosted Cloud deployments. Do not click `testid=onboarding-backend-login-button`: the device-flow login is blocked without an OpenHands Cloud account.\n\nNot reachable locally:", "raw": "**Cloud login column (`F01.onboarding-cloud-login`).** Run `control-openhands browser reset`, `control-openhands browser goto /` and `control-openhands browser click 'testid=onboarding-backend-advanced-toggle'`. `browser attr 'testid=onboarding-backend-advanced-toggle' aria-expanded` is `true`, and `browser attr 'testid=onboarding-backend-cloud-host' placeholder` is `https://app.all-hands.dev`. `browser text 'testid=onboarding-backend-advanced-panel'` mentions self-hosted Cloud deployments. Do not click `testid=onboarding-backend-login-button`: the device-flow login is blocked without an OpenHands Cloud account.\n\nNot reachable locally:", "source_line": 90, "expected": [ "the backend step's \"OpenHands Cloud\" column has \"Connect to OpenHands\" (device-flow login) and an \"Advanced\" toggle. The toggle reveals a Cloud Host field (placeholder `https://app.all-hands.dev`) for self-hosted Cloud deployments. The login itself is blocked: it needs an OpenHands Cloud account." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F01--locked-cloud-first-run", "title": "Locked Cloud first run", "ids": [ "F01.locked-cloud-first-run" ], "body": "Blocked: needs a build locked to a Cloud host (`VITE_LOCK_TO_CLOUD` or `static-server --lock-to-cloud`) and an OpenHands Cloud account. `launch` has no such flag.", "raw": "**Locked Cloud first run (`F01.locked-cloud-first-run`).** Blocked: needs a build locked to a Cloud host (`VITE_LOCK_TO_CLOUD` or `static-server --lock-to-cloud`) and an OpenHands Cloud account. `launch` has no such flag.", "source_line": 94, "expected": [ "on a canvas locked to an OpenHands Cloud host, first run shows only the Cloud login (Add backend) without a progress bar or close button (blocked)." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F01--cookie-auth-redirect", "title": "Cookie-auth redirect", "ids": [ "F01.cookie-auth-redirect" ], "body": "Blocked: needs an OHE cookie-auth deployment.", "raw": "**Cookie-auth redirect (`F01.cookie-auth-redirect`).** Blocked: needs an OHE cookie-auth deployment.", "source_line": 95, "expected": [ "OHE cookie-auth deployments probe the main-app session behind a spinner and redirect to its `/login` when it is missing (blocked)." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F01--bootstrap-spinner", "title": "Bootstrap spinner", "ids": [ "F01.bootstrap-loading" ], "body": "Blocked on the 2026-10-09 run: the agent attempted a fresh first paint, but the spinner lasted under a second on the healthy stack; slowing `/server_info` would require request interception rather than the real service response.", "raw": "**Bootstrap spinner (`F01.bootstrap-loading`).** Blocked on the 2026-10-09 run: the agent attempted a fresh first paint, but the spinner lasted under a second on the healthy stack; slowing `/server_info` would require request interception rather than the real service response.", "source_line": 96, "expected": [ "a centered spinner card shows while `/server_info` loads (transient, not-run)." ], "notes": [], "annotations": [], "states": [ "not-run" ] } ] }, { "id": "F02", "title": "App shell, sidebar and command menu", "file": "F02-app-shell.md", "sha256": "6efc6bd0b46b22dbc1d083e457212d45b851a49b766057cffed107b39b1081b4", "overview": "Every page inside the root layout shares one shell. On desktop it is a left rail\nwith the logo, **Search commands**, **New Chat**, **Customize**, **Automate**\n(when the automation interface is present), the conversation list, the\n**Getting started** checklist, an update tile (only when a newer release exists)\nand the backend selector with a settings gear. The rail collapses to icons and\nremembers that choice. Below 768 px the rail turns into a drawer that opens from\na top bar or the chat header. A user can pin Customize or Automate as the page\n`/` opens. A Ctrl/Cmd+K command menu searches pages, settings and actions.\nFailed requests show error toasts.", "entry": "- Any page in the app: the left rail (desktop) or the hamburger (phone, `browser viewport phone`).\n- Direct URLs: `/` (the default home, or a redirect to the pinned page), `/conversations`, `/customize`, `/automations`, `/settings`.\n- Command menu: press `Control+k` or `Meta+k` anywhere, or click **Search commands** (`command-menu-trigger`) in the rail or drawer. Type a page or setting name, then press Enter or click the row.\n- Hover a sidebar row to reveal its pin. Hover the collapsed rail's logo to reveal the expand chevron.\n- Settings → Application (`/settings/app`): the **Show Getting Started checklist** switch.\n- The **Getting started** card: click an item row to go to its page, or hover it for the preview; hover the card for the X that dismisses it.\n- Keyboard: `Control+k` also works while the composer has focus; Tab moves from the search input into the option list.", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done), desktop viewport, sidebar expanded, no home pin, checklist shown.\n- `F02.checklist-progress` (LLM and chat items) needs `control-openhands llm preset deepseek` and one conversation (`control-openhands conversation start --prompt \"Reply with only the word: ok\" --wait --timeout 180`). Run both before the `F02.conversation-list` bullet. The preset writes through the API, so `control-openhands browser reload` before you read the LLM item. The conversation-header hamburger check in `F02.mobile-drawer` reuses that conversation.\n- `F02.update-tile` (the sidebar tile) is blocked unless the installed version is older than npm `latest` and the browser can reach `registry.npmjs.org`. The Settings → Application check runs either way.\n- `F02.alert-banner` is blocked on the local backend, whose config always reports no alerts; it needs a Cloud backend that publishes them.\n- `F02.settings-404-modal` is blocked: it needs a backend whose settings endpoint returns 404.\n- `F02.checklist-all-complete` needs an automation and an MCP server on top of the four items above. Both are added through the UI from two fixtures (see that bullet) and deleted again afterwards.", "gotchas": "- The rail animates its width for 200 ms. A `bbox` taken right after the collapse click reads an intermediate width (about 70); measure after `browser reload`.\n- `browser url` straight after clicking **Customize** can still read `/customize`; desktop redirects to `/mcp` a moment later. Use `browser wait-url '/mcp$'`.\n- `/settings` redirects to `/settings/agents` on desktop but shows the settings menu on phone. That redirect also marks the checklist's **Customize your agent** item done, so visiting Settings by any route completes it. The sidebar **Customize** link (`/customize`) does not.\n- The pin toggles are hidden until hover (`visible` is `false`), except a pinned row's toggle, which stays visible. They exist only on the expanded rail.\n- The two desktop elements share the label `Sidebar navigation`, and settings pages add a second `complementary` landmark, so `role=complementary` is ambiguous. Use `aside[data-collapsed]` for the desktop rail and `testid=sidebar-mobile-drawer` for the drawer.\n- HeroUI tooltips (the collapsed rail and checklist previews) often stay closed on the first `browser hover` after a navigation. Hover a neighbor first, then the target, and `browser wait` for the tooltip. Never `count` it right away.\n- When collapsed, the expand chevron is `opacity-0 pointer-events-none` until the rail is hovered. A forced click on it lands on the logo link and does nothing; hover the logo first.\n- With the full list open, the command-menu panel covers the viewport center, so clicking the backdrop button by selector only works after a filter shortens the list (type `secr` first). `control-openhands browser mouse-click 1300 500` hits the desktop backdrop with any list length.\n- The new-tab path works with `browser click ... --button middle` and with `--modifiers Control` (or `Meta`). The phone drawer's backdrop has no handle; `browser mouse-click 350 400` clicks it.\n- Escape closes one layer at a time (#18044): with the command menu open over the phone drawer, the first Escape closes the menu and the second closes the drawer. Focus goes back to whatever had it when the menu opened: right after `browser goto /conversations` that is the composer (`chat-input` takes focus on load), and on `/settings/app` it is ``.\n- **Toggle sidebar** at phone width flips the stored desktop state without any visible change. Expand the rail again after returning to desktop.\n- Typing does not reset the command menu's selection. The active option keeps its index, cut down to the shorter list, from the last ArrowUp/ArrowDown or hover, and a pointer resting where the list opens hovers the option under it. A term with several matches can then run a later one on Enter: after two `ArrowDown` presses, typing `LLM profiles` (LLM profiles and Model Router) selects Model Router. Before opening the menu for an Enter check, rest the pointer with `browser hover 'testid=command-menu-trigger'`, and read `aria-activedescendant` before Enter when a term matches several items. Repro candidate, not yet filed.\n- The command menu's **LLM profiles** item uses a magnifier icon (`lucide-search`, the same icon as the menu's search field).\n- Dismissing the checklist with its X and the App settings switch write the same browser-local flag, so either one undoes the other. Clicking **Join the OpenHands Slack** counts as joining and opens an external tab you must close.\n- Registry access depends on the sandbox. Where the browser cannot reach `registry.npmjs.org` (`ERR_CERT_AUTHORITY_INVALID`), the update check reports `Couldn't check for updates`; that is the environment, not the product. Where it can, the badge reads `Up to date` and the check says `You're running the latest version.`\n- Adding a custom MCP server tests the connection first and refuses an unreachable one (`Could not reach the server...`, or `Connection failed: McpError: Connection closed` for `uvx mcp-server-time` here), so the all-complete recipe uses the local stdio fixture.", "features": [ { "id": "F02.nav-links", "description": "the logo and **New Chat** open `/conversations`. **Customize** opens `/customize`, which redirects to `/mcp` on desktop, and stays highlighted and marked `aria-current=\"page\"` on `/skills`, `/plugins`, `/apps` and `/mcp`. **Automate** opens `/automations`. Canvas extension pages add their own rows (`sidebar-canvas-extension--`)." }, { "id": "F02.pin-as-home", "description": "hovering Customize or Automate reveals a pin. Pinning makes `/` redirect to that page, pinning the other one moves the pin, and unpinning restores the default home. The pin is stored per backend and org." }, { "id": "F02.sidebar-collapse", "description": "the chevron collapses the rail to a 60 px icon rail with hover tooltips, a Settings icon and a backend icon with a status dot that opens a backend popover. The checklist is hidden while collapsed. The state survives a reload. To expand, click the chevron that appears when you hover the logo, or click empty rail space." }, { "id": "F02.settings-link", "description": "the gear next to the backend selector, and the collapsed Settings icon, open `/settings` in the same tab for a local backend. On desktop that redirects to `/settings/agents`." }, { "id": "F02.conversation-list", "description": "the expanded rail embeds the conversation list (`conversation-panel`). The list itself belongs to the conversations family." }, { "id": "F02.command-menu-open-close", "description": "Control+k, Meta+k and the sidebar **Search commands** row (also the search icon on the collapsed rail) open a modal `Command menu` dialog with the input focused. Escape or the backdrop closes it, and the query is cleared on close. Escape returns focus to the element that had it before the menu opened (the **Search commands** row when it opened the menu). The row and the menu header show the shortcut as `Ctrl+K` off Apple platforms and `⌘K` on macOS and iOS." }, { "id": "F02.command-menu-focus", "description": "the shortcut works while typing in another field (the composer) and moves focus to the search input. Tab moves focus to the first option. Escape closes the menu from any focused element inside it and returns focus to the composer." }, { "id": "F02.command-menu-items", "description": "the menu shows Navigation (New chat, Customize, Automations, MCP servers), Settings (Settings, Agent settings, LLM profiles, Model Router, Condenser, Agent Context, Verification, Application, Secrets) and Actions (Toggle sidebar). Each **Go** item navigates to its route." }, { "id": "F02.command-menu-toggle-sidebar", "description": "the **Toggle sidebar** action collapses or expands the desktop rail." }, { "id": "F02.command-menu-search", "description": "every search term must match the title, description or keywords. **No commands found** is the empty state. The X button clears the query. Arrow keys wrap around the list, hovering an item selects it, Enter runs the selected item, and Enter does nothing when nothing matches." }, { "id": "F02.command-menu-new-tab", "description": "a middle-click or modifier-click on a **Go** item opens the route in a new tab and leaves the menu open." }, { "id": "F02.command-menu-phone", "description": "at 390 px the menu fits the screen and opens from the drawer's Search row. Escape closes the menu and leaves the drawer open. Running an item closes the drawer." }, { "id": "F02.mobile-drawer", "description": "at phone width the rail is hidden and a hamburger (`sidebar-mobile-menu-toggle`) opens a 300 px drawer. The hamburger is in the top bar, or in the chat header on conversation pages. The drawer closes on its chevron, Escape, a backdrop click or any route change." }, { "id": "F02.mobile-top-bar", "description": "at phone width, settings sub-pages and Customize detail pages (`/skills`, `/mcp`, `/plugins`, `/apps`) show a Back chevron that returns to `/settings` or `/customize`." }, { "id": "F02.checklist", "description": "after onboarding, the expanded rail shows a **Getting started** card with **N complete** and six items. They link to `/settings/llm`, `/conversations`, `/automations`, `/settings/agents`, `/mcp` and the external Slack invite." }, { "id": "F02.checklist-item-links", "description": "clicking an item's row navigates in the same tab to its route (for example **Schedule a task** opens `/automations`) without marking the item done." }, { "id": "F02.checklist-progress", "description": "items get a check and a strikethrough when done. The LLM item completes when a profile is configured, Start your first chat when a conversation exists, Customize your agent after a visit to `/settings/agents`, and Join Slack on click." }, { "id": "F02.checklist-preview", "description": "hovering an item shows a preview with a description, a **Documentation** link and an action button that navigates." }, { "id": "F02.checklist-minimize", "description": "the card header minimizes and expands the card, and the state survives a reload." }, { "id": "F02.checklist-dismiss", "description": "hovering or focusing the card (expanded or minimized) reveals an X (`Dismiss checklist`) in its header; touch devices always show it. Clicking it hides the card for good (it stays hidden after a reload) and turns off the **Show Getting Started checklist** switch in Settings → Application, which brings the card back in its previous expanded or minimized state." }, { "id": "F02.checklist-hide", "description": "the **Show Getting Started checklist** switch in Settings → Application hides the card (it stays hidden after a reload) and can show it again." }, { "id": "F02.checklist-all-complete", "description": "the card disappears once all six items are complete. Completion is recomputed from current data, so deleting the automation or the MCP server brings the card back." }, { "id": "F02.super-admin-setup-guide", "description": "on an OpenHands Enterprise (cloud) backend with `ENABLE_SUPER_ADMIN`, the first Super Admin (`/me` permissions include `manage_super_admins`) sees a floating **Setup guide** panel in the lower-right corner with `n/4` progress, the steps Add an LLM, Choose an automation template, Add an integration, Invite users and Optional: SAML / instance, a `Next: ` status and a **Start** link for that step (the positive Enterprise path remains unverified here). Close collapses it to a pill that reopens it. While the guide loads or shows, the **Getting started** card is hidden. Local backends never show it and make no `/api/admin/setup-state` request (#17969)." }, { "id": "F02.super-admin-setup-tour", "description": "for the two setup steps done in Canvas (Choose an automation template, Add an integration), the guide walks the Super Admin through the page with a guided tour (`.oh-setup-tour-popover` with Back / Next / Done): the template step points at the recommended-automations section (a stop with no footer: no Back, Next or `1/2`, #18243), then the setup dialog; the integration step at the MCP library on `/mcp` (`mcp-marketplace-section`, \"Pick a server from the library\"), then the install dialog (#18261). A plain click on a Canvas step's row in the guide starts that step's tour and closes the panel (a modifier click follows the link), and while the tour waits for its next stop's element it hides its popover and overlay, so it never covers another dialog (#18261). Finishing the guide's next step in Canvas opens the step after it the way **Start** does (a Canvas step's tour or page; an enterprise step reopens the guide to point at Start, or navigates on a locked-Cloud canvas). An enterprise page that links back with `?setup_tour=` starts that step's tour once and drops the parameter. Local backends ignore `?setup_tour=` (no tour, no guide, the parameter stays) (#18138). The positive path is blocked here, with the same Enterprise prerequisite as the setup guide above." }, { "id": "F02.update-tile", "description": "when npm `latest` is newer than the running version, a **New version** tile (`agent-canvas-version-tile`) opens an update modal with the npm and Docker commands. The tile is never shown in locked-Cloud mode." }, { "id": "F02.alert-banner", "description": "when the server config reports maintenance, faulty models or an error message, a dismissible banner (`alert-banner`) appears at the top of the content area." }, { "id": "F02.settings-404-modal", "description": "if the backend's settings return 404, the LLM settings modal opens by itself. Other settings fetch errors show a toast." }, { "id": "F02.error-toasts", "description": "failed requests show a top-right error toast with the product or server message, and the toast closes by itself. Toasts stay readable in every colour theme." } ], "scenarios": [ { "name": "F02--sidebar-links", "title": "Sidebar links", "ids": [ "F02.nav-links" ], "body": "Run `control-openhands browser goto /conversations`, `control-openhands browser click 'testid=sidebar-skills-link'`, then `control-openhands browser wait-url '/mcp$'`. The URL ends in `/mcp` because desktop `/customize` redirects there. Run `control-openhands browser attr 'testid=sidebar-skills-link' class`; the class contains `bg-tertiary` (the active style), and `control-openhands browser attr 'testid=sidebar-skills-link' aria-current` is `page`. Repeat `browser goto` with `/skills`, `/plugins` and `/apps`; each time the class still contains `bg-tertiary` and `aria-current` is still `page`. Run `control-openhands browser click 'testid=sidebar-automations-link'`, `control-openhands browser url` (ends in `/automations`) and `control-openhands browser attr 'testid=sidebar-automations-link' aria-current` (`page`); `control-openhands browser attr 'testid=sidebar-skills-link' aria-current` is now `null`. Then run `control-openhands browser click 'testid=sidebar-conversations-link'` and `control-openhands browser click 'aside[data-collapsed] >> role=link[name=\"OpenHands Logo\"]'`; both leave the URL at `/conversations`.", "raw": "**Sidebar links (`F02.nav-links`).** Run `control-openhands browser goto /conversations`, `control-openhands browser click 'testid=sidebar-skills-link'`, then `control-openhands browser wait-url '/mcp$'`. The URL ends in `/mcp` because desktop `/customize` redirects there. Run `control-openhands browser attr 'testid=sidebar-skills-link' class`; the class contains `bg-tertiary` (the active style), and `control-openhands browser attr 'testid=sidebar-skills-link' aria-current` is `page`. Repeat `browser goto` with `/skills`, `/plugins` and `/apps`; each time the class still contains `bg-tertiary` and `aria-current` is still `page`. Run `control-openhands browser click 'testid=sidebar-automations-link'`, `control-openhands browser url` (ends in `/automations`) and `control-openhands browser attr 'testid=sidebar-automations-link' aria-current` (`page`); `control-openhands browser attr 'testid=sidebar-skills-link' aria-current` is now `null`. Then run `control-openhands browser click 'testid=sidebar-conversations-link'` and `control-openhands browser click 'aside[data-collapsed] >> role=link[name=\"OpenHands Logo\"]'`; both leave the URL at `/conversations`.", "source_line": 67, "expected": [ "the logo and **New Chat** open `/conversations`. **Customize** opens `/customize`, which redirects to `/mcp` on desktop, and stays highlighted and marked `aria-current=\"page\"` on `/skills`, `/plugins`, `/apps` and `/mcp`. **Automate** opens `/automations`. Canvas extension pages add their own rows (`sidebar-canvas-extension--`)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--pin-as-home", "title": "Pin as home", "ids": [ "F02.pin-as-home" ], "body": "From `/conversations` run `control-openhands browser visible 'testid=sidebar-pin-home-toggle-customize'` (`false`), `control-openhands browser hover 'testid=sidebar-skills-link'`, the same `visible` (`true`), then `control-openhands browser click 'testid=sidebar-pin-home-toggle-customize'` and `control-openhands browser attr 'testid=sidebar-pin-home-toggle-customize' aria-pressed` (`true`). Run `control-openhands browser goto /` and `control-openhands browser wait-url '/mcp$'`; the URL ends in `/mcp`. Run `control-openhands browser screenshot 'aside[data-collapsed]' --feature F02.pin-as-home --name customize-pinned`; the screenshot shows a filled pin on the Customize row with no hover. Move the pin: run `control-openhands browser hover 'testid=sidebar-automations-link'` and `control-openhands browser click 'testid=sidebar-pin-home-toggle-automations'`. The Customize pin's `aria-pressed` becomes `false`, and `control-openhands browser goto /` lands on `/automations`. Unpin with `control-openhands browser click 'testid=sidebar-pin-home-toggle-automations'`, then run `control-openhands browser goto /` and `control-openhands browser count 'testid=home-screen'`. The URL stays `/` and the count is `1`. `control-openhands browser storage --values` shows `oh:pinned-home-route::` as `null`.", "raw": "**Pin as home (`F02.pin-as-home`).** From `/conversations` run `control-openhands browser visible 'testid=sidebar-pin-home-toggle-customize'` (`false`), `control-openhands browser hover 'testid=sidebar-skills-link'`, the same `visible` (`true`), then `control-openhands browser click 'testid=sidebar-pin-home-toggle-customize'` and `control-openhands browser attr 'testid=sidebar-pin-home-toggle-customize' aria-pressed` (`true`). Run `control-openhands browser goto /` and `control-openhands browser wait-url '/mcp$'`; the URL ends in `/mcp`. Run `control-openhands browser screenshot 'aside[data-collapsed]' --feature F02.pin-as-home --name customize-pinned`; the screenshot shows a filled pin on the Customize row with no hover. Move the pin: run `control-openhands browser hover 'testid=sidebar-automations-link'` and `control-openhands browser click 'testid=sidebar-pin-home-toggle-automations'`. The Customize pin's `aria-pressed` becomes `false`, and `control-openhands browser goto /` lands on `/automations`. Unpin with `control-openhands browser click 'testid=sidebar-pin-home-toggle-automations'`, then run `control-openhands browser goto /` and `control-openhands browser count 'testid=home-screen'`. The URL stays `/` and the count is `1`. `control-openhands browser storage --values` shows `oh:pinned-home-route::` as `null`.", "source_line": 68, "expected": [ "hovering Customize or Automate reveals a pin. Pinning makes `/` redirect to that page, pinning the other one moves the pin, and unpinning restores the default home. The pin is stored per backend and org." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--collapse-and-expand", "title": "Collapse and expand", "ids": [ "F02.sidebar-collapse" ], "body": "Run `control-openhands browser click 'testid=sidebar-collapse-toggle'` and `control-openhands browser attr 'aside[data-collapsed]' data-collapsed` (`true`). Run `control-openhands browser reload`; the attribute is still `true`, `browser storage --values` shows `openhands-sidebar` with `\"collapsed\":true`, and `control-openhands browser bbox 'aside[data-collapsed]'` reports width `60`. `control-openhands browser count 'testid=sidebar-onboarding-checklist'` is `0`. For the tooltip, hover one icon, then another: `control-openhands browser hover 'testid=sidebar-automations-link'`, `control-openhands browser hover 'testid=sidebar-skills-link'`, then `control-openhands browser wait 'role=tooltip'` and `control-openhands browser text 'role=tooltip'` (`Customize`). For the backend popover, run `control-openhands browser hover 'testid=collapsed-backend-selector-link'`; its `aria-expanded` is `true`, `browser testids --filter backend` lists `add-backend-menu-item` and `manage-backends-menu-item`, and `control-openhands browser screenshot --feature F02.sidebar-collapse --name backend-popover` shows the popover. Expand by clicking empty rail space with `control-openhands browser click 'aside[data-collapsed]'` (`data-collapsed` becomes `false`). Collapse again, then run `control-openhands browser hover 'aside[data-collapsed] >> role=link[name=\"OpenHands Logo\"]'`; the toggle's class has `opacity-100` and its `aria-label` is `Expand sidebar`. `control-openhands browser click 'testid=sidebar-collapse-toggle'` expands the rail.", "raw": "**Collapse and expand (`F02.sidebar-collapse`).** Run `control-openhands browser click 'testid=sidebar-collapse-toggle'` and `control-openhands browser attr 'aside[data-collapsed]' data-collapsed` (`true`). Run `control-openhands browser reload`; the attribute is still `true`, `browser storage --values` shows `openhands-sidebar` with `\"collapsed\":true`, and `control-openhands browser bbox 'aside[data-collapsed]'` reports width `60`. `control-openhands browser count 'testid=sidebar-onboarding-checklist'` is `0`. For the tooltip, hover one icon, then another: `control-openhands browser hover 'testid=sidebar-automations-link'`, `control-openhands browser hover 'testid=sidebar-skills-link'`, then `control-openhands browser wait 'role=tooltip'` and `control-openhands browser text 'role=tooltip'` (`Customize`). For the backend popover, run `control-openhands browser hover 'testid=collapsed-backend-selector-link'`; its `aria-expanded` is `true`, `browser testids --filter backend` lists `add-backend-menu-item` and `manage-backends-menu-item`, and `control-openhands browser screenshot --feature F02.sidebar-collapse --name backend-popover` shows the popover. Expand by clicking empty rail space with `control-openhands browser click 'aside[data-collapsed]'` (`data-collapsed` becomes `false`). Collapse again, then run `control-openhands browser hover 'aside[data-collapsed] >> role=link[name=\"OpenHands Logo\"]'`; the toggle's class has `opacity-100` and its `aria-label` is `Expand sidebar`. `control-openhands browser click 'testid=sidebar-collapse-toggle'` expands the rail.", "source_line": 69, "expected": [ "the chevron collapses the rail to a 60 px icon rail with hover tooltips, a Settings icon and a backend icon with a status dot that opens a backend popover. The checklist is hidden while collapsed. The state survives a reload. To expand, click the chevron that appears when you hover the logo, or click empty rail space." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--settings-gear", "title": "Settings gear", "ids": [ "F02.settings-link" ], "body": "The previous bullet expanded the rail: collapse it with `control-openhands browser click 'testid=sidebar-collapse-toggle'`, then run `control-openhands browser click 'testid=collapsed-settings-link'` and `control-openhands browser wait-url '/settings'`. Expand it again with `control-openhands browser click 'testid=sidebar-collapse-toggle'`, then run `control-openhands browser goto /conversations`, `control-openhands browser click 'testid=backend-selector-settings-link'` and `control-openhands browser url`; the URL ends in `/settings/agents`. `control-openhands browser attr 'testid=backend-selector-settings-link' target` is `null` (same tab).", "raw": "**Settings gear (`F02.settings-link`).** The previous bullet expanded the rail: collapse it with `control-openhands browser click 'testid=sidebar-collapse-toggle'`, then run `control-openhands browser click 'testid=collapsed-settings-link'` and `control-openhands browser wait-url '/settings'`. Expand it again with `control-openhands browser click 'testid=sidebar-collapse-toggle'`, then run `control-openhands browser goto /conversations`, `control-openhands browser click 'testid=backend-selector-settings-link'` and `control-openhands browser url`; the URL ends in `/settings/agents`. `control-openhands browser attr 'testid=backend-selector-settings-link' target` is `null` (same tab).", "source_line": 70, "expected": [ "the gear next to the backend selector, and the collapsed Settings icon, open `/settings` in the same tab for a local backend. On desktop that redirects to `/settings/agents`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--conversation-list", "title": "Conversation list", "ids": [ "F02.conversation-list" ], "body": "After `conversation start` (preconditions), run `control-openhands browser goto /conversations` and `control-openhands browser testids 'testid=conversation-panel'`. The list includes `conversation-card`.", "raw": "**Conversation list (`F02.conversation-list`).** After `conversation start` (preconditions), run `control-openhands browser goto /conversations` and `control-openhands browser testids 'testid=conversation-panel'`. The list includes `conversation-card`.", "source_line": 71, "expected": [ "the expanded rail embeds the conversation list (`conversation-panel`). The list itself belongs to the conversations family." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--open-and-close-the-menu", "title": "Open and close the menu", "ids": [ "F02.command-menu-open-close" ], "body": "From `/conversations` run `control-openhands browser text 'testid=command-menu-trigger'`. It reads `Search commands` and `CTRL+K` on Linux and Windows (the hint is `Ctrl+K`, drawn in capitals; macOS and iOS show `⌘K`). Run `control-openhands browser press Control+k`, then `control-openhands browser visible 'testid=command-menu'` (`true`), `control-openhands browser eval \"document.activeElement && document.activeElement.id\"` (`command-menu-search`) and `control-openhands browser text 'testid=command-menu >> css=kbd'` (`CTRL+K`, the header hint). `control-openhands browser screenshot --feature F02.command-menu-open-close --name open` shows the dimmed, blurred backdrop. Run `control-openhands browser press Escape`; `browser count 'testid=command-menu'` is `0`. Run `control-openhands browser press Meta+k` (count `1`), `control-openhands browser type 'testid=command-menu >> role=combobox' secr`, then `control-openhands browser click 'testid=command-menu >> role=button[name=\"Close command menu\"]'` (the backdrop; count `0`). Run `control-openhands browser click 'testid=command-menu-trigger'` and `control-openhands browser value 'testid=command-menu >> role=combobox'`; the value is `\"\"` because the query was cleared on close. Close it with `control-openhands browser press Escape`; focus is back on the row (`control-openhands browser eval \"document.activeElement.dataset.testid\"` is `command-menu-trigger`). Collapsed rail: run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, `control-openhands browser click 'testid=command-menu-trigger'` (the search icon; count `1`), `control-openhands browser press Escape`, then expand with `control-openhands browser click 'aside[data-collapsed]'`.", "raw": "**Open and close the menu (`F02.command-menu-open-close`).** From `/conversations` run `control-openhands browser text 'testid=command-menu-trigger'`. It reads `Search commands` and `CTRL+K` on Linux and Windows (the hint is `Ctrl+K`, drawn in capitals; macOS and iOS show `⌘K`). Run `control-openhands browser press Control+k`, then `control-openhands browser visible 'testid=command-menu'` (`true`), `control-openhands browser eval \"document.activeElement && document.activeElement.id\"` (`command-menu-search`) and `control-openhands browser text 'testid=command-menu >> css=kbd'` (`CTRL+K`, the header hint). `control-openhands browser screenshot --feature F02.command-menu-open-close --name open` shows the dimmed, blurred backdrop. Run `control-openhands browser press Escape`; `browser count 'testid=command-menu'` is `0`. Run `control-openhands browser press Meta+k` (count `1`), `control-openhands browser type 'testid=command-menu >> role=combobox' secr`, then `control-openhands browser click 'testid=command-menu >> role=button[name=\"Close command menu\"]'` (the backdrop; count `0`). Run `control-openhands browser click 'testid=command-menu-trigger'` and `control-openhands browser value 'testid=command-menu >> role=combobox'`; the value is `\"\"` because the query was cleared on close. Close it with `control-openhands browser press Escape`; focus is back on the row (`control-openhands browser eval \"document.activeElement.dataset.testid\"` is `command-menu-trigger`). Collapsed rail: run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, `control-openhands browser click 'testid=command-menu-trigger'` (the search icon; count `1`), `control-openhands browser press Escape`, then expand with `control-openhands browser click 'aside[data-collapsed]'`.", "source_line": 72, "expected": [ "Control+k, Meta+k and the sidebar **Search commands** row (also the search icon on the collapsed rail) open a modal `Command menu` dialog with the input focused. Escape or the backdrop closes it, and the query is cleared on close. Escape returns focus to the element that had it before the menu opened (the **Search commands** row when it opened the menu). The row and the menu header show the shortcut as `Ctrl+K` off Apple platforms and `⌘K` on macOS and iOS." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--every-item-navigates", "title": "Every item navigates", "ids": [ "F02.command-menu-items" ], "body": "The previous bullet closed the menu: run `control-openhands browser press Control+k`, then `control-openhands browser snapshot 'testid=command-menu' --feature F02.command-menu-items --name all-items`; it lists 14 options under Navigation, Settings and Actions, with Model Router after LLM profiles and Agent Context after Condenser settings. Close it with `control-openhands browser press Escape` and rest the pointer off the list with `control-openhands browser hover 'testid=command-menu-trigger'` (see Gotchas). For each item run `control-openhands browser goto /conversations`, `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' ''`, `control-openhands browser press Enter` and `control-openhands browser wait-url '$'`. Expected routes: `New chat` → `/conversations`, `Customize` → `/mcp` (via `/customize`), `Automations` → `/automations`, `MCP servers` → `/mcp`, `Settings overview` → `/settings/agents` (via `/settings`), `Agent settings` → `/settings/agents`, `LLM profiles` → `/settings/llm`, `Model Router` → `/settings/meta-llm`, `Condenser` → `/settings/condenser`, `Agent Context` → `/settings/agent-context`, `Verification` → `/settings/verification`, `Application` → `/settings/app`, `Secrets` → `/settings/secrets`. `browser count 'testid=command-menu'` is `0` after each run. Click path: `control-openhands browser goto /settings/app`, open the menu, type `new chat`, then `control-openhands browser click 'testid=command-menu >> role=option[name=/New chat/]'`; the URL becomes `/conversations`.", "raw": "**Every item navigates (`F02.command-menu-items`).** The previous bullet closed the menu: run `control-openhands browser press Control+k`, then `control-openhands browser snapshot 'testid=command-menu' --feature F02.command-menu-items --name all-items`; it lists 14 options under Navigation, Settings and Actions, with Model Router after LLM profiles and Agent Context after Condenser settings. Close it with `control-openhands browser press Escape` and rest the pointer off the list with `control-openhands browser hover 'testid=command-menu-trigger'` (see Gotchas). For each item run `control-openhands browser goto /conversations`, `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' ''`, `control-openhands browser press Enter` and `control-openhands browser wait-url '$'`. Expected routes: `New chat` → `/conversations`, `Customize` → `/mcp` (via `/customize`), `Automations` → `/automations`, `MCP servers` → `/mcp`, `Settings overview` → `/settings/agents` (via `/settings`), `Agent settings` → `/settings/agents`, `LLM profiles` → `/settings/llm`, `Model Router` → `/settings/meta-llm`, `Condenser` → `/settings/condenser`, `Agent Context` → `/settings/agent-context`, `Verification` → `/settings/verification`, `Application` → `/settings/app`, `Secrets` → `/settings/secrets`. `browser count 'testid=command-menu'` is `0` after each run. Click path: `control-openhands browser goto /settings/app`, open the menu, type `new chat`, then `control-openhands browser click 'testid=command-menu >> role=option[name=/New chat/]'`; the URL becomes `/conversations`.", "source_line": 73, "expected": [ "the menu shows Navigation (New chat, Customize, Automations, MCP servers), Settings (Settings, Agent settings, LLM profiles, Model Router, Condenser, Agent Context, Verification, Application, Secrets) and Actions (Toggle sidebar). Each **Go** item navigates to its route." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--toggle-sidebar", "title": "Toggle sidebar", "ids": [ "F02.command-menu-toggle-sidebar" ], "body": "Run `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' toggle`, `control-openhands browser press Enter`, then `control-openhands browser attr 'aside[data-collapsed]' data-collapsed` (`true`). Repeat with the term `sidebar`; the value returns to `false`.", "raw": "**Toggle sidebar (`F02.command-menu-toggle-sidebar`).** Run `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' toggle`, `control-openhands browser press Enter`, then `control-openhands browser attr 'aside[data-collapsed]' data-collapsed` (`true`). Repeat with the term `sidebar`; the value returns to `false`.", "source_line": 74, "expected": [ "the **Toggle sidebar** action collapses or expands the desktop rail." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--search-empty-state-keyboard", "title": "Search, empty state, keyboard", "ids": [ "F02.command-menu-search" ], "body": "The Toggle sidebar action closed the menu: open it with `control-openhands browser press Control+k`, then run `control-openhands browser fill 'testid=command-menu >> role=combobox' theme` and `control-openhands browser count 'testid=command-menu >> role=option'` (`1`, Application, matched by its keywords). The term `settings secret` leaves `1` option, `secret automations` leaves `0` (all terms must match) and `model` leaves `3` (MCP servers, LLM profiles, Model Router). Fill `qa-zzz-nothing`, then run `control-openhands browser text 'testid=command-menu >> role=listbox'` (`No commands found` / `Try a page, action, or setting name.`) and `control-openhands browser screenshot --feature F02.command-menu-search --name no-results`. Run `control-openhands browser press Enter`; the menu stays open and the URL does not change. Run `control-openhands browser click 'testid=command-menu >> role=button[name=\"Clear search\"]'`; the value is `\"\"` and the count is back to `14`. Keyboard: `control-openhands browser attr 'testid=command-menu >> role=combobox' aria-activedescendant` starts at `command-menu-option-new-chat`. `control-openhands browser press ArrowUp` wraps to `command-menu-option-toggle-sidebar`, and `ArrowDown` returns to `new-chat`, then moves to `customize`. `control-openhands browser hover 'testid=command-menu >> role=option[name=/Secrets settings/]'` sets the value to `command-menu-option-secrets-settings`.", "raw": "**Search, empty state, keyboard (`F02.command-menu-search`).** The Toggle sidebar action closed the menu: open it with `control-openhands browser press Control+k`, then run `control-openhands browser fill 'testid=command-menu >> role=combobox' theme` and `control-openhands browser count 'testid=command-menu >> role=option'` (`1`, Application, matched by its keywords). The term `settings secret` leaves `1` option, `secret automations` leaves `0` (all terms must match) and `model` leaves `3` (MCP servers, LLM profiles, Model Router). Fill `qa-zzz-nothing`, then run `control-openhands browser text 'testid=command-menu >> role=listbox'` (`No commands found` / `Try a page, action, or setting name.`) and `control-openhands browser screenshot --feature F02.command-menu-search --name no-results`. Run `control-openhands browser press Enter`; the menu stays open and the URL does not change. Run `control-openhands browser click 'testid=command-menu >> role=button[name=\"Clear search\"]'`; the value is `\"\"` and the count is back to `14`. Keyboard: `control-openhands browser attr 'testid=command-menu >> role=combobox' aria-activedescendant` starts at `command-menu-option-new-chat`. `control-openhands browser press ArrowUp` wraps to `command-menu-option-toggle-sidebar`, and `ArrowDown` returns to `new-chat`, then moves to `customize`. `control-openhands browser hover 'testid=command-menu >> role=option[name=/Secrets settings/]'` sets the value to `command-menu-option-secrets-settings`.", "source_line": 75, "expected": [ "every search term must match the title, description or keywords. **No commands found** is the empty state. The X button clears the query. Arrow keys wrap around the list, hovering an item selects it, Enter runs the selected item, and Enter does nothing when nothing matches." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--open-in-a-new-tab", "title": "Open in a new tab", "ids": [ "F02.command-menu-new-tab" ], "body": "With the menu open and `secrets` typed, `control-openhands browser attr 'testid=command-menu >> role=option[name=/Secrets settings/]' href` is `/settings/secrets`. Run `control-openhands browser click 'testid=command-menu >> role=option[name=/Secrets settings/]' --button middle` and `control-openhands browser tabs`; tab 1 is `/settings/secrets`, tab 0 keeps its URL, and `browser count 'testid=command-menu'` is still `1`. Run `control-openhands browser close-tab 1`. The modifier path works the same: `control-openhands browser click 'testid=command-menu >> role=option[name=/Secrets settings/]' --modifiers Control` opens tab 1 with the menu still open; close that tab too. The click leaves focus on the option; `control-openhands browser press Escape` still closes the menu (count `0`).", "raw": "**Open in a new tab (`F02.command-menu-new-tab`).** With the menu open and `secrets` typed, `control-openhands browser attr 'testid=command-menu >> role=option[name=/Secrets settings/]' href` is `/settings/secrets`. Run `control-openhands browser click 'testid=command-menu >> role=option[name=/Secrets settings/]' --button middle` and `control-openhands browser tabs`; tab 1 is `/settings/secrets`, tab 0 keeps its URL, and `browser count 'testid=command-menu'` is still `1`. Run `control-openhands browser close-tab 1`. The modifier path works the same: `control-openhands browser click 'testid=command-menu >> role=option[name=/Secrets settings/]' --modifiers Control` opens tab 1 with the menu still open; close that tab too. The click leaves focus on the option; `control-openhands browser press Escape` still closes the menu (count `0`).", "source_line": 76, "expected": [ "a middle-click or modifier-click on a **Go** item opens the route in a new tab and leaves the menu open." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--keyboard-focus", "title": "Keyboard focus", "ids": [ "F02.command-menu-focus" ], "body": "From `/conversations` run `control-openhands browser click 'testid=chat-input'`, `control-openhands browser press Control+k`, `browser count 'testid=command-menu'` (`1`) and `control-openhands browser eval \"document.activeElement && document.activeElement.id\"` (`command-menu-search`): the shortcut opens the menu from inside the composer. Run `control-openhands browser press Tab`; the same `eval` returns `command-menu-option-new-chat`. `control-openhands browser press Escape` closes the menu from the option (count `0`), and `control-openhands browser eval \"document.activeElement.dataset.testid\"` returns `chat-input`: focus is back in the composer.", "raw": "**Keyboard focus (`F02.command-menu-focus`).** From `/conversations` run `control-openhands browser click 'testid=chat-input'`, `control-openhands browser press Control+k`, `browser count 'testid=command-menu'` (`1`) and `control-openhands browser eval \"document.activeElement && document.activeElement.id\"` (`command-menu-search`): the shortcut opens the menu from inside the composer. Run `control-openhands browser press Tab`; the same `eval` returns `command-menu-option-new-chat`. `control-openhands browser press Escape` closes the menu from the option (count `0`), and `control-openhands browser eval \"document.activeElement.dataset.testid\"` returns `chat-input`: focus is back in the composer.", "source_line": 77, "expected": [ "the shortcut works while typing in another field (the composer) and moves focus to the search input. Tab moves focus to the first option. Escape closes the menu from any focused element inside it and returns focus to the composer." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--phone-drawer", "title": "Phone drawer", "ids": [ "F02.mobile-drawer" ], "body": "Run `control-openhands browser viewport phone` and `control-openhands browser goto /conversations`. `control-openhands browser visible 'aside[data-collapsed]'` is `false`. Run `control-openhands browser click 'testid=sidebar-mobile-menu-toggle'` and `control-openhands browser wait 'testid=sidebar-mobile-drawer'`. `control-openhands browser bbox 'testid=sidebar-mobile-drawer'` gives width `300`, `insideViewport` `true` and `pageHorizontalOverflow` `false`, and `control-openhands browser screenshot --feature F02.mobile-drawer --name open` shows the full rail. Close paths, each followed by `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state detached`: `control-openhands browser click 'testid=sidebar-mobile-drawer-close'`; `control-openhands browser press Escape`; `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=sidebar-automations-link'` (route change; the URL becomes `/automations`); and a backdrop click to the right of the drawer, `control-openhands browser mouse-click 350 400` (on `/conversations`, `control-openhands browser click 'testid=submit-button' --force` also lands on the backdrop, because the empty composer's send button sits at x≈326 under it). On a conversation page (`browser goto` the URL from `conversation start`), `control-openhands browser bbox 'testid=sidebar-mobile-menu-toggle'` is at the top-left of the chat header. `control-openhands browser screenshot --feature F02.mobile-drawer --name conversation-header` shows it beside the title, and clicking it opens the drawer. Return with `control-openhands browser viewport desktop`.", "raw": "**Phone drawer (`F02.mobile-drawer`).** Run `control-openhands browser viewport phone` and `control-openhands browser goto /conversations`. `control-openhands browser visible 'aside[data-collapsed]'` is `false`. Run `control-openhands browser click 'testid=sidebar-mobile-menu-toggle'` and `control-openhands browser wait 'testid=sidebar-mobile-drawer'`. `control-openhands browser bbox 'testid=sidebar-mobile-drawer'` gives width `300`, `insideViewport` `true` and `pageHorizontalOverflow` `false`, and `control-openhands browser screenshot --feature F02.mobile-drawer --name open` shows the full rail. Close paths, each followed by `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state detached`: `control-openhands browser click 'testid=sidebar-mobile-drawer-close'`; `control-openhands browser press Escape`; `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=sidebar-automations-link'` (route change; the URL becomes `/automations`); and a backdrop click to the right of the drawer, `control-openhands browser mouse-click 350 400` (on `/conversations`, `control-openhands browser click 'testid=submit-button' --force` also lands on the backdrop, because the empty composer's send button sits at x≈326 under it). On a conversation page (`browser goto` the URL from `conversation start`), `control-openhands browser bbox 'testid=sidebar-mobile-menu-toggle'` is at the top-left of the chat header. `control-openhands browser screenshot --feature F02.mobile-drawer --name conversation-header` shows it beside the title, and clicking it opens the drawer. Return with `control-openhands browser viewport desktop`.", "source_line": 78, "expected": [ "at phone width the rail is hidden and a hamburger (`sidebar-mobile-menu-toggle`) opens a 300 px drawer. The hamburger is in the top bar, or in the chat header on conversation pages. The drawer closes on its chevron, Escape, a backdrop click or any route change." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--back-chevron", "title": "Back chevron", "ids": [ "F02.mobile-top-bar" ], "body": "The Phone drawer bullet returned to desktop: run `control-openhands browser viewport phone`, `control-openhands browser goto /settings/llm` and `control-openhands browser attr 'testid=sidebar-mobile-back-button' aria-label` (`Settings`). Then run `control-openhands browser click 'testid=sidebar-mobile-back-button'` and `control-openhands browser url`; the URL is `/settings`, and on that page `browser count 'testid=sidebar-mobile-back-button'` is `0`. `control-openhands browser screenshot --feature F02.mobile-top-bar --name settings-root` shows only the hamburger above the settings menu. Do the same from `/skills`: the label is `Customize` and the click lands on `/customize`, which on phone is the hub (`extensions-mobile-hub`).", "raw": "**Back chevron (`F02.mobile-top-bar`).** The Phone drawer bullet returned to desktop: run `control-openhands browser viewport phone`, `control-openhands browser goto /settings/llm` and `control-openhands browser attr 'testid=sidebar-mobile-back-button' aria-label` (`Settings`). Then run `control-openhands browser click 'testid=sidebar-mobile-back-button'` and `control-openhands browser url`; the URL is `/settings`, and on that page `browser count 'testid=sidebar-mobile-back-button'` is `0`. `control-openhands browser screenshot --feature F02.mobile-top-bar --name settings-root` shows only the hamburger above the settings menu. Do the same from `/skills`: the label is `Customize` and the click lands on `/customize`, which on phone is the hub (`extensions-mobile-hub`).", "source_line": 79, "expected": [ "at phone width, settings sub-pages and Customize detail pages (`/skills`, `/mcp`, `/plugins`, `/apps`) show a Back chevron that returns to `/settings` or `/customize`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--phone-command-menu", "title": "Phone command menu", "ids": [ "F02.command-menu-phone" ], "body": "At phone width open the drawer, then run `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=command-menu-trigger'`. `control-openhands browser bbox 'testid=command-menu >> role=listbox'` reports `insideViewport` `true` and `pageHorizontalOverflow` `false`, and `control-openhands browser screenshot --feature F02.command-menu-phone --name open` shows it. `control-openhands browser press Escape` closes only the menu: `browser count 'testid=command-menu'` is `0` and `control-openhands browser visible 'testid=sidebar-mobile-drawer'` is still `true`. Reopen it with `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=command-menu-trigger'`, type `secrets` and press Enter; `control-openhands browser wait-url 'secrets$'` passes and `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state detached` passes.", "raw": "**Phone command menu (`F02.command-menu-phone`).** At phone width open the drawer, then run `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=command-menu-trigger'`. `control-openhands browser bbox 'testid=command-menu >> role=listbox'` reports `insideViewport` `true` and `pageHorizontalOverflow` `false`, and `control-openhands browser screenshot --feature F02.command-menu-phone --name open` shows it. `control-openhands browser press Escape` closes only the menu: `browser count 'testid=command-menu'` is `0` and `control-openhands browser visible 'testid=sidebar-mobile-drawer'` is still `true`. Reopen it with `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=command-menu-trigger'`, type `secrets` and press Enter; `control-openhands browser wait-url 'secrets$'` passes and `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state detached` passes.", "source_line": 80, "expected": [ "at 390 px the menu fits the screen and opens from the drawer's Search row. Escape closes the menu and leaves the drawer open. Running an item closes the drawer." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--checklist-card", "title": "Checklist card", "ids": [ "F02.checklist" ], "body": "The phone bullets above leave the phone viewport on `/settings/secrets`: run `control-openhands browser viewport desktop`, `control-openhands browser goto /conversations`, then `control-openhands browser count 'testid=sidebar-onboarding-checklist >> role=listitem'` (`6`) and `control-openhands browser text 'testid=sidebar-onboarding-checklist'`. The first lines are `Getting started` and `N complete`. `control-openhands browser snapshot 'aside[data-collapsed]'` shows the six links: `/settings/llm`, `/conversations`, `/automations`, `/settings/agents`, `/mcp` and `https://openhands.dev/joinslack`.", "raw": "**Checklist card (`F02.checklist`).** The phone bullets above leave the phone viewport on `/settings/secrets`: run `control-openhands browser viewport desktop`, `control-openhands browser goto /conversations`, then `control-openhands browser count 'testid=sidebar-onboarding-checklist >> role=listitem'` (`6`) and `control-openhands browser text 'testid=sidebar-onboarding-checklist'`. The first lines are `Getting started` and `N complete`. `control-openhands browser snapshot 'aside[data-collapsed]'` shows the six links: `/settings/llm`, `/conversations`, `/automations`, `/settings/agents`, `/mcp` and `https://openhands.dev/joinslack`.", "source_line": 81, "expected": [ "after onboarding, the expanded rail shows a **Getting started** card with **N complete** and six items. They link to `/settings/llm`, `/conversations`, `/automations`, `/settings/agents`, `/mcp` and the external Slack invite." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--item-links", "title": "Item links", "ids": [ "F02.checklist-item-links" ], "body": "Run `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-schedule-task' --expect-url '/automations(\\?|$)'` and `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-configure-llm' --expect-url '/settings/llm(\\?|$)'`; both pass, `browser tabs` still lists one tab, and `browser count 'testid=sidebar-onboarding-checklist-item-schedule-task >> css=span.line-through'` stays `0` (visiting is not completing).", "raw": "**Item links (`F02.checklist-item-links`).** Run `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-schedule-task' --expect-url '/automations(\\?|$)'` and `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-configure-llm' --expect-url '/settings/llm(\\?|$)'`; both pass, `browser tabs` still lists one tab, and `browser count 'testid=sidebar-onboarding-checklist-item-schedule-task >> css=span.line-through'` stays `0` (visiting is not completing).", "source_line": 82, "expected": [ "clicking an item's row navigates in the same tab to its route (for example **Schedule a task** opens `/automations`) without marking the item done." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--progress", "title": "Progress", "ids": [ "F02.checklist-progress" ], "body": "Each completed item renders a struck-through label, so `control-openhands browser count 'testid=sidebar-onboarding-checklist-item- >> css=span.line-through'` is `1`. Run `control-openhands browser goto /settings/agents`, then come back; `customize-agent` is done. Run `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-join-slack'`; `join-slack` is done and `browser tabs` shows an external Slack tab, which you close with `control-openhands browser close-tab 1`. After `llm preset deepseek`, run `control-openhands browser reload` (the open page does not refetch settings written through the API), then `control-openhands browser wait 'testid=sidebar-onboarding-checklist-item-configure-llm >> css=span.line-through' --timeout 15000`. After `conversation start`, `start-conversation` is done. The header count rises each time (`4 complete` after these four).", "raw": "**Progress (`F02.checklist-progress`).** Each completed item renders a struck-through label, so `control-openhands browser count 'testid=sidebar-onboarding-checklist-item- >> css=span.line-through'` is `1`. Run `control-openhands browser goto /settings/agents`, then come back; `customize-agent` is done. Run `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-join-slack'`; `join-slack` is done and `browser tabs` shows an external Slack tab, which you close with `control-openhands browser close-tab 1`. After `llm preset deepseek`, run `control-openhands browser reload` (the open page does not refetch settings written through the API), then `control-openhands browser wait 'testid=sidebar-onboarding-checklist-item-configure-llm >> css=span.line-through' --timeout 15000`. After `conversation start`, `start-conversation` is done. The header count rises each time (`4 complete` after these four).", "source_line": 83, "expected": [ "items get a check and a strikethrough when done. The LLM item completes when a profile is configured, Start your first chat when a conversation exists, Customize your agent after a visit to `/settings/agents`, and Join Slack on click." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--item-preview", "title": "Item preview", "ids": [ "F02.checklist-preview" ], "body": "Run `control-openhands browser hover 'testid=sidebar-onboarding-checklist-item-join-slack'`, `control-openhands browser hover 'testid=sidebar-onboarding-checklist-item-connect-mcp'` and `control-openhands browser wait 'testid=sidebar-onboarding-checklist-preview-connect-mcp' --timeout 5000`. Then run `control-openhands browser text 'testid=sidebar-onboarding-checklist-preview-connect-mcp'`; it reads the title, the description, `Documentation` and `Connect`. `control-openhands browser attr 'testid=sidebar-onboarding-checklist-preview-docs-connect-mcp' href` is `https://docs.openhands.dev/overview/model-context-protocol`. Take `control-openhands browser screenshot --feature F02.checklist-preview --name connect-mcp`, then run `control-openhands browser click 'testid=sidebar-onboarding-checklist-preview-action-connect-mcp'` and `control-openhands browser wait-url '/mcp$'`.", "raw": "**Item preview (`F02.checklist-preview`).** Run `control-openhands browser hover 'testid=sidebar-onboarding-checklist-item-join-slack'`, `control-openhands browser hover 'testid=sidebar-onboarding-checklist-item-connect-mcp'` and `control-openhands browser wait 'testid=sidebar-onboarding-checklist-preview-connect-mcp' --timeout 5000`. Then run `control-openhands browser text 'testid=sidebar-onboarding-checklist-preview-connect-mcp'`; it reads the title, the description, `Documentation` and `Connect`. `control-openhands browser attr 'testid=sidebar-onboarding-checklist-preview-docs-connect-mcp' href` is `https://docs.openhands.dev/overview/model-context-protocol`. Take `control-openhands browser screenshot --feature F02.checklist-preview --name connect-mcp`, then run `control-openhands browser click 'testid=sidebar-onboarding-checklist-preview-action-connect-mcp'` and `control-openhands browser wait-url '/mcp$'`.", "source_line": 84, "expected": [ "hovering an item shows a preview with a description, a **Documentation** link and an action button that navigates." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--minimize", "title": "Minimize", "ids": [ "F02.checklist-minimize" ], "body": "Run `control-openhands browser click 'testid=sidebar-onboarding-checklist-toggle'`, `control-openhands browser attr 'testid=sidebar-onboarding-checklist' data-minimized` (`true`) and `browser count 'testid=sidebar-onboarding-checklist >> role=listitem'` (`0`). After `control-openhands browser reload` the attribute is still `true` and the toggle's `aria-label` is `Expand checklist`. Click the toggle again to restore `false`.", "raw": "**Minimize (`F02.checklist-minimize`).** Run `control-openhands browser click 'testid=sidebar-onboarding-checklist-toggle'`, `control-openhands browser attr 'testid=sidebar-onboarding-checklist' data-minimized` (`true`) and `browser count 'testid=sidebar-onboarding-checklist >> role=listitem'` (`0`). After `control-openhands browser reload` the attribute is still `true` and the toggle's `aria-label` is `Expand checklist`. Click the toggle again to restore `false`.", "source_line": 85, "expected": [ "the card header minimizes and expands the card, and the state survives a reload." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--dismiss", "title": "Dismiss", "ids": [ "F02.checklist-dismiss" ], "body": "On `/conversations`, `control-openhands browser count 'role=button[name=\"Dismiss checklist\"]'` is `0` (the X is `opacity-0` and `invisible` until the card is hovered). Run `control-openhands browser hover 'testid=command-menu-trigger'`, `control-openhands browser hover 'testid=sidebar-onboarding-checklist-toggle'` and `control-openhands browser wait 'role=button[name=\"Dismiss checklist\"]' --state visible --timeout 5000` (`browser screenshot 'aside[data-collapsed]' --feature F02.checklist-dismiss --name hover-close`). Run `control-openhands browser click 'role=button[name=\"Dismiss checklist\"]'`: `browser count 'testid=sidebar-onboarding-checklist'` is `0`, still `0` after `control-openhands browser reload`, and `browser storage` lists `openhands-sidebar-onboarding-checklist-dismissed`. On `/settings/app`, `control-openhands browser eval \"document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked\"` is `false`; `control-openhands browser click 'text=Show Getting Started checklist'` brings the card back. Minimized: `control-openhands browser click 'testid=sidebar-onboarding-checklist-toggle'` (`data-minimized` `true`), hover as above, click the X (count `0`), click the switch text again: the card returns with `data-minimized` `true`; click the toggle to expand it. At phone width the X shows in the drawer's card the same way.", "raw": "**Dismiss (`F02.checklist-dismiss`).** On `/conversations`, `control-openhands browser count 'role=button[name=\"Dismiss checklist\"]'` is `0` (the X is `opacity-0` and `invisible` until the card is hovered). Run `control-openhands browser hover 'testid=command-menu-trigger'`, `control-openhands browser hover 'testid=sidebar-onboarding-checklist-toggle'` and `control-openhands browser wait 'role=button[name=\"Dismiss checklist\"]' --state visible --timeout 5000` (`browser screenshot 'aside[data-collapsed]' --feature F02.checklist-dismiss --name hover-close`). Run `control-openhands browser click 'role=button[name=\"Dismiss checklist\"]'`: `browser count 'testid=sidebar-onboarding-checklist'` is `0`, still `0` after `control-openhands browser reload`, and `browser storage` lists `openhands-sidebar-onboarding-checklist-dismissed`. On `/settings/app`, `control-openhands browser eval \"document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked\"` is `false`; `control-openhands browser click 'text=Show Getting Started checklist'` brings the card back. Minimized: `control-openhands browser click 'testid=sidebar-onboarding-checklist-toggle'` (`data-minimized` `true`), hover as above, click the X (count `0`), click the switch text again: the card returns with `data-minimized` `true`; click the toggle to expand it. At phone width the X shows in the drawer's card the same way.", "source_line": 86, "expected": [ "hovering or focusing the card (expanded or minimized) reveals an X (`Dismiss checklist`) in its header; touch devices always show it. Clicking it hides the card for good (it stays hidden after a reload) and turns off the **Show Getting Started checklist** switch in Settings → Application, which brings the card back in its previous expanded or minimized state." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--hide-from-settings", "title": "Hide from settings", "ids": [ "F02.checklist-hide" ], "body": "Run `control-openhands browser goto /settings/app` and `control-openhands browser click 'text=Show Getting Started checklist'`; `browser count 'testid=sidebar-onboarding-checklist'` is `0`. After `control-openhands browser reload` the count is still `0`, and `control-openhands browser eval \"document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked\"` is `false`. Click the same text again; the count is `1`.", "raw": "**Hide from settings (`F02.checklist-hide`).** Run `control-openhands browser goto /settings/app` and `control-openhands browser click 'text=Show Getting Started checklist'`; `browser count 'testid=sidebar-onboarding-checklist'` is `0`. After `control-openhands browser reload` the count is still `0`, and `control-openhands browser eval \"document.querySelector('[data-testid=show-getting-started-checklist-switch]').checked\"` is `false`. Click the same text again; the count is `1`.", "source_line": 87, "expected": [ "the **Show Getting Started checklist** switch in Settings → Application hides the card (it stays hidden after a reload) and can show it again." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--all-complete", "title": "All complete", "ids": [ "F02.checklist-all-complete" ], "body": "Start with the four items above done (`5 complete` would mean one fixture is left over). Arrange two fixtures: a minimal stdio MCP server, `control-openhands fixture file --name qa-f02-mcp.py --content \"$(printf '%s\\n' 'import sys,json' 'for line in sys.stdin:' ' m=json.loads(line)' ' if \"id\" not in m: continue' ' r={}' ' if m.get(\"method\")==\"initialize\": r={\"protocolVersion\":m[\"params\"].get(\"protocolVersion\",\"2024-11-05\"),\"capabilities\":{\"tools\":{}},\"serverInfo\":{\"name\":\"qa\",\"version\":\"1\"}}' ' elif m.get(\"method\")==\"tools/list\": r={\"tools\":[]}' ' sys.stdout.write(json.dumps({\"jsonrpc\":\"2.0\",\"id\":m[\"id\"],\"result\":r})+\"\\n\"); sys.stdout.flush()')\"`, and `control-openhands fixture file --name qa-f02.automation.json --content '{\"version\":1,\"kind\":\"automation\",\"spec\":{\"name\":\"QA_F02 Done\",\"trigger\":{\"type\":\"cron\",\"schedule\":\"0 0 1 1 *\",\"timezone\":\"UTC\"},\"enabled\":false,\"prompt\":\"Reply with the single word: pong. Do not run any tools.\"}}'`. Add the server in the UI: `control-openhands browser goto /mcp`, `control-openhands browser click 'testid=mcp-add-custom-server'`, `control-openhands browser click 'testid=server-type-dropdown'`, `control-openhands browser click 'role=option[name=\"STDIO\"]'`, `control-openhands browser fill 'testid=name-input' qa_f02_mcp`, `control-openhands browser fill 'testid=command-input' python3`, `control-openhands browser fill 'testid=args-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-f02-mcp.py\"`, `control-openhands browser click 'testid=mcp-custom-editor >> testid=submit-button'` and `control-openhands browser wait 'testid=mcp-custom-editor' --state detached`. The header reads `5 complete`. Import the automation: `control-openhands browser goto /automations`, `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-import-automation'`, `control-openhands browser upload 'testid=automations-import-file' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-f02.automation.json\"`, `control-openhands browser click 'testid=import-automation-confirm'` and `control-openhands browser wait 'testid=import-automation-modal' --state detached`. Now `browser count 'testid=sidebar-onboarding-checklist'` is `0`, still `0` after `control-openhands browser reload`, and `control-openhands browser screenshot 'aside[data-collapsed]' --feature F02.checklist-all-complete --name rail` shows the rail without the card. Restore and prove it comes back: open the card's kebab with `control-openhands browser click '[data-testid^=\"automation-card-\"] >> has-text=QA_F02 Done >> role=button[name=\"Automation actions\"]'`, then `control-openhands browser click 'role=list >> has-text=Export >> role=button[name=\"Delete\"]'`, `control-openhands browser click 'role=heading[name=\"Delete automation\"] >> xpath=.. >> role=button[name=\"Delete\"]'` and `control-openhands browser reload`; the card count is `1` again (`5 complete`). Then `control-openhands browser goto /mcp`, `control-openhands browser click 'testid=mcp-server-detail-qa_f02_mcp'`, `control-openhands browser click 'testid=mcp-custom-editor-delete'`, `control-openhands browser click 'testid=confirmation-modal >> testid=confirm-button'` and `control-openhands browser reload`; `testid=mcp-installed-empty` is back and the card reads `4 complete`.", "raw": "**All complete (`F02.checklist-all-complete`).** Start with the four items above done (`5 complete` would mean one fixture is left over). Arrange two fixtures: a minimal stdio MCP server, `control-openhands fixture file --name qa-f02-mcp.py --content \"$(printf '%s\\n' 'import sys,json' 'for line in sys.stdin:' ' m=json.loads(line)' ' if \"id\" not in m: continue' ' r={}' ' if m.get(\"method\")==\"initialize\": r={\"protocolVersion\":m[\"params\"].get(\"protocolVersion\",\"2024-11-05\"),\"capabilities\":{\"tools\":{}},\"serverInfo\":{\"name\":\"qa\",\"version\":\"1\"}}' ' elif m.get(\"method\")==\"tools/list\": r={\"tools\":[]}' ' sys.stdout.write(json.dumps({\"jsonrpc\":\"2.0\",\"id\":m[\"id\"],\"result\":r})+\"\\n\"); sys.stdout.flush()')\"`, and `control-openhands fixture file --name qa-f02.automation.json --content '{\"version\":1,\"kind\":\"automation\",\"spec\":{\"name\":\"QA_F02 Done\",\"trigger\":{\"type\":\"cron\",\"schedule\":\"0 0 1 1 *\",\"timezone\":\"UTC\"},\"enabled\":false,\"prompt\":\"Reply with the single word: pong. Do not run any tools.\"}}'`. Add the server in the UI: `control-openhands browser goto /mcp`, `control-openhands browser click 'testid=mcp-add-custom-server'`, `control-openhands browser click 'testid=server-type-dropdown'`, `control-openhands browser click 'role=option[name=\"STDIO\"]'`, `control-openhands browser fill 'testid=name-input' qa_f02_mcp`, `control-openhands browser fill 'testid=command-input' python3`, `control-openhands browser fill 'testid=args-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-f02-mcp.py\"`, `control-openhands browser click 'testid=mcp-custom-editor >> testid=submit-button'` and `control-openhands browser wait 'testid=mcp-custom-editor' --state detached`. The header reads `5 complete`. Import the automation: `control-openhands browser goto /automations`, `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-import-automation'`, `control-openhands browser upload 'testid=automations-import-file' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-f02.automation.json\"`, `control-openhands browser click 'testid=import-automation-confirm'` and `control-openhands browser wait 'testid=import-automation-modal' --state detached`. Now `browser count 'testid=sidebar-onboarding-checklist'` is `0`, still `0` after `control-openhands browser reload`, and `control-openhands browser screenshot 'aside[data-collapsed]' --feature F02.checklist-all-complete --name rail` shows the rail without the card. Restore and prove it comes back: open the card's kebab with `control-openhands browser click '[data-testid^=\"automation-card-\"] >> has-text=QA_F02 Done >> role=button[name=\"Automation actions\"]'`, then `control-openhands browser click 'role=list >> has-text=Export >> role=button[name=\"Delete\"]'`, `control-openhands browser click 'role=heading[name=\"Delete automation\"] >> xpath=.. >> role=button[name=\"Delete\"]'` and `control-openhands browser reload`; the card count is `1` again (`5 complete`). Then `control-openhands browser goto /mcp`, `control-openhands browser click 'testid=mcp-server-detail-qa_f02_mcp'`, `control-openhands browser click 'testid=mcp-custom-editor-delete'`, `control-openhands browser click 'testid=confirmation-modal >> testid=confirm-button'` and `control-openhands browser reload`; `testid=mcp-installed-empty` is back and the card reads `4 complete`.", "source_line": 88, "expected": [ "the card disappears once all six items are complete. Completion is recomputed from current data, so deleting the automation or the MCP server brings the card back." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--setup-guide", "title": "Setup guide", "ids": [ "F02.super-admin-setup-guide" ], "body": "Local: `control-openhands browser count 'testid=super-admin-setup-guide'` is `0` at `desktop` and `phone`, `browser count 'testid=sidebar-onboarding-checklist'` is `1`, and `control-openhands browser network --filter setup-state` reports `total` `0`. The positive path is blocked locally: it needs an OHE backend with `ENABLE_SUPER_ADMIN` whose `/api/admin/setup-state` returns a `guide_org_id`, `guide_dismissed` `false` and at least one unfinished required step, signed in as the first Super Admin.", "raw": "**Setup guide (`F02.super-admin-setup-guide`).** Local: `control-openhands browser count 'testid=super-admin-setup-guide'` is `0` at `desktop` and `phone`, `browser count 'testid=sidebar-onboarding-checklist'` is `1`, and `control-openhands browser network --filter setup-state` reports `total` `0`. The positive path is blocked locally: it needs an OHE backend with `ENABLE_SUPER_ADMIN` whose `/api/admin/setup-state` returns a `guide_org_id`, `guide_dismissed` `false` and at least one unfinished required step, signed in as the first Super Admin.", "source_line": 89, "expected": [ "on an OpenHands Enterprise (cloud) backend with `ENABLE_SUPER_ADMIN`, the first Super Admin (`/me` permissions include `manage_super_admins`) sees a floating **Setup guide** panel in the lower-right corner with `n/4` progress, the steps Add an LLM, Choose an automation template, Add an integration, Invite users and Optional: SAML / instance, a `Next: ` status and a **Start** link for that step (the positive Enterprise path remains unverified here). Close collapses it to a pill that reopens it. While the guide loads or shows, the **Getting started** card is hidden. Local backends never show it and make no `/api/admin/setup-state` request (#17969)." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F02--setup-tour", "title": "Setup tour", "ids": [ "F02.super-admin-setup-tour" ], "body": "Local, at `desktop` and `phone`: run `control-openhands browser network --clear`, `control-openhands browser goto '/?setup_tour=first-automation'`, wait about 6 s, then `control-openhands browser count '.oh-setup-tour-popover'` and `browser count 'testid=super-admin-setup-guide'` are `0`, `browser count 'testid=home-screen'` is `1`, `browser url` still ends in `?setup_tour=first-automation`, and `browser network --filter setup-state` reports `total` `0`. The tour itself is blocked with the same Enterprise prerequisite as the guide.", "raw": "**Setup tour (`F02.super-admin-setup-tour`).** Local, at `desktop` and `phone`: run `control-openhands browser network --clear`, `control-openhands browser goto '/?setup_tour=first-automation'`, wait about 6 s, then `control-openhands browser count '.oh-setup-tour-popover'` and `browser count 'testid=super-admin-setup-guide'` are `0`, `browser count 'testid=home-screen'` is `1`, `browser url` still ends in `?setup_tour=first-automation`, and `browser network --filter setup-state` reports `total` `0`. The tour itself is blocked with the same Enterprise prerequisite as the guide.", "source_line": 90, "expected": [ "for the two setup steps done in Canvas (Choose an automation template, Add an integration), the guide walks the Super Admin through the page with a guided tour (`.oh-setup-tour-popover` with Back / Next / Done): the template step points at the recommended-automations section (a stop with no footer: no Back, Next or `1/2`, #18243), then the setup dialog; the integration step at the MCP library on `/mcp` (`mcp-marketplace-section`, \"Pick a server from the library\"), then the install dialog (#18261). A plain click on a Canvas step's row in the guide starts that step's tour and closes the panel (a modifier click follows the link), and while the tour waits for its next stop's element it hides its popover and overlay, so it never covers another dialog (#18261). Finishing the guide's next step in Canvas opens the step after it the way **Start** does (a Canvas step's tour or page; an enterprise step reopens the guide to point at Start, or navigates on a locked-Cloud canvas). An enterprise page that links back with `?setup_tour=` starts that step's tour once and drops the parameter. Local backends ignore `?setup_tour=` (no tour, no guide, the parameter stays) (#18138). The positive path is blocked here, with the same Enterprise prerequisite as the setup guide above." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F02--update-tile", "title": "Update tile", "ids": [ "F02.update-tile" ], "body": "The tile is blocked in an up-to-date checkout or without registry access. Check with `control-openhands browser count 'testid=agent-canvas-version-tile'`. When the count is `1`, click it; the modal has `agent-canvas-update-command-npm`, `agent-canvas-update-command-docker` and `agent-canvas-update-check-button`. The manual check runs either way: on `/settings/app` (`control-openhands browser goto /settings/app`; the previous bullet ends on `/mcp`), `testid=agent-canvas-update-badge` reads `Up to date` (or `New version`), `control-openhands browser click 'testid=agent-canvas-update-toggle'` opens `agent-canvas-update-modal`, and `control-openhands browser click 'testid=agent-canvas-update-check-button'` reports the result in `agent-canvas-update-status` (`You're running the latest version. What's new` when current). Close it with `control-openhands browser click 'testid=close-agent-canvas-update-modal'`.", "raw": "**Update tile (`F02.update-tile`).** The tile is blocked in an up-to-date checkout or without registry access. Check with `control-openhands browser count 'testid=agent-canvas-version-tile'`. When the count is `1`, click it; the modal has `agent-canvas-update-command-npm`, `agent-canvas-update-command-docker` and `agent-canvas-update-check-button`. The manual check runs either way: on `/settings/app` (`control-openhands browser goto /settings/app`; the previous bullet ends on `/mcp`), `testid=agent-canvas-update-badge` reads `Up to date` (or `New version`), `control-openhands browser click 'testid=agent-canvas-update-toggle'` opens `agent-canvas-update-modal`, and `control-openhands browser click 'testid=agent-canvas-update-check-button'` reports the result in `agent-canvas-update-status` (`You're running the latest version. What's new` when current). Close it with `control-openhands browser click 'testid=close-agent-canvas-update-modal'`.", "source_line": 91, "expected": [ "when npm `latest` is newer than the running version, a **New version** tile (`agent-canvas-version-tile`) opens an update modal with the npm and Docker commands. The tile is never shown in locked-Cloud mode." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--error-toast", "title": "Error toast", "ids": [ "F02.error-toasts" ], "body": "Run `control-openhands browser goto /conversations/00000000-0000-0000-0000-000000000001` and `control-openhands browser wait-text 'This conversation does not exist'`. The URL becomes `/conversations`, and `control-openhands browser screenshot --feature F02.error-toasts --name missing-conversation` shows a top-right error toast. It is the only toast: `control-openhands browser toasts` lists just the \"This conversation does not exist, or you do not have permission to access it…\" text, and `control-openhands browser toasts --history` has no entry containing `this UI does not understand`. `control-openhands browser wait 'text=This conversation does not exist' --state hidden` passes within 30 s. Light theme: arrange `control-openhands browser eval \"localStorage.setItem('openhands-color-theme','light-plus')\"`, repeat the `goto`, and read `control-openhands browser eval \"(()=>{const t=[...document.querySelectorAll('[role=status]')].find(e=>e.textContent.includes('does not exist'));return getComputedStyle(t).color+' on '+getComputedStyle(t.closest('div[style]')).backgroundColor})()\"`: `rgb(31, 31, 31) on rgb(229, 229, 229)`: every toast uses the theme's high-emphasis ink (`--oh-contrast`) on its surface (the default theme reads `rgb(255, 255, 255) on rgb(49, 49, 49)`). A success toast reads the same: save any change on `/settings/app` (for example `control-openhands browser click 'testid=app-settings-screen >> text=Sound Notifications'`, `control-openhands browser click 'testid=submit-button'`, then toggle it back and save again) and read the `Settings saved` toast the same way: `rgb(31, 31, 31) on rgb(229, 229, 229)`. Solarized Light (`solarized-light`) gives `rgb(0, 43, 54) on rgb(230, 223, 204)` for both toasts. Restore with `browser eval \"localStorage.removeItem('openhands-color-theme')\"`.", "raw": "**Error toast (`F02.error-toasts`).** Run `control-openhands browser goto /conversations/00000000-0000-0000-0000-000000000001` and `control-openhands browser wait-text 'This conversation does not exist'`. The URL becomes `/conversations`, and `control-openhands browser screenshot --feature F02.error-toasts --name missing-conversation` shows a top-right error toast. It is the only toast: `control-openhands browser toasts` lists just the \"This conversation does not exist, or you do not have permission to access it…\" text, and `control-openhands browser toasts --history` has no entry containing `this UI does not understand`. `control-openhands browser wait 'text=This conversation does not exist' --state hidden` passes within 30 s. Light theme: arrange `control-openhands browser eval \"localStorage.setItem('openhands-color-theme','light-plus')\"`, repeat the `goto`, and read `control-openhands browser eval \"(()=>{const t=[...document.querySelectorAll('[role=status]')].find(e=>e.textContent.includes('does not exist'));return getComputedStyle(t).color+' on '+getComputedStyle(t.closest('div[style]')).backgroundColor})()\"`: `rgb(31, 31, 31) on rgb(229, 229, 229)`: every toast uses the theme's high-emphasis ink (`--oh-contrast`) on its surface (the default theme reads `rgb(255, 255, 255) on rgb(49, 49, 49)`). A success toast reads the same: save any change on `/settings/app` (for example `control-openhands browser click 'testid=app-settings-screen >> text=Sound Notifications'`, `control-openhands browser click 'testid=submit-button'`, then toggle it back and save again) and read the `Settings saved` toast the same way: `rgb(31, 31, 31) on rgb(229, 229, 229)`. Solarized Light (`solarized-light`) gives `rgb(0, 43, 54) on rgb(230, 223, 204)` for both toasts. Restore with `browser eval \"localStorage.removeItem('openhands-color-theme')\"`.", "source_line": 92, "expected": [ "failed requests show a top-right error toast with the product or server message, and the toast closes by itself. Toasts stay readable in every colour theme." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F02--check-for-errors", "title": "Check for errors", "ids": [], "body": "Run `control-openhands browser errors --app-only` after each group; it reports `pageErrors` `0` and `appErrors` `0`. External `registry.npmjs.org` and `cdn.simpleicons.org` failures are sandbox noise.", "raw": "**Check for errors.** Run `control-openhands browser errors --app-only` after each group; it reports `pageErrors` `0` and `appErrors` `0`. External `registry.npmjs.org` and `cdn.simpleicons.org` failures are sandbox noise.", "source_line": 93, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F02--restore", "title": "Restore", "ids": [], "body": "Leave the sidebar expanded, unpinned, with the checklist shown and expanded, at `browser viewport desktop`.", "raw": "**Restore.** Leave the sidebar expanded, unpinned, with the checklist shown and expanded, at `browser viewport desktop`.", "source_line": 94, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F02--existing-conversations-while-collapsed", "title": "Existing conversations while collapsed", "ids": [ "F02.sidebar-collapse" ], "body": "With at least one sidebar conversation, run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, `control-openhands browser snapshot` and `control-openhands browser count 'testid=conversation-panel'`. Known failure (reproduced 2026-10-12, [#15662](https://github.com/OpenHands/OpenHands/issues/15662)): the count is `0`, and the rail has no chooser for existing conversations; expansion is required. Restore with `control-openhands browser hover 'role=link[name=\"OpenHands Logo\"]'` and `control-openhands browser click 'testid=sidebar-collapse-toggle'`.", "raw": "**Existing conversations while collapsed (`F02.sidebar-collapse`).** With at least one sidebar conversation, run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, `control-openhands browser snapshot` and `control-openhands browser count 'testid=conversation-panel'`. Known failure (reproduced 2026-10-12, [#15662](https://github.com/OpenHands/OpenHands/issues/15662)): the count is `0`, and the rail has no chooser for existing conversations; expansion is required. Restore with `control-openhands browser hover 'role=link[name=\"OpenHands Logo\"]'` and `control-openhands browser click 'testid=sidebar-collapse-toggle'`.", "source_line": 96, "expected": [ "the chevron collapses the rail to a 60 px icon rail with hover tooltips, a Settings icon and a backend icon with a status dot that opens a backend popover. The checklist is hidden while collapsed. The state survives a reload. To expand, click the chevron that appears when you hover the logo, or click empty rail space." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [#15662](https://github.com/OpenHands/OpenHands/issues/15662)): the count is `0`, and the rail has no chooser for existing conversations; expansion is required. Restore with `control-openhands browser hover 'role=link[name=\"OpenHands Logo\"]'` and `control-openhands browser click 'testid=sidebar-collapse-toggle'`." ], "states": [ "failure" ] }, { "name": "F02--contract-f02-alert-banner", "title": "Alert banner", "ids": [ "F02.alert-banner" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "when the server config reports maintenance, faulty models or an error message, a dismissible banner (`alert-banner`) appears at the top of the content area." ], "notes": [], "annotations": [], "states": [ "contract" ] }, { "name": "F02--contract-f02-settings-404-modal", "title": "Settings 404 modal", "ids": [ "F02.settings-404-modal" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "if the backend's settings return 404, the LLM settings modal opens by itself. Other settings fetch errors show a toast." ], "notes": [], "annotations": [], "states": [ "contract" ] } ] }, { "id": "F03", "title": "Home and starting work", "file": "F03-home.md", "sha256": "f5a3aca0a9b9af8d753633c5592bd2cbea087d98c87d222dc5776ef9e1cc28f2", "overview": "Home is the \"What do you want to work on?\" screen at `/` and `/conversations`.\nA **Code / Automate** toggle above the composer picks what a submit starts: a\ncoding conversation, or an automation setup conversation where the agent fills\nan automation form. A user describes a task in the composer and, on a local\nbackend, can first pick a workspace folder (Open Workspace, folder browser, Manage Workspaces), a\nworkspace mode and plugins; submitting creates a conversation and opens it. A\nwarning banner replaces a silent failure when no LLM is configured. Below the\ncomposer, home offers recommended automations, a pinned-automations grid and an\nactivity list of enabled automations with per-item actions.", "entry": "- Sidebar **New Chat** (`sidebar-conversations-link`), which opens `/conversations`; the logo does the same (see F02).\n- Direct URL `/` (unless a page is pinned as home, see F02) or `/conversations`.\n- Command menu (`Control+k`, `Meta+k` or `command-menu-trigger`): search \"New chat\", choose **New chat**.\n- Above the composer: the **Code** / **Automate** toggle (`home-launcher-mode-code`, `home-launcher-mode-automate`).\n- Under the composer: **Open Workspace** (`open-workspace-button`) or, once confirmed, the preview bar (`home-git-control-bar-preview`); **Plugins** (`open-plugin-picker`).\n- In the Open Workspace dialog: the combobox (`workspace-dropdown`) and its footer **+ Add Workspace** (`add-workspaces-button`) and **Manage Workspaces** (`manage-workspaces-button`, shown once any workspace or parent exists); in **Add workspaces**, the Favorites/Locations sidebar (`folder-browser-sidebar-`).\n- Below the composer: recommended cards (`recommended-automation-rail-card-`), pinned cards (`pinned-automation-card-`) and rows (`running-automation-row-`) with their kebabs; **+ Add** (`home-automations-manage`).", "given": "Preconditions:\n\n- `F03.llm-not-configured-banner` and `F03.composer-blocked-without-llm` need a fresh `control-openhands launch --new` with only `control-openhands onboard --skip` done (no `llm preset` yet). Run them first.\n- Everything else: baseline state with `control-openhands llm preset deepseek` (deepseek-flash active).\n- Workspace fixtures: `control-openhands fixture git-repo --name qa-f03-repo` and `control-openhands fixture folder --name qa-f03-folder` (both under `/workspace`).\n- Automation fixtures (arranged through F22 import and F21 Turn on, not proof): `control-openhands fixture file --name qa-f03-pong.automation.json --content '{\"version\":1,\"kind\":\"automation\",\"spec\":{\"name\":\"QA_F03 Pong\",\"trigger\":{\"type\":\"cron\",\"schedule\":\"0 0 1 1 *\",\"timezone\":\"UTC\"},\"enabled\":true,\"prompt\":\"Reply with the single word: pong. Do not run any tools.\"}}'` and the same with `qa-f03-beta` / `QA_F03 Beta`. Run `control-openhands browser goto /automations`, then for each file run `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-import-automation'`, `control-openhands browser upload 'testid=automations-import-file' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-f03-pong.automation.json\"`, `control-openhands browser click 'testid=import-automation-confirm'`, `control-openhands browser wait 'testid=import-automation-modal' --state detached`. Imports arrive disabled: turn each on with `control-openhands browser click '[data-testid^=\"automation-card-\"] >> has-text=QA_F03 Pong >> role=button[name=\"Automation actions\"]'` and `control-openhands browser click 'role=list >> has-text=Export >> role=button[name=\"Turn on\"]'`. The yearly cron never fires by itself.\n- Blocked here: `F03.open-repository-dialog` (OpenHands Cloud backend with a connected GitHub/GitLab/Bitbucket provider), `F03.isolated-workspace-notice` (isolated-runtime backend), `F03.workspaces-unsupported` (an Agent Server without `/api/workspaces`), `F03.recommended-prompt-launch` (a Tavily/Linear/Jira integration installed so some card has nothing missing).\n\nStarting work:", "gotchas": "- Once a workspace is confirmed, `open-workspace-button` is not rendered (the preview bar replaces it): a click on it waits the full 30 s. Reopen through `home-git-control-bar-preview`, or reload.\n- A pick in the workspace dropdown is stored in sessionStorage at once, before Confirm or Launch, and closing the dialog with the X keeps it: a bullet that needs the earlier workspace selected must pick it again instead of relying on the last confirmation.\n- `control-openhands workspace open` and `conversation start --workspace` save that folder as a standalone workspace (`api GET /api/workspaces` lists it under `workspaces`). When another family used them first on the same stack, Add all subdirectories leaves those folders out of the `workspace` parent and the dropdown is grouped: options read `workspace, qa-f03-repo` and `Other, qa-f18-repo`. `role=option[name=\"qa-f03-repo\"]` still matches (substring), but the menu text and the `workspace-group-header` count `2` in the Search bullet assume F03's own fixtures only.\n- Folder rows navigate on a single click; there is no path field. The browser starts at the server home, `/private/home`, two levels below ``. After any conversation without a workspace, `/private/home/workspace` exists: it shows as a `workspace` row and, after the next page load, a `workspace` favorite (`folder-browser-sidebar-workspace`), which is not `/workspace`.\n- A workspace and its parent can share a name (`workspace`); their test ids differ only by prefix (`manage-workspaces-row-` vs `manage-workspaces-parent-row-`).\n- `chat-input` is a contenteditable `div`: it stays editable when the composer is \"disabled\"; assert the Send button and the absence of a new conversation instead.\n- `conversation start` without `--stay` navigates home and loses a picked workspace and plugins; drive the submit button as above.\n- Responder **Continue with local setup** writes an `OPENHANDS_URL` secret; delete it so F14 counts stay clean.\n- The rail lists about 15 cards (proven ones, then conversation launchers such as `research-brief-writer`); `browser testids` reports only the first few, so check a card with `browser count 'testid=recommended-automation-rail-card-'`. Any automation whose name matches a card (`Daily news digest`) hides that card, so avoid catalog names in fixtures.\n- The home draft (`oh:home-prompt-draft`) survives the no-LLM bullets; `browser type` appends to it, so `browser fill 'testid=chat-input' ''` before typing a slash command.\n- Imported automations arrive disabled and home lists only enabled ones; turn them on before looking for rows. A run dispatched from home and then cancelled still leaves a `QA_F03 Pong — ` conversation in the sidebar.\n- With the Agent Server stopped, a reload does not render home at all; trigger the error path before reloading, then `restart`.\n- Deleting the last automation leaves its id in the pinned localStorage key (pruning skips an empty list); the grid still hides it.\n- Pinned cards reorder only by mouse drag; there is no keyboard or menu alternative. Drive it with `browser drag --by DX,DY --steps 20`; `browser drag ` reports ok but leaves the order unchanged.\n- `VITE_HOME_AUTOMATIONS_DEMO=true` (build-time) swaps the home automation sections for demo data; never verify against such a build.\n- The Code / Automate choice is component state, not stored: every page load and every return to home starts in Code. A toggle check after a navigation must click Automate again.\n- When the Agent Server is stopped, background refetches can add a generic `An error occurred` toast (#18051). For `F03.create-error-toast`, retain the complete video and toast history and report any extra toast with its timing; do not waive it from the count.\n- The workspace combobox uses one message for an empty list and for a search with no match: both read `No workspaces yet` (`workspace-dropdown-empty`), with **+ Add Workspace** still below it (#18037).\n- Known issue [#17108](https://github.com/OpenHands/OpenHands/issues/17108) (`F03.open-repository-dialog`): The repository picker lists a search result for input the user has already cleared. Live prerequisite: An authenticated Cloud Git provider and a real repository search response that remains pending after clearing the input.\n- Known issue [#16801](https://github.com/OpenHands/OpenHands/issues/16801) (`F03.launch-without-workspace`): Agent starts by looking in workspace, even when started with \"no workspace\". Live prerequisite: A working Claude Opus agent and the original task/prompt that makes it explore the default workspace; the prompt is absent.\n- Known issue [#16210](https://github.com/OpenHands/OpenHands/issues/16210) (`F03.plugin-picker`): [Cloud Backends] Plugins not available when starting a new conversation. Live prerequisite: An authenticated Cloud organization with configured account plugins before opening New Chat > Plugins.\n- Known issue [#15701](https://github.com/OpenHands/OpenHands/issues/15701) (`F03.launch-in-workspace`): Creating Conversation toast stuck visible. Live prerequisite: The original stuck creation request/outcome sequence; the report is intermittent and gives no deterministic trigger.\n- Known issue [#18480](https://github.com/OpenHands/OpenHands/issues/18480) (`F03.open-repository-dialog`): an Azure DevOps Enterprise user belonging to several organizations can see only one organization's repositories. Live prerequisite: a real Cloud/Enterprise sign-in with two organizations and distinct repositories; the report's mocked two-organization capture is not a real-backend reproduction.\n- Known issue [#18418](https://github.com/OpenHands/OpenHands/issues/18418) (`F03.create-error-toast`, `F03.launch-with-plugin`): hosted runtime 1.50.1 can fail during skills/plugins setup because its packaged plugin-schema resource is missing. Live prerequisite: the affected Cloud runtime/PyInstaller image and original plugin setup plus its account. A Local 1.55.0 launch does not verify that deployment.\n- Known issue [OpenHands/software-agent-sdk#5837](https://github.com/OpenHands/software-agent-sdk/issues/5837) (`F03.folder-browser`): case-only directory names at a page boundary can repeat the server cursor; Canvas detects the repeated cursor and the directory listing fails instead of showing all choices. Live prerequisite: a case-sensitive filesystem with enough directories to cross the real picker page boundary, including two distinct case-only names. A case-insensitive Mac volume cannot hold that fixture.\n- Known issue [OpenHands/software-agent-sdk#5849](https://github.com/OpenHands/software-agent-sdk/issues/5849) (`F03.folder-browser`, `F03.manage-workspaces`): re-adding a saved workspace or removing an already-removed stale row can return500 because the registry writes even for a no-op. Live prerequisite: an owned read-only workspace registry plus the already-saved or stale picker row; changes to registry mtime/inode alone are not a Canvas failure.", "features": [ { "id": "F03.home-entry-points", "description": "`/`, `/conversations`, the sidebar **New Chat** and the command-menu item **New chat** show the launcher (`home-chat-launcher`) with the heading, subtitle, the Code / Automate toggle and the composer, whose Code-mode placeholder reads `What do you want to build?`." }, { "id": "F03.llm-not-configured-banner", "description": "with no usable LLM, a `role=alert` banner reads `Your LLM isn't set up yet, so conversations won't run. Finish setup to get started.`; **Set up LLM** opens `/settings/llm`; the banner is gone once a profile with a key is active." }, { "id": "F03.composer-blocked-without-llm", "description": "with no LLM, Send is disabled and Enter creates nothing (the text stays editable)." }, { "id": "F03.launcher-mode-toggle", "description": "a `Choose launcher mode` radio group above the composer offers **Code** (checked on every page load) and **Automate**. Automate changes the placeholder to `What work do you want to automate?` and greys out and disables the Open Workspace and Plugins row (`home-composer-actions` `aria-disabled=\"true\"`); Code restores both. The active option stays readable in every colour theme." }, { "id": "F03.launch-automate-mode", "description": "submitting in Automate mode creates a conversation and opens `/conversations/` as an automation setup split view: the chat on the left and the automation form (`automation-setup-panel`) on the right, whose fields the agent fills with `✦ Filled by OpenHands` markers. Attached plugins are not sent, and nothing is created until **Create Automation**. The setup view itself is F22's." }, { "id": "F03.draft-persistence", "description": "the home draft survives a reload and is cleared after a conversation is created." }, { "id": "F03.home-model-command", "description": "`/model ` sent from home activates that LLM profile without creating a conversation." }, { "id": "F03.launch-in-workspace", "description": "submitting with a workspace picked shows `Creating conversation…`, opens `/conversations/`, and the agent runs in that folder." }, { "id": "F03.launch-without-workspace", "description": "with nothing picked, the conversation runs in a fresh per-conversation folder under the server home." }, { "id": "F03.create-error-toast", "description": "when creation fails, the loading toast is replaced by one error toast and the prompt is not lost." }, { "id": "F03.plugin-picker", "description": "**Plugins** opens **Attach plugins** (catalog, search, Add/Remove switch, `N selected`, Done and X); the pill then shows a count badge." }, { "id": "F03.launch-with-plugin", "description": "a conversation started with a plugin attached loads it; the selection resets afterwards. Workspaces (local backend):" }, { "id": "F03.open-workspace-dialog", "description": "**Open Workspace** opens a dialog with a workspace combobox; Confirm stays disabled until a pick; confirming swaps the pill for a preview bar that reopens the dialog; X and Escape close it." }, { "id": "F03.workspace-dropdown-search", "description": "typing filters workspaces; with more than one group, workspaces are grouped by parent with an **Other** group; no match (or no workspaces) shows `No workspaces yet`." }, { "id": "F03.selection-after-reload", "description": "after a reload the preview bar is gone, but the dialog preselects the last workspace picked in its dropdown, confirmed or not: a pick closed with the X survives leaving for another page and coming back, and a reload." }, { "id": "F03.workspace-mode", "description": "the preview bar's mode menu switches **Local Repo** / **New Worktree**; the choice persists across reloads." }, { "id": "F03.folder-browser", "description": "**+ Add Workspace** opens **Add workspaces** at the server home with Favorites/Locations, Up and the current path; **Add this directory** adds and selects the folder, **Add all subdirectories** adds a parent whose children become workspaces, **Cancel** adds nothing." }, { "id": "F03.folder-browser-locations", "description": "the browser's **Favorites** (Home, plus `workspace` once a no-folder conversation exists) and **Locations** (`/`) jump to that folder; Up is disabled at `/`." }, { "id": "F03.manage-workspaces", "description": "**Manage Workspaces** lists workspaces and parents (with children); every Remove asks for confirmation; Cancel keeps, Confirm removes for good and clears the selection; with nothing left it reads `You don't have any workspaces yet.`" }, { "id": "F03.open-repository-dialog", "description": "on a Cloud backend, **Open repository** picks provider, repository and branch (blocked locally)." }, { "id": "F03.isolated-workspace-notice", "description": "on an isolated-runtime backend a status line explains the new isolated workspace and offers **Clear host workspace** (blocked locally)." }, { "id": "F03.workspaces-unsupported", "description": "on an Agent Server without workspace APIs the pill is disabled with a tooltip (blocked locally). Automations on home:" }, { "id": "F03.recommended-automations-rail", "description": "a horizontally scrolling rail of recommended automations with edge fades; cards with a setup form open `/automations/new/`." }, { "id": "F03.recommended-responder-choice", "description": "GitHub/Slack responder cards on a local backend first ask **Choose how your responder should run**." }, { "id": "F03.responder-cloud-option", "description": "in that dialog **Open OpenHands Cloud integrations** opens `https://app.all-hands.dev/settings/integrations` in a new tab, closes the dialog and saves no secret; X closes it without a choice." }, { "id": "F03.recommended-missing-integration", "description": "a card whose required integration is missing opens that integration's install dialog first." }, { "id": "F03.recommended-hides-added", "description": "a catalog automation that already exists (matched by name, enabled or not) drops off the rail; deleting it brings the card back." }, { "id": "F03.recommended-prompt-launch", "description": "a card with no setup form and no missing integration creates a conversation that sends the automation prompt (blocked locally)." }, { "id": "F03.home-automations-list", "description": "**Automations** lists enabled automations with schedule, last run and status, a sparkline, **+ Add**, and `No automations yet. Add one to automate repetitive work.` when empty." }, { "id": "F03.home-automation-links", "description": "a row opens `/automations/`, a sparkline bar opens `/automations/?run=`, **+ Add** opens `/automations`." }, { "id": "F03.home-run-tooltip", "description": "hovering a row shows its trigger, status, last run and task." }, { "id": "F03.home-run-outcome-health", "description": "an automation whose latest Completed run reported a task outcome shows it on Home (#17682): `blocked` and `partial_success` read **Blocked** and **Partial**, `unknown` or an unrecognised status reads **Needs review**, each with an amber (`bg-warning`) health dot, in the row text, the row link's aria-label, the tooltip's Status line and the pinned card's strip. A finish response without a string status counts as success." }, { "id": "F03.home-row-menu", "description": "the row kebab offers Run now, View, Edit, Turn off and Pin to dashboard; Run now toasts `Automation dispatched` and is disabled while a run is in flight; Edit opens the automation setup page `/automations/setup?automationId=` with the form prefilled (no modal), where **Save** toasts `Automation updated`." }, { "id": "F03.cancel-in-flight-run", "description": "**Cancel run** appears while a run is pending or running and cancels it (`Automation run cancelled`)." }, { "id": "F03.turn-off-confirmation", "description": "Turn off asks `Turn off \"\"? It will stop running on its schedule until you turn it back on.`; Cancel keeps it on, Turn off disables it and drops it from home." }, { "id": "F03.pinned-automations-grid", "description": "**Pin to dashboard** adds a card to **Pinned automations** (prompt, pills, status strip, Runs); the title opens the detail page; a deleted automation's card disappears." }, { "id": "F03.pinned-card-menu", "description": "a pinned card's kebab has the same actions with **Unpin from dashboard**." }, { "id": "F03.pinned-card-links", "description": "a pinned card's sparkline bar opens `/automations/?run=`, and Enter on its focused title opens the detail page." }, { "id": "F03.pinned-reorder-drag", "description": "dragging a pinned card's title row (`Reorder `) onto another card reorders the pins, and the order survives a reload." }, { "id": "F03.home-automations-view-more", "description": "past 10 rows **View more**/**View all**, past 6 pins **View more**/**View less** (not arranged)." }, { "id": "F03.automations-backend-down", "description": "with the automation service down, the installed list and pinned grid are hidden and the composer still works. The recommendation rail returns after loading from its static catalog; opening a setup card reaches its route and pre-setup dialog. The rail asks for the installed automations only once `/api/automation/health` answers ok (#17838), so home raises no error toast for the stopped service." }, { "id": "F03.phone", "description": "at 390 px the banner, launcher, rail, pinned grid, list and the workspace dialogs fit without horizontal overflow, and folder names stay readable." } ], "scenarios": [ { "name": "F03--no-llm-banner", "title": "No-LLM banner", "ids": [ "F03.llm-not-configured-banner" ], "body": "On a fresh run after `onboard --skip`, run `control-openhands browser text 'testid=home-llm-not-configured-banner'`: `Your LLM isn't set up yet, so conversations won't run. Finish setup to get started.` and `Set up LLM`; `control-openhands browser screenshot --feature F03.llm-not-configured-banner --name banner`. At phone width (`control-openhands browser viewport phone`), `control-openhands browser bbox 'testid=home-llm-not-configured-banner'` is inside the viewport (`F03.phone`; the banner exists only before `llm preset`); return with `control-openhands browser viewport desktop`. `control-openhands browser click 'testid=home-llm-not-configured-action' --expect-url '/settings/llm'` lands on `/settings/llm`. Run the blocked-composer bullet and the draft bullet next (they also need no LLM); only then `control-openhands llm preset deepseek`, `control-openhands browser goto /` and `control-openhands browser count 'testid=home-llm-not-configured-banner'` is `0`.", "raw": "**No-LLM banner (`F03.llm-not-configured-banner`).** On a fresh run after `onboard --skip`, run `control-openhands browser text 'testid=home-llm-not-configured-banner'`: `Your LLM isn't set up yet, so conversations won't run. Finish setup to get started.` and `Set up LLM`; `control-openhands browser screenshot --feature F03.llm-not-configured-banner --name banner`. At phone width (`control-openhands browser viewport phone`), `control-openhands browser bbox 'testid=home-llm-not-configured-banner'` is inside the viewport (`F03.phone`; the banner exists only before `llm preset`); return with `control-openhands browser viewport desktop`. `control-openhands browser click 'testid=home-llm-not-configured-action' --expect-url '/settings/llm'` lands on `/settings/llm`. Run the blocked-composer bullet and the draft bullet next (they also need no LLM); only then `control-openhands llm preset deepseek`, `control-openhands browser goto /` and `control-openhands browser count 'testid=home-llm-not-configured-banner'` is `0`.", "source_line": 90, "expected": [ "with no usable LLM, a `role=alert` banner reads `Your LLM isn't set up yet, so conversations won't run. Finish setup to get started.`; **Set up LLM** opens `/settings/llm`; the banner is gone once a profile with a key is active." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--blocked-composer", "title": "Blocked composer", "ids": [ "F03.composer-blocked-without-llm" ], "body": "Still without an LLM, run `control-openhands browser goto /` (the banner's action left `/settings/llm`), `control-openhands browser type 'testid=chat-input' 'QA no llm'`, `control-openhands browser enabled 'testid=home-chat-launcher >> testid=submit-button'` (`false`), `control-openhands browser press Enter --selector 'testid=chat-input'`, then `control-openhands browser url` (still `/`) and `control-openhands api GET '/api/conversations/search?limit=5'` (`items` is empty).", "raw": "**Blocked composer (`F03.composer-blocked-without-llm`).** Still without an LLM, run `control-openhands browser goto /` (the banner's action left `/settings/llm`), `control-openhands browser type 'testid=chat-input' 'QA no llm'`, `control-openhands browser enabled 'testid=home-chat-launcher >> testid=submit-button'` (`false`), `control-openhands browser press Enter --selector 'testid=chat-input'`, then `control-openhands browser url` (still `/`) and `control-openhands api GET '/api/conversations/search?limit=5'` (`items` is empty).", "source_line": 91, "expected": [ "with no LLM, Send is disabled and Enter creates nothing (the text stays editable)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--draft", "title": "Draft", "ids": [ "F03.draft-persistence" ], "body": "With text in the home composer, `control-openhands browser reload` and `control-openhands browser text 'testid=chat-input'` returns the same text; `control-openhands browser storage --session` lists `oh:home-prompt-draft`. That the draft clears after a successful launch is checked in the first launch (Launch without a folder).", "raw": "**Draft (`F03.draft-persistence`).** With text in the home composer, `control-openhands browser reload` and `control-openhands browser text 'testid=chat-input'` returns the same text; `control-openhands browser storage --session` lists `oh:home-prompt-draft`. That the draft clears after a successful launch is checked in the first launch (Launch without a folder).", "source_line": 92, "expected": [ "the home draft survives a reload and is cleared after a conversation is created." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--entry-points", "title": "Entry points", "ids": [ "F03.home-entry-points" ], "body": "`control-openhands browser goto /conversations` and `control-openhands browser count 'testid=home-chat-launcher'` is `1` (same for `/`). From `/settings/secrets` (`control-openhands browser goto /settings/secrets`), `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' 'New chat'`, `control-openhands browser snapshot 'testid=command-menu'` (one option `New chat Start a fresh OpenHands conversation. Go` [selected]), `control-openhands browser press Enter`, `control-openhands browser url`: `/conversations`. From `/settings/secrets` (`control-openhands browser goto /settings/secrets` again), `control-openhands browser click 'testid=sidebar-conversations-link' --expect-url '/conversations(\\?|$)'` and `browser count 'testid=home-chat-launcher'` is `1`.", "raw": "**Entry points (`F03.home-entry-points`).** `control-openhands browser goto /conversations` and `control-openhands browser count 'testid=home-chat-launcher'` is `1` (same for `/`). From `/settings/secrets` (`control-openhands browser goto /settings/secrets`), `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' 'New chat'`, `control-openhands browser snapshot 'testid=command-menu'` (one option `New chat Start a fresh OpenHands conversation. Go` [selected]), `control-openhands browser press Enter`, `control-openhands browser url`: `/conversations`. From `/settings/secrets` (`control-openhands browser goto /settings/secrets` again), `control-openhands browser click 'testid=sidebar-conversations-link' --expect-url '/conversations(\\?|$)'` and `browser count 'testid=home-chat-launcher'` is `1`.", "source_line": 93, "expected": [ "`/`, `/conversations`, the sidebar **New Chat** and the command-menu item **New chat** show the launcher (`home-chat-launcher`) with the heading, subtitle, the Code / Automate toggle and the composer, whose Code-mode placeholder reads `What do you want to build?`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--model-on-home", "title": "`/model` on home", "ids": [ "F03.home-model-command" ], "body": "`control-openhands api GET /api/profiles` shows `active_profile` `deepseek-flash`. Run `control-openhands browser goto /` (Entry points ended on `/conversations`). The draft from the earlier bullets is still in the composer: clear it with `control-openhands browser fill 'testid=chat-input' ''` (typing would append). Run `control-openhands browser type 'testid=chat-input' '/model deepseek-pro'`, `control-openhands browser press Escape --selector 'testid=chat-input'` (closes the slash menu), `control-openhands browser press Enter --selector 'testid=chat-input'`. The URL stays `/`, `chat-input` is empty, the composer's model pill (`browser text 'testid=chat-input-llm-profile'`) reads `deepseek-pro`, `api GET /api/profiles` shows `active_profile` `deepseek-pro`, and `api GET '/api/conversations/search?limit=20'` lists no new conversation. Restore with the same steps and `/model deepseek-flash`.", "raw": "**`/model` on home (`F03.home-model-command`).** `control-openhands api GET /api/profiles` shows `active_profile` `deepseek-flash`. Run `control-openhands browser goto /` (Entry points ended on `/conversations`). The draft from the earlier bullets is still in the composer: clear it with `control-openhands browser fill 'testid=chat-input' ''` (typing would append). Run `control-openhands browser type 'testid=chat-input' '/model deepseek-pro'`, `control-openhands browser press Escape --selector 'testid=chat-input'` (closes the slash menu), `control-openhands browser press Enter --selector 'testid=chat-input'`. The URL stays `/`, `chat-input` is empty, the composer's model pill (`browser text 'testid=chat-input-llm-profile'`) reads `deepseek-pro`, `api GET /api/profiles` shows `active_profile` `deepseek-pro`, and `api GET '/api/conversations/search?limit=20'` lists no new conversation. Restore with the same steps and `/model deepseek-flash`.", "source_line": 94, "expected": [ "`/model ` sent from home activates that LLM profile without creating a conversation." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--launch-without-a-folder", "title": "Launch without a folder", "ids": [ "F03.launch-without-workspace", "F03.draft-persistence" ], "body": "Run `control-openhands browser goto /conversations` (no workspace is picked yet; the pill reads Open Workspace), then fill `chat-input` with `Reply with only the word ok.` and click `testid=home-chat-launcher >> testid=submit-button` with `--expect-url '/conversations/[^/]+$'`. `control-openhands conversation wait --timeout 180` reports `workspace` `/private/home/workspace/project/`. This is the first successful launch: `control-openhands browser click 'testid=sidebar-conversations-link' --expect-url '/conversations$'` shows an empty `chat-input` and `browser storage --session` no longer lists `oh:home-prompt-draft`.", "raw": "**Launch without a folder (`F03.launch-without-workspace`, `F03.draft-persistence`).** Run `control-openhands browser goto /conversations` (no workspace is picked yet; the pill reads Open Workspace), then fill `chat-input` with `Reply with only the word ok.` and click `testid=home-chat-launcher >> testid=submit-button` with `--expect-url '/conversations/[^/]+$'`. `control-openhands conversation wait --timeout 180` reports `workspace` `/private/home/workspace/project/`. This is the first successful launch: `control-openhands browser click 'testid=sidebar-conversations-link' --expect-url '/conversations$'` shows an empty `chat-input` and `browser storage --session` no longer lists `oh:home-prompt-draft`.", "source_line": 95, "expected": [ "with nothing picked, the conversation runs in a fresh per-conversation folder under the server home.", "the home draft survives a reload and is cleared after a conversation is created." ], "notes": [ "Known issue [#16801](https://github.com/OpenHands/OpenHands/issues/16801) (`F03.launch-without-workspace`): Agent starts by looking in workspace, even when started with \"no workspace\". Live prerequisite: A working Claude Opus agent and the original task/prompt that makes it explore the default workspace; the prompt is absent." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--code-automate-toggle", "title": "Code / Automate toggle", "ids": [ "F03.launcher-mode-toggle" ], "body": "Run `control-openhands browser goto /`, `control-openhands browser fill 'testid=chat-input' ''` and `control-openhands browser snapshot 'testid=home-launcher-mode-toggle'`: `radiogroup \"Choose launcher mode\"` with `radio \"Code\" [checked]` and `radio \"Automate\"`; `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').getAttribute('data-placeholder')\"` is `What do you want to build?`. Run `control-openhands browser click 'testid=home-launcher-mode-automate'`: Automate is checked, the placeholder is `What work do you want to automate?`, `control-openhands browser attr 'testid=home-composer-actions' aria-disabled` is `true`, and `control-openhands browser enabled 'testid=open-workspace-button'` and `control-openhands browser enabled 'testid=open-plugin-picker'` are `false` (`browser screenshot --feature F03.launcher-mode-toggle --name automate`). `control-openhands browser reload` brings Code back (`browser attr 'testid=home-launcher-mode-code' aria-checked` is `true`). Light theme: arrange `control-openhands browser eval \"localStorage.setItem('openhands-color-theme','light-plus')\"` and `browser reload`, then `control-openhands browser eval \"getComputedStyle(document.querySelector('[data-testid=home-launcher-mode-code]')).color\"` and `browser screenshot 'testid=home-launcher-mode-toggle' --feature F03.launcher-mode-toggle --name light-plus-code`. The checked label uses the theme's high-emphasis ink (`text-contrast`): `rgb(31, 31, 31)` on the light pill, and a hovered inactive option (`control-openhands browser hover 'testid=home-launcher-mode-automate'`) turns the same dark ink; Solarized Light reads `rgb(0, 43, 54)`, also at phone width. Restore with `browser eval \"localStorage.removeItem('openhands-color-theme')\"` and `browser reload`.", "raw": "**Code / Automate toggle (`F03.launcher-mode-toggle`).** Run `control-openhands browser goto /`, `control-openhands browser fill 'testid=chat-input' ''` and `control-openhands browser snapshot 'testid=home-launcher-mode-toggle'`: `radiogroup \"Choose launcher mode\"` with `radio \"Code\" [checked]` and `radio \"Automate\"`; `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').getAttribute('data-placeholder')\"` is `What do you want to build?`. Run `control-openhands browser click 'testid=home-launcher-mode-automate'`: Automate is checked, the placeholder is `What work do you want to automate?`, `control-openhands browser attr 'testid=home-composer-actions' aria-disabled` is `true`, and `control-openhands browser enabled 'testid=open-workspace-button'` and `control-openhands browser enabled 'testid=open-plugin-picker'` are `false` (`browser screenshot --feature F03.launcher-mode-toggle --name automate`). `control-openhands browser reload` brings Code back (`browser attr 'testid=home-launcher-mode-code' aria-checked` is `true`). Light theme: arrange `control-openhands browser eval \"localStorage.setItem('openhands-color-theme','light-plus')\"` and `browser reload`, then `control-openhands browser eval \"getComputedStyle(document.querySelector('[data-testid=home-launcher-mode-code]')).color\"` and `browser screenshot 'testid=home-launcher-mode-toggle' --feature F03.launcher-mode-toggle --name light-plus-code`. The checked label uses the theme's high-emphasis ink (`text-contrast`): `rgb(31, 31, 31)` on the light pill, and a hovered inactive option (`control-openhands browser hover 'testid=home-launcher-mode-automate'`) turns the same dark ink; Solarized Light reads `rgb(0, 43, 54)`, also at phone width. Restore with `browser eval \"localStorage.removeItem('openhands-color-theme')\"` and `browser reload`.", "source_line": 96, "expected": [ "a `Choose launcher mode` radio group above the composer offers **Code** (checked on every page load) and **Automate**. Automate changes the placeholder to `What work do you want to automate?` and greys out and disables the Open Workspace and Plugins row (`home-composer-actions` `aria-disabled=\"true\"`); Code restores both. The active option stays readable in every colour theme." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--automate-launch", "title": "Automate launch", "ids": [ "F03.launch-automate-mode" ], "body": "On `/` click `testid=home-launcher-mode-automate`, fill `chat-input` with `Set up a disabled automation named QA_F03 Auto that runs every day at 09:00 UTC with the prompt: Reply with the single word pong. Only fill the automation form; do not run any other tools and do not create the automation.` and run `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --expect-url '/conversations/[^/]+'`. The page shows `automation-setup-panel` beside the chat (`browser testids --filter automation-setup` lists `automation-setup-form`, `automation-setup-create`, `automation-setup-save-draft`). After `control-openhands conversation wait --timeout 240` is `finished`, `control-openhands browser value 'testid=automation-setup-name'` is `QA_F03 Auto` and the filled fields carry `✦ Filled by OpenHands` (`browser screenshot --feature F03.launch-automate-mode --name setup-split`), while `control-openhands api GET /api/automation/v1 --pick total` is still `0`. The sidebar row is tagged `Automation setup: Draft`.", "raw": "**Automate launch (`F03.launch-automate-mode`).** On `/` click `testid=home-launcher-mode-automate`, fill `chat-input` with `Set up a disabled automation named QA_F03 Auto that runs every day at 09:00 UTC with the prompt: Reply with the single word pong. Only fill the automation form; do not run any other tools and do not create the automation.` and run `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --expect-url '/conversations/[^/]+'`. The page shows `automation-setup-panel` beside the chat (`browser testids --filter automation-setup` lists `automation-setup-form`, `automation-setup-create`, `automation-setup-save-draft`). After `control-openhands conversation wait --timeout 240` is `finished`, `control-openhands browser value 'testid=automation-setup-name'` is `QA_F03 Auto` and the filled fields carry `✦ Filled by OpenHands` (`browser screenshot --feature F03.launch-automate-mode --name setup-split`), while `control-openhands api GET /api/automation/v1 --pick total` is still `0`. The sidebar row is tagged `Automation setup: Draft`.", "source_line": 97, "expected": [ "submitting in Automate mode creates a conversation and opens `/conversations/` as an automation setup split view: the chat on the left and the automation form (`automation-setup-panel`) on the right, whose fields the agent fills with `✦ Filled by OpenHands` markers. Attached plugins are not sent, and nothing is created until **Create Automation**. The setup view itself is F22's." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--plugins", "title": "Plugins", "ids": [ "F03.plugin-picker" ], "body": "Run `control-openhands browser goto /conversations` (a fresh page), `control-openhands browser click 'testid=open-plugin-picker'` and `control-openhands browser wait 'testid=plugin-picker-loading' --state hidden --timeout 60000`; cards `plugin-picker-card-` are listed. `control-openhands browser fill 'testid=plugin-picker-search-input' weather` leaves one card (`control-openhands browser count '[data-testid^=\"plugin-picker-card-\"]'` is `1`); `zzz-none` makes `browser text 'testid=plugin-picker'` read `No plugins available.` With `weather` again, `control-openhands browser click 'testid=plugin-picker-toggle-city-weather' --hover-first 150`: the footer reads `1 selected` and `control-openhands browser attr 'testid=plugin-picker-toggle-city-weather' aria-label` is `Remove`. `control-openhands browser click 'testid=plugin-picker-done'`, then `control-openhands browser text 'testid=plugin-picker-count'` is `1`; reopening keeps the switch checked, and `testid=plugin-picker-close` also closes it. On hover-swapping cards, use `--hover-first 150` for subsequent Add/Remove clicks too and assert both the label and `aria-checked`; the shared index gotcha explains why a plain mouse click can be swallowed (including at phone width).", "raw": "**Plugins (`F03.plugin-picker`).** Run `control-openhands browser goto /conversations` (a fresh page), `control-openhands browser click 'testid=open-plugin-picker'` and `control-openhands browser wait 'testid=plugin-picker-loading' --state hidden --timeout 60000`; cards `plugin-picker-card-` are listed. `control-openhands browser fill 'testid=plugin-picker-search-input' weather` leaves one card (`control-openhands browser count '[data-testid^=\"plugin-picker-card-\"]'` is `1`); `zzz-none` makes `browser text 'testid=plugin-picker'` read `No plugins available.` With `weather` again, `control-openhands browser click 'testid=plugin-picker-toggle-city-weather' --hover-first 150`: the footer reads `1 selected` and `control-openhands browser attr 'testid=plugin-picker-toggle-city-weather' aria-label` is `Remove`. `control-openhands browser click 'testid=plugin-picker-done'`, then `control-openhands browser text 'testid=plugin-picker-count'` is `1`; reopening keeps the switch checked, and `testid=plugin-picker-close` also closes it. On hover-swapping cards, use `--hover-first 150` for subsequent Add/Remove clicks too and assert both the label and `aria-checked`; the shared index gotcha explains why a plain mouse click can be swallowed (including at phone width).", "source_line": 98, "expected": [ "**Plugins** opens **Attach plugins** (catalog, search, Add/Remove switch, `N selected`, Done and X); the pill then shows a count badge." ], "notes": [ "Known issue [#16210](https://github.com/OpenHands/OpenHands/issues/16210) (`F03.plugin-picker`): [Cloud Backends] Plugins not available when starting a new conversation. Live prerequisite: An authenticated Cloud organization with configured account plugins before opening New Chat > Plugins." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--plugin-reaches-the-agent", "title": "Plugin reaches the agent", "ids": [ "F03.launch-with-plugin" ], "body": "With `city-weather` attached, start the no-folder launch above without its `goto` (a page load drops the attached plugin), then `control-openhands api GET '/api/conversations//events/search?limit=20'`: the system prompt contains `city-weather:now` (a conversation started without the plugin has none). Back on `/conversations` (`control-openhands browser goto /conversations`), `control-openhands browser count 'testid=plugin-picker-count'` is `0`.", "raw": "**Plugin reaches the agent (`F03.launch-with-plugin`).** With `city-weather` attached, start the no-folder launch above without its `goto` (a page load drops the attached plugin), then `control-openhands api GET '/api/conversations//events/search?limit=20'`: the system prompt contains `city-weather:now` (a conversation started without the plugin has none). Back on `/conversations` (`control-openhands browser goto /conversations`), `control-openhands browser count 'testid=plugin-picker-count'` is `0`.", "source_line": 99, "expected": [ "a conversation started with a plugin attached loads it; the selection resets afterwards. Workspaces (local backend):" ], "notes": [ "Known issue [#18418](https://github.com/OpenHands/OpenHands/issues/18418) (`F03.create-error-toast`, `F03.launch-with-plugin`): hosted runtime 1.50.1 can fail during skills/plugins setup because its packaged plugin-schema resource is missing. Live prerequisite: the affected Cloud runtime/PyInstaller image and original plugin setup plus its account. A Local 1.55.0 launch does not verify that deployment." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--creation-failure", "title": "Creation failure", "ids": [ "F03.create-error-toast" ], "body": "Fill `chat-input` with `QA error path`, run `control-openhands service stop agent-server`, then clear the observation window with `control-openhands browser toasts --clear`. Start genuine temporal evidence before submitting: `control-openhands browser record start --feature F03.create-error-toast --name failed-launch`. Run `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --observe 'role=status' --observe-ms 500`, then immediately `control-openhands browser screenshot --feature F03.create-error-toast --name toasts` (this early still may show the loading state). Keep recording and wait for the real error with `control-openhands browser wait-text 'HTTP request failed (502 Bad Gateway)' --timeout 15000`, then `control-openhands browser wait 'role=status >> has-text=Creating conversation…' --state hidden --timeout 15000`. `browser toasts` and `browser toasts --history` show `Creating conversation…` replaced by exactly one error toast, such as `HTTP request failed (502 Bad Gateway): \"Bad Gateway: connect ECONNREFUSED 127.0.0.1:\"` (transport detail is preserved by design, #17985). Count every new error toast in this window; an additional toast is a failure to record, not one to discard. `browser text 'testid=chat-input'` still reads `QA error path` and `browser enabled 'testid=home-chat-launcher >> testid=submit-button'` is `true`. Stop the recording with `control-openhands browser record stop`; inspect its loading→error transition and recovered prompt, paired with the full screenshot. Bring the stack back with `control-openhands restart` and `control-openhands browser reload`; the prompt remains `QA error path`.\n\nWorkspaces:", "raw": "**Creation failure (`F03.create-error-toast`).** Fill `chat-input` with `QA error path`, run `control-openhands service stop agent-server`, then clear the observation window with `control-openhands browser toasts --clear`. Start genuine temporal evidence before submitting: `control-openhands browser record start --feature F03.create-error-toast --name failed-launch`. Run `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --observe 'role=status' --observe-ms 500`, then immediately `control-openhands browser screenshot --feature F03.create-error-toast --name toasts` (this early still may show the loading state). Keep recording and wait for the real error with `control-openhands browser wait-text 'HTTP request failed (502 Bad Gateway)' --timeout 15000`, then `control-openhands browser wait 'role=status >> has-text=Creating conversation…' --state hidden --timeout 15000`. `browser toasts` and `browser toasts --history` show `Creating conversation…` replaced by exactly one error toast, such as `HTTP request failed (502 Bad Gateway): \"Bad Gateway: connect ECONNREFUSED 127.0.0.1:\"` (transport detail is preserved by design, #17985). Count every new error toast in this window; an additional toast is a failure to record, not one to discard. `browser text 'testid=chat-input'` still reads `QA error path` and `browser enabled 'testid=home-chat-launcher >> testid=submit-button'` is `true`. Stop the recording with `control-openhands browser record stop`; inspect its loading→error transition and recovered prompt, paired with the full screenshot. Bring the stack back with `control-openhands restart` and `control-openhands browser reload`; the prompt remains `QA error path`.\n\nWorkspaces:", "source_line": 100, "expected": [ "when creation fails, the loading toast is replaced by one error toast and the prompt is not lost." ], "notes": [ "Known issue [#18418](https://github.com/OpenHands/OpenHands/issues/18418) (`F03.create-error-toast`, `F03.launch-with-plugin`): hosted runtime 1.50.1 can fail during skills/plugins setup because its packaged plugin-schema resource is missing. Live prerequisite: the affected Cloud runtime/PyInstaller image and original plugin setup plus its account. A Local 1.55.0 launch does not verify that deployment." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--folder-browser", "title": "Folder browser", "ids": [ "F03.folder-browser" ], "body": "Run `control-openhands browser click 'testid=open-workspace-button'`; before any workspace is picked, `control-openhands browser snapshot 'role=dialog'` shows `Open Workspace`, `combobox \"Select a workspace\"` and `button \"Confirm\" [disabled]` (`F03.open-workspace-dialog`). Run `control-openhands browser click 'testid=workspace-dropdown'`; with no saved workspace, `control-openhands browser text 'testid=workspace-dropdown-empty'` is `No workspaces yet`, above **+ Add Workspace** (`F03.workspace-dropdown-search`; `browser screenshot --feature F03.workspace-dropdown-search --name empty-list`). Run `control-openhands browser click 'testid=add-workspaces-button'`. `control-openhands browser text 'testid=folder-browser-current-path'` is `/private/home`. Run `control-openhands browser click 'testid=folder-browser-up'` twice (now ``) and `control-openhands browser click 'testid=folder-browser-entry-workspace'`; `control-openhands browser text 'testid=folder-browser-list'` lists `qa-f03-folder` and `qa-f03-repo` as `Folder` (`browser screenshot --feature F03.folder-browser --name workspace-dir`). Then, reopening the browser each time with `browser click 'testid=workspace-dropdown'` and `browser click 'testid=add-workspaces-button'` (the Open Workspace dialog stays open behind it; the browser always restarts at `/private/home`, so repeat Up twice and `folder-browser-entry-workspace`):\n - `control-openhands browser click 'testid=folder-browser-cancel'`: `browser count 'testid=folder-browser-modal'` is `0` and the selection is unchanged.\n - `control-openhands browser click 'testid=folder-browser-add-all-subdirs'`: the browser closes; after `browser click 'testid=workspace-dropdown'` the dropdown offers `qa-f03-folder` and `qa-f03-repo` (`browser snapshot 'role=dialog'`: `option \"qa-f03-folder\"`, `option \"qa-f03-repo\"`, `button \"Manage Workspaces\"`).\n - `control-openhands browser click 'testid=folder-browser-use'` (Add this directory) at `/workspace`: `control-openhands browser value 'testid=workspace-dropdown'` is `workspace` (auto-selected).", "raw": "**Folder browser (`F03.folder-browser`).** Run `control-openhands browser click 'testid=open-workspace-button'`; before any workspace is picked, `control-openhands browser snapshot 'role=dialog'` shows `Open Workspace`, `combobox \"Select a workspace\"` and `button \"Confirm\" [disabled]` (`F03.open-workspace-dialog`). Run `control-openhands browser click 'testid=workspace-dropdown'`; with no saved workspace, `control-openhands browser text 'testid=workspace-dropdown-empty'` is `No workspaces yet`, above **+ Add Workspace** (`F03.workspace-dropdown-search`; `browser screenshot --feature F03.workspace-dropdown-search --name empty-list`). Run `control-openhands browser click 'testid=add-workspaces-button'`. `control-openhands browser text 'testid=folder-browser-current-path'` is `/private/home`. Run `control-openhands browser click 'testid=folder-browser-up'` twice (now ``) and `control-openhands browser click 'testid=folder-browser-entry-workspace'`; `control-openhands browser text 'testid=folder-browser-list'` lists `qa-f03-folder` and `qa-f03-repo` as `Folder` (`browser screenshot --feature F03.folder-browser --name workspace-dir`). Then, reopening the browser each time with `browser click 'testid=workspace-dropdown'` and `browser click 'testid=add-workspaces-button'` (the Open Workspace dialog stays open behind it; the browser always restarts at `/private/home`, so repeat Up twice and `folder-browser-entry-workspace`):\n - `control-openhands browser click 'testid=folder-browser-cancel'`: `browser count 'testid=folder-browser-modal'` is `0` and the selection is unchanged.\n - `control-openhands browser click 'testid=folder-browser-add-all-subdirs'`: the browser closes; after `browser click 'testid=workspace-dropdown'` the dropdown offers `qa-f03-folder` and `qa-f03-repo` (`browser snapshot 'role=dialog'`: `option \"qa-f03-folder\"`, `option \"qa-f03-repo\"`, `button \"Manage Workspaces\"`).\n - `control-openhands browser click 'testid=folder-browser-use'` (Add this directory) at `/workspace`: `control-openhands browser value 'testid=workspace-dropdown'` is `workspace` (auto-selected).", "source_line": 104, "expected": [ "**+ Add Workspace** opens **Add workspaces** at the server home with Favorites/Locations, Up and the current path; **Add this directory** adds and selects the folder, **Add all subdirectories** adds a parent whose children become workspaces, **Cancel** adds nothing." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5837](https://github.com/OpenHands/software-agent-sdk/issues/5837) (`F03.folder-browser`): case-only directory names at a page boundary can repeat the server cursor; Canvas detects the repeated cursor and the directory listing fails instead of showing all choices. Live prerequisite: a case-sensitive filesystem with enough directories to cross the real picker page boundary, including two distinct case-only names. A case-insensitive Mac volume cannot hold that fixture.", "Known issue [OpenHands/software-agent-sdk#5849](https://github.com/OpenHands/software-agent-sdk/issues/5849) (`F03.folder-browser`, `F03.manage-workspaces`): re-adding a saved workspace or removing an already-removed stale row can return500 because the registry writes even for a no-op. Live prerequisite: an owned read-only workspace registry plus the already-saved or stale picker row; changes to registry mtime/inode alone are not a Canvas failure." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--favorites-and-locations", "title": "Favorites and locations", "ids": [ "F03.folder-browser-locations" ], "body": "With the browser open, `control-openhands browser testids 'testid=folder-browser-sidebar'` lists `folder-browser-sidebar-home`, `folder-browser-sidebar-/` and, after any no-folder conversation, `folder-browser-sidebar-workspace`. `control-openhands browser click 'testid=folder-browser-sidebar-/'` makes `browser text 'testid=folder-browser-current-path'` read `/` and `control-openhands browser enabled 'testid=folder-browser-up'` `false`; `folder-browser-sidebar-workspace` goes to `/private/home/workspace` and `folder-browser-sidebar-home` back to `/private/home`. Close with `testid=folder-browser-cancel`.", "raw": "**Favorites and locations (`F03.folder-browser-locations`).** With the browser open, `control-openhands browser testids 'testid=folder-browser-sidebar'` lists `folder-browser-sidebar-home`, `folder-browser-sidebar-/` and, after any no-folder conversation, `folder-browser-sidebar-workspace`. `control-openhands browser click 'testid=folder-browser-sidebar-/'` makes `browser text 'testid=folder-browser-current-path'` read `/` and `control-openhands browser enabled 'testid=folder-browser-up'` `false`; `folder-browser-sidebar-workspace` goes to `/private/home/workspace` and `folder-browser-sidebar-home` back to `/private/home`. Close with `testid=folder-browser-cancel`.", "source_line": 108, "expected": [ "the browser's **Favorites** (Home, plus `workspace` once a no-folder conversation exists) and **Locations** (`/`) jump to that folder; Up is disabled at `/`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--open-workspace", "title": "Open Workspace", "ids": [ "F03.open-workspace-dialog" ], "body": "The dialog is still open from the previous bullets, with `workspace` selected (its disabled Confirm before a pick is checked at the start of Folder browser). Run `control-openhands browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=workspace-dropdown-menu >> role=option[name=\"qa-f03-repo\"]'`, `control-openhands browser enabled 'testid=workspace-launch-button'` (`true`), `control-openhands browser click 'testid=workspace-launch-button'`. The dialog closes and `control-openhands browser text 'testid=home-git-control-bar-preview'` is `qa-f03-repo` and `Local Repo` (`browser screenshot --feature F03.open-workspace-dialog --name preview-bar`). `control-openhands browser click 'testid=home-git-control-bar-preview >> role=button[name*=\"qa-f03-repo\"]'` reopens the dialog with `browser value 'testid=workspace-dropdown'` `qa-f03-repo`; `control-openhands browser click 'testid=close-open-workspace-dialog'` closes it, and so does `control-openhands browser press Escape`.", "raw": "**Open Workspace (`F03.open-workspace-dialog`).** The dialog is still open from the previous bullets, with `workspace` selected (its disabled Confirm before a pick is checked at the start of Folder browser). Run `control-openhands browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=workspace-dropdown-menu >> role=option[name=\"qa-f03-repo\"]'`, `control-openhands browser enabled 'testid=workspace-launch-button'` (`true`), `control-openhands browser click 'testid=workspace-launch-button'`. The dialog closes and `control-openhands browser text 'testid=home-git-control-bar-preview'` is `qa-f03-repo` and `Local Repo` (`browser screenshot --feature F03.open-workspace-dialog --name preview-bar`). `control-openhands browser click 'testid=home-git-control-bar-preview >> role=button[name*=\"qa-f03-repo\"]'` reopens the dialog with `browser value 'testid=workspace-dropdown'` `qa-f03-repo`; `control-openhands browser click 'testid=close-open-workspace-dialog'` closes it, and so does `control-openhands browser press Escape`.", "source_line": 109, "expected": [ "**Open Workspace** opens a dialog with a workspace combobox; Confirm stays disabled until a pick; confirming swaps the pill for a preview bar that reopens the dialog; X and Escape close it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--launch-in-a-folder", "title": "Launch in a folder", "ids": [ "F03.launch-in-workspace" ], "body": "Open Workspace (above) left `qa-f03-repo` confirmed in the preview bar. Run `control-openhands browser fill 'testid=chat-input' 'Run pwd and reply with only its output.'` and `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --observe 'text=Creating conversation' --expect-url '/conversations/[^/]+$'`. `observed` shows `Creating conversation…` then ``; the returned URL holds ``. `control-openhands conversation wait --timeout 180` reports `workspace` `/workspace/qa-f03-repo`, and `control-openhands conversation events --kinds ObservationEvent,MessageEvent` shows the terminal observation and the reply `/workspace/qa-f03-repo`.", "raw": "**Launch in a folder (`F03.launch-in-workspace`).** Open Workspace (above) left `qa-f03-repo` confirmed in the preview bar. Run `control-openhands browser fill 'testid=chat-input' 'Run pwd and reply with only its output.'` and `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --observe 'text=Creating conversation' --expect-url '/conversations/[^/]+$'`. `observed` shows `Creating conversation…` then ``; the returned URL holds ``. `control-openhands conversation wait --timeout 180` reports `workspace` `/workspace/qa-f03-repo`, and `control-openhands conversation events --kinds ObservationEvent,MessageEvent` shows the terminal observation and the reply `/workspace/qa-f03-repo`.", "source_line": 110, "expected": [ "submitting with a workspace picked shows `Creating conversation…`, opens `/conversations/`, and the agent runs in that folder." ], "notes": [ "Known issue [#15701](https://github.com/OpenHands/OpenHands/issues/15701) (`F03.launch-in-workspace`): Creating Conversation toast stuck visible. Live prerequisite: The original stuck creation request/outcome sequence; the report is intermittent and gives no deterministic trigger." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--search-groups-and-empty", "title": "Search, groups and empty", "ids": [ "F03.workspace-dropdown-search" ], "body": "Run `control-openhands browser goto /` (Launch in a folder left its conversation), `control-openhands browser click 'testid=open-workspace-button'` and `control-openhands browser click 'testid=workspace-dropdown'`. With the dropdown open, `control-openhands browser fill 'testid=workspace-dropdown' repo` and `control-openhands browser text 'testid=workspace-dropdown-menu'` is `qa-f03-repo`. With the parent and the standalone `workspace` saved, the menu text is `workspace`, `qa-f03-folder`, `qa-f03-repo`, `Other`, `workspace` and `control-openhands browser count 'testid=workspace-group-header'` is `2` (`--name grouped`). `control-openhands browser fill 'testid=workspace-dropdown' zzz-none` shows the empty state: `control-openhands browser text 'testid=workspace-dropdown-empty'` is `No workspaces yet` (`--name no-match`) and `control-openhands browser count 'testid=add-workspaces-button'` is still `1`. Clear with `control-openhands browser fill 'testid=workspace-dropdown' ''`; `browser count 'testid=workspace-dropdown-empty'` is `0` again.", "raw": "**Search, groups and empty (`F03.workspace-dropdown-search`).** Run `control-openhands browser goto /` (Launch in a folder left its conversation), `control-openhands browser click 'testid=open-workspace-button'` and `control-openhands browser click 'testid=workspace-dropdown'`. With the dropdown open, `control-openhands browser fill 'testid=workspace-dropdown' repo` and `control-openhands browser text 'testid=workspace-dropdown-menu'` is `qa-f03-repo`. With the parent and the standalone `workspace` saved, the menu text is `workspace`, `qa-f03-folder`, `qa-f03-repo`, `Other`, `workspace` and `control-openhands browser count 'testid=workspace-group-header'` is `2` (`--name grouped`). `control-openhands browser fill 'testid=workspace-dropdown' zzz-none` shows the empty state: `control-openhands browser text 'testid=workspace-dropdown-empty'` is `No workspaces yet` (`--name no-match`) and `control-openhands browser count 'testid=add-workspaces-button'` is still `1`. Clear with `control-openhands browser fill 'testid=workspace-dropdown' ''`; `browser count 'testid=workspace-dropdown-empty'` is `0` again.", "source_line": 111, "expected": [ "typing filters workspaces; with more than one group, workspaces are grouped by parent with an **Other** group; no match (or no workspaces) shows `No workspaces yet`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--reload", "title": "Reload", "ids": [ "F03.selection-after-reload" ], "body": "After confirming qa-f03-repo, `control-openhands browser reload`; `control-openhands browser count 'testid=home-git-control-bar-preview'` is `0` and the pill is back. Click it; `control-openhands browser value 'testid=workspace-dropdown'` is `qa-f03-repo` (sessionStorage `oh:home-selected-workspace-path`). An unconfirmed pick persists the same way: `control-openhands browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=workspace-dropdown-menu >> role=option[name=\"qa-f03-folder\"]'` (value `qa-f03-folder`), then `control-openhands browser click 'testid=close-open-workspace-dialog'` (never Confirm or Launch): the preview count stays `0` and `control-openhands browser storage --session --values` shows `oh:home-selected-workspace-path` ending in `/workspace/qa-f03-folder`. Leave and come back with `control-openhands browser goto /settings` (lands on `/settings/agents`, `browser count 'testid=home-screen'` `0`) and `control-openhands browser goto /conversations`, click `open-workspace-button`: `browser value 'testid=workspace-dropdown'` is `qa-f03-folder` (`browser screenshot --feature F03.selection-after-reload --name unconfirmed-pick-restored`). Close with the X, `control-openhands browser reload`, click the pill again: the value is still `qa-f03-folder` and `browser enabled 'testid=workspace-launch-button'` is `true`. Restore the state the next bullets start from with `browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=workspace-dropdown-menu >> role=option[name=\"qa-f03-repo\"]'` (value `qa-f03-repo`, the storage value ends in `/workspace/qa-f03-repo`) and `control-openhands browser click 'testid=workspace-launch-button'`: the dialog closes and `control-openhands browser text 'testid=home-git-control-bar-preview'` is `qa-f03-repo` and `Local Repo` again.", "raw": "**Reload (`F03.selection-after-reload`).** After confirming qa-f03-repo, `control-openhands browser reload`; `control-openhands browser count 'testid=home-git-control-bar-preview'` is `0` and the pill is back. Click it; `control-openhands browser value 'testid=workspace-dropdown'` is `qa-f03-repo` (sessionStorage `oh:home-selected-workspace-path`). An unconfirmed pick persists the same way: `control-openhands browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=workspace-dropdown-menu >> role=option[name=\"qa-f03-folder\"]'` (value `qa-f03-folder`), then `control-openhands browser click 'testid=close-open-workspace-dialog'` (never Confirm or Launch): the preview count stays `0` and `control-openhands browser storage --session --values` shows `oh:home-selected-workspace-path` ending in `/workspace/qa-f03-folder`. Leave and come back with `control-openhands browser goto /settings` (lands on `/settings/agents`, `browser count 'testid=home-screen'` `0`) and `control-openhands browser goto /conversations`, click `open-workspace-button`: `browser value 'testid=workspace-dropdown'` is `qa-f03-folder` (`browser screenshot --feature F03.selection-after-reload --name unconfirmed-pick-restored`). Close with the X, `control-openhands browser reload`, click the pill again: the value is still `qa-f03-folder` and `browser enabled 'testid=workspace-launch-button'` is `true`. Restore the state the next bullets start from with `browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=workspace-dropdown-menu >> role=option[name=\"qa-f03-repo\"]'` (value `qa-f03-repo`, the storage value ends in `/workspace/qa-f03-repo`) and `control-openhands browser click 'testid=workspace-launch-button'`: the dialog closes and `control-openhands browser text 'testid=home-git-control-bar-preview'` is `qa-f03-repo` and `Local Repo` again.", "source_line": 112, "expected": [ "after a reload the preview bar is gone, but the dialog preselects the last workspace picked in its dropdown, confirmed or not: a pick closed with the X survives leaving for another page and coming back, and a reload." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--workspace-mode", "title": "Workspace mode", "ids": [ "F03.workspace-mode" ], "body": "With the preview bar showing, `control-openhands browser click 'testid=workspace-mode-selector'`, `control-openhands browser snapshot 'testid=workspace-mode-selector-menu'` (`menuitemradio \"Local Repo\" [checked]`, `menuitemradio \"New Worktree\"`), `control-openhands browser click 'testid=workspace-mode-selector-option-new_worktree'`. `browser storage --values` shows `openhands-last-local-workspace-mode` `new_worktree`; after `browser reload` and reconfirming the workspace, `browser text 'testid=home-git-control-bar-preview'` reads `qa-f03-repo` and `New Worktree`. Restore with `testid=workspace-mode-selector-option-local_repo`.", "raw": "**Workspace mode (`F03.workspace-mode`).** With the preview bar showing, `control-openhands browser click 'testid=workspace-mode-selector'`, `control-openhands browser snapshot 'testid=workspace-mode-selector-menu'` (`menuitemradio \"Local Repo\" [checked]`, `menuitemradio \"New Worktree\"`), `control-openhands browser click 'testid=workspace-mode-selector-option-new_worktree'`. `browser storage --values` shows `openhands-last-local-workspace-mode` `new_worktree`; after `browser reload` and reconfirming the workspace, `browser text 'testid=home-git-control-bar-preview'` reads `qa-f03-repo` and `New Worktree`. Restore with `testid=workspace-mode-selector-option-local_repo`.", "source_line": 113, "expected": [ "the preview bar's mode menu switches **Local Repo** / **New Worktree**; the choice persists across reloads." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--manage", "title": "Manage", "ids": [ "F03.manage-workspaces" ], "body": "With the parent and `workspace` saved: `control-openhands browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=manage-workspaces-button'`; `control-openhands browser testids 'testid=manage-workspaces-modal'` shows `manage-workspaces-row-workspace`, `manage-workspaces-parent-row-workspace`, `manage-workspaces-child-qa-f03-folder` and `manage-workspaces-child-qa-f03-repo`. `control-openhands browser click 'testid=manage-workspaces-remove-workspace'`; `control-openhands browser text 'testid=confirmation-modal'` reads `Remove \"workspace\" from your saved workspaces?`. `control-openhands browser click 'testid=confirmation-modal >> testid=cancel-button'` keeps the row (`browser count 'testid=manage-workspaces-row-workspace'` is `1`); repeat Remove and `control-openhands browser click 'testid=confirmation-modal >> testid=confirm-button'`: the count is `0`. Removing a workspace clears the dialog's selection only when it was the selected one: with `qa-f03-repo` selected, `control-openhands browser click 'testid=manage-workspaces-done'` leaves `browser value 'testid=workspace-dropdown'` at `qa-f03-repo`. After `browser reload` the dropdown lists only the two children, and `control-openhands api GET /api/workspaces` shows `workspaces: []` plus the parent. Cleanup proves Remove parent: `control-openhands browser click 'testid=manage-workspaces-remove-parent-workspace'` reads `Remove \"workspace\" and its 2 child workspaces?`; Confirm leaves `browser text 'testid=manage-workspaces-list'` at `You don't have any workspaces yet.` and `api GET /api/workspaces` empty; because the selected `qa-f03-repo` belonged to that parent, after `browser click 'testid=manage-workspaces-done'` `browser value 'testid=workspace-dropdown'` is empty and `browser enabled 'testid=workspace-launch-button'` is `false`.\n\nAutomations on home (fixtures imported and turned on):", "raw": "**Manage (`F03.manage-workspaces`).** With the parent and `workspace` saved: `control-openhands browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'testid=manage-workspaces-button'`; `control-openhands browser testids 'testid=manage-workspaces-modal'` shows `manage-workspaces-row-workspace`, `manage-workspaces-parent-row-workspace`, `manage-workspaces-child-qa-f03-folder` and `manage-workspaces-child-qa-f03-repo`. `control-openhands browser click 'testid=manage-workspaces-remove-workspace'`; `control-openhands browser text 'testid=confirmation-modal'` reads `Remove \"workspace\" from your saved workspaces?`. `control-openhands browser click 'testid=confirmation-modal >> testid=cancel-button'` keeps the row (`browser count 'testid=manage-workspaces-row-workspace'` is `1`); repeat Remove and `control-openhands browser click 'testid=confirmation-modal >> testid=confirm-button'`: the count is `0`. Removing a workspace clears the dialog's selection only when it was the selected one: with `qa-f03-repo` selected, `control-openhands browser click 'testid=manage-workspaces-done'` leaves `browser value 'testid=workspace-dropdown'` at `qa-f03-repo`. After `browser reload` the dropdown lists only the two children, and `control-openhands api GET /api/workspaces` shows `workspaces: []` plus the parent. Cleanup proves Remove parent: `control-openhands browser click 'testid=manage-workspaces-remove-parent-workspace'` reads `Remove \"workspace\" and its 2 child workspaces?`; Confirm leaves `browser text 'testid=manage-workspaces-list'` at `You don't have any workspaces yet.` and `api GET /api/workspaces` empty; because the selected `qa-f03-repo` belonged to that parent, after `browser click 'testid=manage-workspaces-done'` `browser value 'testid=workspace-dropdown'` is empty and `browser enabled 'testid=workspace-launch-button'` is `false`.\n\nAutomations on home (fixtures imported and turned on):", "source_line": 114, "expected": [ "**Manage Workspaces** lists workspaces and parents (with children); every Remove asks for confirmation; Cancel keeps, Confirm removes for good and clears the selection; with nothing left it reads `You don't have any workspaces yet.`" ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5849](https://github.com/OpenHands/software-agent-sdk/issues/5849) (`F03.folder-browser`, `F03.manage-workspaces`): re-adding a saved workspace or removing an already-removed stale row can return500 because the registry writes even for a no-op. Live prerequisite: an owned read-only workspace registry plus the already-saved or stale picker row; changes to registry mtime/inode alone are not a Canvas failure." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--recommended-rail", "title": "Recommended rail", "ids": [ "F03.recommended-automations-rail" ], "body": "On `/` after `browser reload`, `control-openhands browser attr 'testid=recommended-automations-rail-fade-left' data-visible` is `false` and `...-fade-right` is `true`; `control-openhands browser scroll 'testid=recommended-automation-rail-card-github-agents-md-maintainer'` turns the left fade `true` (`browser screenshot --feature F03.recommended-automations-rail --name scrolled-end`). `control-openhands browser click 'testid=recommended-automation-rail-card-custom-automation' --expect-url 'automations/new'` lands on `/automations/new/custom-automation` (`github-pr-reviewer`, `github-issue-to-pr` and `news-digest` do the same).", "raw": "**Recommended rail (`F03.recommended-automations-rail`).** On `/` after `browser reload`, `control-openhands browser attr 'testid=recommended-automations-rail-fade-left' data-visible` is `false` and `...-fade-right` is `true`; `control-openhands browser scroll 'testid=recommended-automation-rail-card-github-agents-md-maintainer'` turns the left fade `true` (`browser screenshot --feature F03.recommended-automations-rail --name scrolled-end`). `control-openhands browser click 'testid=recommended-automation-rail-card-custom-automation' --expect-url 'automations/new'` lands on `/automations/new/custom-automation` (`github-pr-reviewer`, `github-issue-to-pr` and `news-digest` do the same).", "source_line": 118, "expected": [ "a horizontally scrolling rail of recommended automations with edge fades; cards with a setup form open `/automations/new/`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--responder-choice", "title": "Responder choice", "ids": [ "F03.recommended-responder-choice" ], "body": "From `/conversations` (`control-openhands browser goto /conversations`; the rail click left `/automations/new/...`), `control-openhands browser click 'testid=recommended-automation-rail-card-github-agents-md-maintainer'`; `control-openhands browser snapshot 'testid=responder-deployment-modal'` shows `Choose how your responder should run`, **Continue with local setup** and **Open OpenHands Cloud integrations** (`slack-channel-monitor` too). `control-openhands browser click 'testid=responder-deployment-continue-local'` saves an `OPENHANDS_URL` secret and opens the GitHub MCP install dialog (`browser snapshot 'role=dialog'`: `dialog \"GitHub\"`, Personal access token); `control-openhands browser click 'role=dialog >> role=button[name=\"Cancel\"]'` closes it on `/conversations`. Delete `OPENHANDS_URL` afterwards in Settings → Secrets (F14): `control-openhands browser goto /settings/secrets`, `control-openhands browser click 'testid=secret-item >> has-text=OPENHANDS_URL >> testid=delete-secret-button'`, then `control-openhands browser click 'role=dialog >> role=button[name=\"Confirm\"]'`.", "raw": "**Responder choice (`F03.recommended-responder-choice`).** From `/conversations` (`control-openhands browser goto /conversations`; the rail click left `/automations/new/...`), `control-openhands browser click 'testid=recommended-automation-rail-card-github-agents-md-maintainer'`; `control-openhands browser snapshot 'testid=responder-deployment-modal'` shows `Choose how your responder should run`, **Continue with local setup** and **Open OpenHands Cloud integrations** (`slack-channel-monitor` too). `control-openhands browser click 'testid=responder-deployment-continue-local'` saves an `OPENHANDS_URL` secret and opens the GitHub MCP install dialog (`browser snapshot 'role=dialog'`: `dialog \"GitHub\"`, Personal access token); `control-openhands browser click 'role=dialog >> role=button[name=\"Cancel\"]'` closes it on `/conversations`. Delete `OPENHANDS_URL` afterwards in Settings → Secrets (F14): `control-openhands browser goto /settings/secrets`, `control-openhands browser click 'testid=secret-item >> has-text=OPENHANDS_URL >> testid=delete-secret-button'`, then `control-openhands browser click 'role=dialog >> role=button[name=\"Confirm\"]'`.", "source_line": 119, "expected": [ "GitHub/Slack responder cards on a local backend first ask **Choose how your responder should run**." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--cloud-option", "title": "Cloud option", "ids": [ "F03.responder-cloud-option" ], "body": "From `/conversations` (`control-openhands browser goto /conversations`), `control-openhands browser click 'testid=recommended-automation-rail-card-slack-channel-monitor'`, then `control-openhands browser click 'testid=responder-deployment-open-openhands-cloud'`: `browser count 'testid=responder-deployment-modal'` is `0`, `control-openhands browser tabs` lists a second page `https://app.all-hands.dev/settings/integrations` (it can take a second to appear: rerun `tabs` before concluding) (close it with `control-openhands browser close-tab 1`), and `api GET /api/settings/secrets` has no `OPENHANDS_URL`. Reopen the card and `control-openhands browser click 'testid=responder-deployment-modal-close'`: the dialog is gone and nothing else opens.", "raw": "**Cloud option (`F03.responder-cloud-option`).** From `/conversations` (`control-openhands browser goto /conversations`), `control-openhands browser click 'testid=recommended-automation-rail-card-slack-channel-monitor'`, then `control-openhands browser click 'testid=responder-deployment-open-openhands-cloud'`: `browser count 'testid=responder-deployment-modal'` is `0`, `control-openhands browser tabs` lists a second page `https://app.all-hands.dev/settings/integrations` (it can take a second to appear: rerun `tabs` before concluding) (close it with `control-openhands browser close-tab 1`), and `api GET /api/settings/secrets` has no `OPENHANDS_URL`. Reopen the card and `control-openhands browser click 'testid=responder-deployment-modal-close'`: the dialog is gone and nothing else opens.", "source_line": 120, "expected": [ "in that dialog **Open OpenHands Cloud integrations** opens `https://app.all-hands.dev/settings/integrations` in a new tab, closes the dialog and saves no secret; X closes it without a choice." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--missing-integration", "title": "Missing integration", "ids": [ "F03.recommended-missing-integration" ], "body": "`control-openhands browser click 'testid=recommended-automation-rail-card-research-brief-writer'` opens `dialog \"Tavily\"` (Command `npx -y tavily-mcp`, Tavily API key); Cancel closes it and no conversation is created.", "raw": "**Missing integration (`F03.recommended-missing-integration`).** `control-openhands browser click 'testid=recommended-automation-rail-card-research-brief-writer'` opens `dialog \"Tavily\"` (Command `npx -y tavily-mcp`, Tavily API key); Cancel closes it and no conversation is created.", "source_line": 121, "expected": [ "a card whose required integration is missing opens that integration's install dialog first." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--already-added", "title": "Already added", "ids": [ "F03.recommended-hides-added" ], "body": "`control-openhands browser count 'testid=recommended-automation-rail-card-news-digest'` is `1`. Import `qa-f03-news.automation.json` (same fixture recipe as Preconditions, with `\"name\":\"Daily news digest\"`; it stays disabled), `browser goto /`, `browser wait 'testid=recommended-automation-rail-card-custom-automation'`: the news-digest count is `0`. Delete it on `/automations` (`control-openhands browser goto /automations`, then the Cleanup recipe below with `has-text=Daily news digest`); back on `/` (`control-openhands browser goto /`) the count is `1` again.", "raw": "**Already added (`F03.recommended-hides-added`).** `control-openhands browser count 'testid=recommended-automation-rail-card-news-digest'` is `1`. Import `qa-f03-news.automation.json` (same fixture recipe as Preconditions, with `\"name\":\"Daily news digest\"`; it stays disabled), `browser goto /`, `browser wait 'testid=recommended-automation-rail-card-custom-automation'`: the news-digest count is `0`. Delete it on `/automations` (`control-openhands browser goto /automations`, then the Cleanup recipe below with `has-text=Daily news digest`); back on `/` (`control-openhands browser goto /`) the count is `1` again.", "source_line": 122, "expected": [ "a catalog automation that already exists (matched by name, enabled or not) drops off the rail; deleting it brings the card back." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--activity-list", "title": "Activity list", "ids": [ "F03.home-automations-list", "F03.home-automation-links" ], "body": "`control-openhands browser wait '[data-testid^=\"running-automation-row-\"]'`, `control-openhands browser text 'testid=running-automations-list'`: both fixtures with `0 0 1 1 *` and `No activity yet` (`browser screenshot --feature F03.home-automations-list --name list`); no row has a sparkline link yet (the run-bar link is checked after the first run, in the next bullet). `control-openhands browser click '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Beta >> role=link' --expect-url '/automations/[^/?]+'` opens the detail page; after `control-openhands browser goto /`, `control-openhands browser click 'testid=home-automations-manage' --expect-url '/automations$'` opens the dashboard.", "raw": "**Activity list (`F03.home-automations-list`, `F03.home-automation-links`).** `control-openhands browser wait '[data-testid^=\"running-automation-row-\"]'`, `control-openhands browser text 'testid=running-automations-list'`: both fixtures with `0 0 1 1 *` and `No activity yet` (`browser screenshot --feature F03.home-automations-list --name list`); no row has a sparkline link yet (the run-bar link is checked after the first run, in the next bullet). `control-openhands browser click '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Beta >> role=link' --expect-url '/automations/[^/?]+'` opens the detail page; after `control-openhands browser goto /`, `control-openhands browser click 'testid=home-automations-manage' --expect-url '/automations$'` opens the dashboard.", "source_line": 123, "expected": [ "**Automations** lists enabled automations with schedule, last run and status, a sparkline, **+ Add**, and `No automations yet. Add one to automate repetitive work.` when empty.", "a row opens `/automations/`, a sparkline bar opens `/automations/?run=`, **+ Add** opens `/automations`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--row-menu-run-now-and-cancel", "title": "Row menu, Run now and Cancel", "ids": [ "F03.home-row-menu", "F03.cancel-in-flight-run", "F03.home-automation-links" ], "body": "Run `control-openhands browser goto /` (the Manage link left the dashboard), then `control-openhands browser click '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Pong >> [data-testid^=\"running-automation-menu-\"]'` and `control-openhands browser snapshot '[data-testid^=\"running-automation-menu-panel-\"]'`: Run now, View, Edit, Turn off, separator, Pin to dashboard. `control-openhands browser click '[data-testid^=\"running-automation-run-\"]' --observe 'role=status'` observes `Automation dispatched`. Reopen the menu: `Run now [disabled]` and **Cancel run**. `control-openhands browser click '[data-testid^=\"running-automation-cancel-\"]'` and `control-openhands browser toasts` reads `Automation run cancelled`; `control-openhands api GET /api/automation/v1//runs` (id from `browser testids --filter running-automation-row`) shows `CANCELLED`, and after `browser reload` the row reads `Just now · Cancelled`. Pong now has a run: `control-openhands browser click '[data-testid^=\"running-automation-activity-\"] >> role=link' --expect-url '/automations/[^/?]+\\?run='` opens it; return with `control-openhands browser goto /`. Known failure (reproduced 2026-10-12, [OpenHands/automation#514](https://github.com/OpenHands/automation/issues/514)): a bounded script run cancelled from this Home menu shows `Automation run cancelled` and `Cancelled` in the row and Activity Log, but its launcher-owned `python3 worker.py` process remains running after cancellation. This was reproduced in `backfill-18309-integrations-worker` on `fa1f724ab5737a17453ad6081129eb91c5a423c6`, SDK 1.54.0 and automation 1.19.3: package a run-owned `worker.py` that prints one tick and sleeps 2 seconds for 90 iterations; upload its tarball with `control-openhands api POST '/api/automation/v1/uploads?name=qa-backfill-sleep' --write --data-file --content-type application/gzip`, then create an active scripted automation through `api POST /api/automation/v1 --write --data '@'` with `entrypoint: python3 worker.py`, the returned `tarball_path`, a UTC `0 0 1 1 *` cron trigger and timeout 240. Open `/automations/`, click `role=button[name=\"Run now\"]`, wait for the actual worker to start, return Home and click `testid=running-automation-menu-` then `testid=running-automation-cancel-`. Read-only process checks restricted to this launcher's descendants still found the worker after the cancellation UI settled. Evidence: `F03.cancel-in-flight-run/local-worker-cancel-514.png`, `F03.cancel-in-flight-run/owned-worker-post-cancel.json` and `F23.activity-log/cancelled-514.aria.txt`; the worker is bounded to exit without external calls. Let it exit, then delete the temporary automation with `control-openhands api DELETE /api/automation/v1/ --write`.", "raw": "**Row menu, Run now and Cancel (`F03.home-row-menu`, `F03.cancel-in-flight-run`, `F03.home-automation-links`).** Run `control-openhands browser goto /` (the Manage link left the dashboard), then `control-openhands browser click '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Pong >> [data-testid^=\"running-automation-menu-\"]'` and `control-openhands browser snapshot '[data-testid^=\"running-automation-menu-panel-\"]'`: Run now, View, Edit, Turn off, separator, Pin to dashboard. `control-openhands browser click '[data-testid^=\"running-automation-run-\"]' --observe 'role=status'` observes `Automation dispatched`. Reopen the menu: `Run now [disabled]` and **Cancel run**. `control-openhands browser click '[data-testid^=\"running-automation-cancel-\"]'` and `control-openhands browser toasts` reads `Automation run cancelled`; `control-openhands api GET /api/automation/v1//runs` (id from `browser testids --filter running-automation-row`) shows `CANCELLED`, and after `browser reload` the row reads `Just now · Cancelled`. Pong now has a run: `control-openhands browser click '[data-testid^=\"running-automation-activity-\"] >> role=link' --expect-url '/automations/[^/?]+\\?run='` opens it; return with `control-openhands browser goto /`. Known failure (reproduced 2026-10-12, [OpenHands/automation#514](https://github.com/OpenHands/automation/issues/514)): a bounded script run cancelled from this Home menu shows `Automation run cancelled` and `Cancelled` in the row and Activity Log, but its launcher-owned `python3 worker.py` process remains running after cancellation. This was reproduced in `backfill-18309-integrations-worker` on `fa1f724ab5737a17453ad6081129eb91c5a423c6`, SDK 1.54.0 and automation 1.19.3: package a run-owned `worker.py` that prints one tick and sleeps 2 seconds for 90 iterations; upload its tarball with `control-openhands api POST '/api/automation/v1/uploads?name=qa-backfill-sleep' --write --data-file --content-type application/gzip`, then create an active scripted automation through `api POST /api/automation/v1 --write --data '@'` with `entrypoint: python3 worker.py`, the returned `tarball_path`, a UTC `0 0 1 1 *` cron trigger and timeout 240. Open `/automations/`, click `role=button[name=\"Run now\"]`, wait for the actual worker to start, return Home and click `testid=running-automation-menu-` then `testid=running-automation-cancel-`. Read-only process checks restricted to this launcher's descendants still found the worker after the cancellation UI settled. Evidence: `F03.cancel-in-flight-run/local-worker-cancel-514.png`, `F03.cancel-in-flight-run/owned-worker-post-cancel.json` and `F23.activity-log/cancelled-514.aria.txt`; the worker is bounded to exit without external calls. Let it exit, then delete the temporary automation with `control-openhands api DELETE /api/automation/v1/ --write`.", "source_line": 124, "expected": [ "the row kebab offers Run now, View, Edit, Turn off and Pin to dashboard; Run now toasts `Automation dispatched` and is disabled while a run is in flight; Edit opens the automation setup page `/automations/setup?automationId=` with the form prefilled (no modal), where **Save** toasts `Automation updated`.", "**Cancel run** appears while a run is pending or running and cancels it (`Automation run cancelled`).", "a row opens `/automations/`, a sparkline bar opens `/automations/?run=`, **+ Add** opens `/automations`." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [OpenHands/automation#514](https://github.com/OpenHands/automation/issues/514)): a bounded script run cancelled from this Home menu shows `Automation run cancelled` and `Cancelled` in the row and Activity Log, but its launcher-owned `python3 worker.py` process remains running after cancellation. This was reproduced in `backfill-18309-integrations-worker` on `fa1f724ab5737a17453ad6081129eb91c5a423c6`, SDK 1.54.0 and automation 1.19.3: package a run-owned `worker.py` that prints one tick and sleeps 2 seconds for 90 iterations; upload its tarball with `control-openhands api POST '/api/automation/v1/uploads?name=qa-backfill-sleep' --write --data-file --content-type application/gzip`, then create an active scripted automation through `api POST /api/automation/v1 --write --data '@'` with `entrypoint: python3 worker.py`, the returned `tarball_path`, a UTC `0 0 1 1 *` cron trigger and timeout 240. Open `/automations/`, click `role=button[name=\"Run now\"]`, wait for the actual worker to start, return Home and click `testid=running-automation-menu-` then `testid=running-automation-cancel-`. Read-only process checks restricted to this launcher's descendants still found the worker after the cancellation UI settled. Evidence: `F03.cancel-in-flight-run/local-worker-cancel-514.png`, `F03.cancel-in-flight-run/owned-worker-post-cancel.json` and `F23.activity-log/cancelled-514.aria.txt`; the worker is bounded to exit without external calls. Let it exit, then delete the temporary automation with `control-openhands api DELETE /api/automation/v1/ --write`." ], "states": [ "failure" ] }, { "name": "F03--tooltip", "title": "Tooltip", "ids": [ "F03.home-run-tooltip" ], "body": "`control-openhands browser tooltip '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Pong'` returns `QA_F03 Pong`, `Trigger 0 0 1 1 *`, `Status Cancelled`, `Last run Just now`, `Task Cancelled by user`.", "raw": "**Tooltip (`F03.home-run-tooltip`).** `control-openhands browser tooltip '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Pong'` returns `QA_F03 Pong`, `Trigger 0 0 1 1 *`, `Status Cancelled`, `Last run Just now`, `Task Cancelled by user`.", "source_line": 125, "expected": [ "hovering a row shows its trigger, status, last run and task." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--run-outcomes", "title": "Run outcomes", "ids": [ "F03.home-run-outcome-health" ], "body": "Arrange as `F23.run-task-outcome` does (API `preset/prompt`, yearly cron `0 0 1 1 *`, enabled) `QA_F03 Blocked`, which finishes with status `blocked`, and `QA_F03 Review`, which finishes with status `unknown`, each prompt ending `Do not run any tools.` On `/`, for each row open `'[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Blocked >> [data-testid^=\"running-automation-menu-\"]'` and click `[data-testid^=\"running-automation-run-\"]` (`Automation dispatched`); poll `control-openhands api GET /api/automation/v1//runs` until the run is `COMPLETED` (about 35 s), then `control-openhands browser reload`. `control-openhands browser text 'testid=running-automations-list'` shows `… · Blocked` and `… · Needs review`; `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=running-automation-row-] a[aria-label]')].filter(a=>a.querySelector('.rounded-full')).map(a=>a.getAttribute('aria-label')+' | '+a.querySelector('.rounded-full').className.match(/bg-[a-z-]+/g)).join('; ')\"` lists `QA_F03 Blocked Blocked | bg-warning` and `QA_F03 Review Needs review | bg-warning` (in list order, which can vary). `control-openhands browser tooltip` on each row reads `Status Blocked` / `Task ` and `Status Needs review` / `Task ` (before each tooltip read, `browser hover 'testid=chat-input'` and `browser wait 'role=tooltip' --state detached --timeout 3000`, or the previous row's tooltip is returned). Pin both from their row menus (`[data-testid^=\"running-automation-pin-\"]`): each card's strip shows the Blocked (!) (`run-status-icon-warning`) or Needs review (?) (`run-status-icon-needs-review`) icon, then the run's summary and its conversation title. Expected: the summary is visible (`browser eval \"[...document.querySelectorAll('[data-testid^=pinned-automation-card-] [data-testid=run-status-badge-icon]')].map(i=>Math.round(i.nextElementSibling.getBoundingClientRect().width))\"` is above `0`). Known failure (reproduced 2026-10-09, #18234): when the run has a conversation, the summary is 0 px wide at desktop and phone, so only the conversation title shows (the summary is still in `browser text`). Repeat at `browser viewport phone`. The status-less success case has no live producer (automation 1.19.0 with SDK 1.53.0 always sends a status), as `F23.run-task-outcome` records: `blocked`. Delete both automations afterwards.", "raw": "**Run outcomes (`F03.home-run-outcome-health`).** Arrange as `F23.run-task-outcome` does (API `preset/prompt`, yearly cron `0 0 1 1 *`, enabled) `QA_F03 Blocked`, which finishes with status `blocked`, and `QA_F03 Review`, which finishes with status `unknown`, each prompt ending `Do not run any tools.` On `/`, for each row open `'[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Blocked >> [data-testid^=\"running-automation-menu-\"]'` and click `[data-testid^=\"running-automation-run-\"]` (`Automation dispatched`); poll `control-openhands api GET /api/automation/v1//runs` until the run is `COMPLETED` (about 35 s), then `control-openhands browser reload`. `control-openhands browser text 'testid=running-automations-list'` shows `… · Blocked` and `… · Needs review`; `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=running-automation-row-] a[aria-label]')].filter(a=>a.querySelector('.rounded-full')).map(a=>a.getAttribute('aria-label')+' | '+a.querySelector('.rounded-full').className.match(/bg-[a-z-]+/g)).join('; ')\"` lists `QA_F03 Blocked Blocked | bg-warning` and `QA_F03 Review Needs review | bg-warning` (in list order, which can vary). `control-openhands browser tooltip` on each row reads `Status Blocked` / `Task ` and `Status Needs review` / `Task ` (before each tooltip read, `browser hover 'testid=chat-input'` and `browser wait 'role=tooltip' --state detached --timeout 3000`, or the previous row's tooltip is returned). Pin both from their row menus (`[data-testid^=\"running-automation-pin-\"]`): each card's strip shows the Blocked (!) (`run-status-icon-warning`) or Needs review (?) (`run-status-icon-needs-review`) icon, then the run's summary and its conversation title. Expected: the summary is visible (`browser eval \"[...document.querySelectorAll('[data-testid^=pinned-automation-card-] [data-testid=run-status-badge-icon]')].map(i=>Math.round(i.nextElementSibling.getBoundingClientRect().width))\"` is above `0`). Known failure (reproduced 2026-10-09, #18234): when the run has a conversation, the summary is 0 px wide at desktop and phone, so only the conversation title shows (the summary is still in `browser text`). Repeat at `browser viewport phone`. The status-less success case has no live producer (automation 1.19.0 with SDK 1.53.0 always sends a status), as `F23.run-task-outcome` records: `blocked`. Delete both automations afterwards.", "source_line": 126, "expected": [ "an automation whose latest Completed run reported a task outcome shows it on Home (#17682): `blocked` and `partial_success` read **Blocked** and **Partial**, `unknown` or an unrecognised status reads **Needs review**, each with an amber (`bg-warning`) health dot, in the row text, the row link's aria-label, the tooltip's Status line and the pinned card's strip. A finish response without a string status counts as success." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-09, #18234): when the run has a conversation, the summary is 0 px wide at desktop and phone, so only the conversation title shows (the summary is still in `browser text`). Repeat at `browser viewport phone`. The status-less success case has no live producer (automation 1.19.0 with SDK 1.53.0 always sends a status), as `F23.run-task-outcome` records: `blocked`. Delete both automations afterwards." ], "states": [ "failure", "blocked" ] }, { "name": "F03--view-and-edit", "title": "View and Edit", "ids": [ "F03.home-row-menu" ], "body": "From the Beta row menu, `control-openhands browser click '[data-testid^=\"running-automation-view-\"]' --expect-url '/automations/[^/?]+$'` opens the detail page. Back on `/` (`control-openhands browser goto /`), reopen the menu and click `[data-testid^=\"running-automation-edit-\"]`, then `control-openhands browser wait-url 'automations/setup\\?automationId='`: the setup page opens prefilled (no modal; `browser count 'text=Edit automation'` is `0`), `browser value 'testid=automation-setup-name'` is `QA_F03 Beta` and the prompt is the stored one (`browser screenshot --feature F03.home-row-menu --name edit-opens-setup`). `control-openhands browser fill 'testid=automation-setup-prompt' 'Reply with the single word: beta2. Do not run any tools.'` and `control-openhands browser click 'testid=automation-setup-save' --observe 'role=status'` (`Automation updated`); `api GET /api/automation/v1/` shows the new prompt. The page's own controls (Test, Close, the timeout limit) are F22's.", "raw": "**View and Edit (`F03.home-row-menu`).** From the Beta row menu, `control-openhands browser click '[data-testid^=\"running-automation-view-\"]' --expect-url '/automations/[^/?]+$'` opens the detail page. Back on `/` (`control-openhands browser goto /`), reopen the menu and click `[data-testid^=\"running-automation-edit-\"]`, then `control-openhands browser wait-url 'automations/setup\\?automationId='`: the setup page opens prefilled (no modal; `browser count 'text=Edit automation'` is `0`), `browser value 'testid=automation-setup-name'` is `QA_F03 Beta` and the prompt is the stored one (`browser screenshot --feature F03.home-row-menu --name edit-opens-setup`). `control-openhands browser fill 'testid=automation-setup-prompt' 'Reply with the single word: beta2. Do not run any tools.'` and `control-openhands browser click 'testid=automation-setup-save' --observe 'role=status'` (`Automation updated`); `api GET /api/automation/v1/` shows the new prompt. The page's own controls (Test, Close, the timeout limit) are F22's.", "source_line": 127, "expected": [ "the row kebab offers Run now, View, Edit, Turn off and Pin to dashboard; Run now toasts `Automation dispatched` and is disabled while a run is in flight; Edit opens the automation setup page `/automations/setup?automationId=` with the form prefilled (no modal), where **Save** toasts `Automation updated`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--turn-off", "title": "Turn off", "ids": [ "F03.turn-off-confirmation" ], "body": "From the Beta row menu click `[data-testid^=\"running-automation-turn-off-\"]`; `control-openhands browser snapshot 'role=dialog'` reads `Turn off \"QA_F03 Beta\"? It will stop running on its schedule until you turn it back on.` with Cancel and Turn off. `control-openhands browser click 'role=dialog >> role=button[name=\"Cancel\"]'` keeps it enabled; repeat and `control-openhands browser click 'role=dialog >> role=button[name=\"Turn off\"]'`, `browser reload`: `control-openhands browser count '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Beta'` is `0` and the API shows `enabled: false`. Turn it back on from `/automations` (Preconditions) for the pin checks.", "raw": "**Turn off (`F03.turn-off-confirmation`).** From the Beta row menu click `[data-testid^=\"running-automation-turn-off-\"]`; `control-openhands browser snapshot 'role=dialog'` reads `Turn off \"QA_F03 Beta\"? It will stop running on its schedule until you turn it back on.` with Cancel and Turn off. `control-openhands browser click 'role=dialog >> role=button[name=\"Cancel\"]'` keeps it enabled; repeat and `control-openhands browser click 'role=dialog >> role=button[name=\"Turn off\"]'`, `browser reload`: `control-openhands browser count '[data-testid^=\"running-automation-row-\"] >> has-text=QA_F03 Beta'` is `0` and the API shows `enabled: false`. Turn it back on from `/automations` (Preconditions) for the pin checks.", "source_line": 128, "expected": [ "Turn off asks `Turn off \"\"? It will stop running on its schedule until you turn it back on.`; Cancel keeps it on, Turn off disables it and drops it from home." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--pin", "title": "Pin", "ids": [ "F03.pinned-automations-grid" ], "body": "Run `control-openhands browser goto /` (turning Beta back on left `/automations`); from Pong's row menu click `[data-testid^=\"running-automation-pin-\"]`; `control-openhands browser text 'testid=pinned-automations-dashboard'` shows `Pinned automations`, `QA_F03 Pong`, the prompt, `cron` and the last status; `browser storage --values` holds the id under `oh:home-pinned-automations::-`. Pin Beta too; `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=pinned-automation-card-]')].map(e=>e.innerText.split('\\n')[0]).join(',')\"` is `QA_F03 Pong,QA_F03 Beta`. `control-openhands browser click '[data-testid^=\"pinned-automation-card-\"] >> role=link[name=\"QA_F03 Pong\"]' --expect-url '/automations/[^/?]+$'` opens the detail page.", "raw": "**Pin (`F03.pinned-automations-grid`).** Run `control-openhands browser goto /` (turning Beta back on left `/automations`); from Pong's row menu click `[data-testid^=\"running-automation-pin-\"]`; `control-openhands browser text 'testid=pinned-automations-dashboard'` shows `Pinned automations`, `QA_F03 Pong`, the prompt, `cron` and the last status; `browser storage --values` holds the id under `oh:home-pinned-automations::-`. Pin Beta too; `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=pinned-automation-card-]')].map(e=>e.innerText.split('\\n')[0]).join(',')\"` is `QA_F03 Pong,QA_F03 Beta`. `control-openhands browser click '[data-testid^=\"pinned-automation-card-\"] >> role=link[name=\"QA_F03 Pong\"]' --expect-url '/automations/[^/?]+$'` opens the detail page.", "source_line": 129, "expected": [ "**Pin to dashboard** adds a card to **Pinned automations** (prompt, pills, status strip, Runs); the title opens the detail page; a deleted automation's card disappears." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--pinned-menu", "title": "Pinned menu", "ids": [ "F03.pinned-card-menu" ], "body": "Run `control-openhands browser goto /` (the Pin bullet ends on Pong's detail page), then `control-openhands browser click '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Beta >> [data-testid^=\"pinned-automation-menu-\"]'` lists Run now, View, Edit, Turn off and Unpin from dashboard. `[data-testid^=\"pinned-automation-run-\"]` toasts `Automation dispatched`; reopen and `[data-testid^=\"pinned-automation-cancel-\"]` toasts `Automation run cancelled` (the run is `CANCELLED`). Reopen and click `[data-testid^=\"unpin-automation-\"]`; after `browser reload` only `QA_F03 Pong` is pinned and Beta stays in the list.", "raw": "**Pinned menu (`F03.pinned-card-menu`).** Run `control-openhands browser goto /` (the Pin bullet ends on Pong's detail page), then `control-openhands browser click '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Beta >> [data-testid^=\"pinned-automation-menu-\"]'` lists Run now, View, Edit, Turn off and Unpin from dashboard. `[data-testid^=\"pinned-automation-run-\"]` toasts `Automation dispatched`; reopen and `[data-testid^=\"pinned-automation-cancel-\"]` toasts `Automation run cancelled` (the run is `CANCELLED`). Reopen and click `[data-testid^=\"unpin-automation-\"]`; after `browser reload` only `QA_F03 Pong` is pinned and Beta stays in the list.", "source_line": 130, "expected": [ "a pinned card's kebab has the same actions with **Unpin from dashboard**." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--pinned-links", "title": "Pinned links", "ids": [ "F03.pinned-card-links" ], "body": "With Pong pinned and run once, `control-openhands browser click '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Pong >> [data-testid^=\"pinned-automation-activity-\"] >> role=link' --expect-url '/automations/[^/?]+\\?run='` opens the run. Back on `/` (`control-openhands browser goto /`), `control-openhands browser focus '[data-testid^=\"pinned-automation-card-\"] >> role=link[name=\"QA_F03 Beta\"]'`, `control-openhands browser press Enter`, `control-openhands browser wait-url '/automations/[^/?]+$'` opens Beta's detail page (pin Beta again first if it was unpinned).", "raw": "**Pinned links (`F03.pinned-card-links`).** With Pong pinned and run once, `control-openhands browser click '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Pong >> [data-testid^=\"pinned-automation-activity-\"] >> role=link' --expect-url '/automations/[^/?]+\\?run='` opens the run. Back on `/` (`control-openhands browser goto /`), `control-openhands browser focus '[data-testid^=\"pinned-automation-card-\"] >> role=link[name=\"QA_F03 Beta\"]'`, `control-openhands browser press Enter`, `control-openhands browser wait-url '/automations/[^/?]+$'` opens Beta's detail page (pin Beta again first if it was unpinned).", "source_line": 131, "expected": [ "a pinned card's sparkline bar opens `/automations/?run=`, and Enter on its focused title opens the detail page." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--reorder", "title": "Reorder", "ids": [ "F03.pinned-reorder-drag" ], "body": "Run `control-openhands browser goto /` (Pinned links ends on Beta's detail page). With Pong and Beta pinned (order `QA_F03 Pong,QA_F03 Beta`, side by side at desktop), read the Beta handle and Pong card boxes with `control-openhands browser bbox '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Beta >> [data-testid^=\"pinned-automation-drag-\"]'` and `control-openhands browser bbox '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Pong'`, then drag the handle onto the left fifth of Pong's card: `control-openhands browser drag '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Beta >> [data-testid^=\"pinned-automation-drag-\"]' --by -522,0 --steps 20` (at 1440×1000 the handle centre is x≈1065 and Pong's card spans x 486–864). The order eval from the Pin bullet reads `QA_F03 Beta,QA_F03 Pong`, `oh:home-pinned-automations::-` lists Beta's id first, and the order survives `browser reload`. Drag Pong's handle the same way to restore. The target form `browser drag [--position before]` does not reorder (the drop lands before the drag-over position is committed).", "raw": "**Reorder (`F03.pinned-reorder-drag`).** Run `control-openhands browser goto /` (Pinned links ends on Beta's detail page). With Pong and Beta pinned (order `QA_F03 Pong,QA_F03 Beta`, side by side at desktop), read the Beta handle and Pong card boxes with `control-openhands browser bbox '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Beta >> [data-testid^=\"pinned-automation-drag-\"]'` and `control-openhands browser bbox '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Pong'`, then drag the handle onto the left fifth of Pong's card: `control-openhands browser drag '[data-testid^=\"pinned-automation-card-\"] >> has-text=QA_F03 Beta >> [data-testid^=\"pinned-automation-drag-\"]' --by -522,0 --steps 20` (at 1440×1000 the handle centre is x≈1065 and Pong's card spans x 486–864). The order eval from the Pin bullet reads `QA_F03 Beta,QA_F03 Pong`, `oh:home-pinned-automations::-` lists Beta's id first, and the order survives `browser reload`. Drag Pong's handle the same way to restore. The target form `browser drag [--position before]` does not reorder (the drop lands before the drag-over position is committed).", "source_line": 132, "expected": [ "dragging a pinned card's title row (`Reorder `) onto another card reorders the pins, and the order survives a reload." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--backend-down", "title": "Backend down", "ids": [ "F03.automations-backend-down" ], "body": "Stop only this run's automation service with `control-openhands service stop automation`, then `control-openhands browser reload`. Once loading settles, `browser count` on `testid=running-automations-list` and `testid=pinned-automations-dashboard` is `0`, while `testid=recommended-automations-rail` is `1`: its cards come from the static catalog even when `/api/automation/health` and `/v1` return 502. The composer remains enabled. `control-openhands browser toasts --history` is `[]`. Click the Custom automation recommendation: it navigates to `/automations/new/custom-automation` and opens the **Before you start** dialog; the toast history is still `[]` (before #17838 a `Request failed with status code 502` toast appeared here). Capture both Home and the pre-setup dialog, then `control-openhands restart` and `browser reload`; doctor should show the owned automation service healthy again.\n\nPhone and cleanup:", "raw": "**Backend down (`F03.automations-backend-down`).** Stop only this run's automation service with `control-openhands service stop automation`, then `control-openhands browser reload`. Once loading settles, `browser count` on `testid=running-automations-list` and `testid=pinned-automations-dashboard` is `0`, while `testid=recommended-automations-rail` is `1`: its cards come from the static catalog even when `/api/automation/health` and `/v1` return 502. The composer remains enabled. `control-openhands browser toasts --history` is `[]`. Click the Custom automation recommendation: it navigates to `/automations/new/custom-automation` and opens the **Before you start** dialog; the toast history is still `[]` (before #17838 a `Request failed with status code 502` toast appeared here). Capture both Home and the pre-setup dialog, then `control-openhands restart` and `browser reload`; doctor should show the owned automation service healthy again.\n\nPhone and cleanup:", "source_line": 133, "expected": [ "with the automation service down, the installed list and pinned grid are hidden and the composer still works. The recommendation rail returns after loading from its static catalog; opening a setup card reaches its route and pre-setup dialog. The rail asks for the installed automations only once `/api/automation/health` answers ok (#17838), so home raises no error toast for the stopped service." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--phone", "title": "Phone", "ids": [ "F03.phone" ], "body": "`control-openhands browser viewport phone`, `control-openhands browser bbox 'testid=pinned-automations-dashboard'` and `... 'testid=running-automations-list'` (358 px wide, `pageHorizontalOverflow` `false`), `browser screenshot --feature F03.phone --name automations` (the banner was checked at phone width in the No-LLM banner bullet). Open Workspace: `control-openhands browser click 'testid=open-workspace-button'`, then `browser bbox 'testid=open-workspace-dialog-body'` is inside the viewport. Folder browser: arrange a long name first with `control-openhands fixture folder --name qa-f03-folder-with-a-name-too-long-for-a-phone` (it lands in `/workspace`), then `control-openhands browser click 'testid=workspace-dropdown'` and `control-openhands browser click 'testid=add-workspaces-button'`; `control-openhands browser eval \"(()=>{const b=document.querySelector('[data-testid=folder-browser-entry-workspace] span span.truncate');return b.getBoundingClientRect().width})()\"` is greater than `0` (about 67). Run `control-openhands browser click 'testid=folder-browser-up'` twice (now ``) and `control-openhands browser click 'testid=folder-browser-entry-workspace'`, then `control-openhands browser eval \"(()=>{const b=document.querySelector('[data-testid=folder-browser-entry-qa-f03-folder-with-a-name-too-long-for-a-phone] span span.truncate');return [b.scrollWidth>b.clientWidth,getComputedStyle(b).textOverflow]})()\"`: `[true,\"ellipsis\"]` (about 109 of 293 px shown). `control-openhands browser screenshot --feature F03.phone --name folder-names` shows each name beside `Folder` under separate `Name` and `Kind` headers, the long one cut to `qa-f03-folder-w…`. Close with `control-openhands browser click 'testid=folder-browser-cancel'` and `control-openhands browser click 'testid=close-open-workspace-dialog'`, then return with `control-openhands browser viewport desktop`.", "raw": "**Phone (`F03.phone`).** `control-openhands browser viewport phone`, `control-openhands browser bbox 'testid=pinned-automations-dashboard'` and `... 'testid=running-automations-list'` (358 px wide, `pageHorizontalOverflow` `false`), `browser screenshot --feature F03.phone --name automations` (the banner was checked at phone width in the No-LLM banner bullet). Open Workspace: `control-openhands browser click 'testid=open-workspace-button'`, then `browser bbox 'testid=open-workspace-dialog-body'` is inside the viewport. Folder browser: arrange a long name first with `control-openhands fixture folder --name qa-f03-folder-with-a-name-too-long-for-a-phone` (it lands in `/workspace`), then `control-openhands browser click 'testid=workspace-dropdown'` and `control-openhands browser click 'testid=add-workspaces-button'`; `control-openhands browser eval \"(()=>{const b=document.querySelector('[data-testid=folder-browser-entry-workspace] span span.truncate');return b.getBoundingClientRect().width})()\"` is greater than `0` (about 67). Run `control-openhands browser click 'testid=folder-browser-up'` twice (now ``) and `control-openhands browser click 'testid=folder-browser-entry-workspace'`, then `control-openhands browser eval \"(()=>{const b=document.querySelector('[data-testid=folder-browser-entry-qa-f03-folder-with-a-name-too-long-for-a-phone] span span.truncate');return [b.scrollWidth>b.clientWidth,getComputedStyle(b).textOverflow]})()\"`: `[true,\"ellipsis\"]` (about 109 of 293 px shown). `control-openhands browser screenshot --feature F03.phone --name folder-names` shows each name beside `Folder` under separate `Name` and `Kind` headers, the long one cut to `qa-f03-folder-w…`. Close with `control-openhands browser click 'testid=folder-browser-cancel'` and `control-openhands browser click 'testid=close-open-workspace-dialog'`, then return with `control-openhands browser viewport desktop`.", "source_line": 137, "expected": [ "at 390 px the banner, launcher, rail, pinned grid, list and the workspace dialogs fit without horizontal overflow, and folder names stay readable." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--cleanup", "title": "Cleanup", "ids": [ "F03.pinned-automations-grid" ], "body": "Delete both fixtures on `/automations` (`control-openhands browser goto /automations`; F21: kebab `role=list >> has-text=Export >> role=button[name=\"Delete\"]`, then `role=heading[name=\"Delete automation\"] >> xpath=.. >> role=button[name=\"Delete\"]`). On `/` (`control-openhands browser goto /`), `browser count 'testid=pinned-automations-dashboard'` is `0` and `browser text 'testid=running-automations-empty-hint'` is `No automations yet. Add one to automate repetitive work.`", "raw": "**Cleanup (`F03.pinned-automations-grid`).** Delete both fixtures on `/automations` (`control-openhands browser goto /automations`; F21: kebab `role=list >> has-text=Export >> role=button[name=\"Delete\"]`, then `role=heading[name=\"Delete automation\"] >> xpath=.. >> role=button[name=\"Delete\"]`). On `/` (`control-openhands browser goto /`), `browser count 'testid=pinned-automations-dashboard'` is `0` and `browser text 'testid=running-automations-empty-hint'` is `No automations yet. Add one to automate repetitive work.`", "source_line": 138, "expected": [ "**Pin to dashboard** adds a card to **Pinned automations** (prompt, pills, status strip, Runs); the title opens the detail page; a deleted automation's card disappears." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F03--implicit-launch-readiness", "title": "Implicit-launch readiness", "ids": [ "F03.composer-blocked-without-llm" ], "body": "Arrange named OpenHands agent `QA_PINNED` with `tools:[]`, concurrency7, and two LLM profiles on refused loopback `http://127.0.0.1:1/v1`: `QA_READY` has a disposable key and `QA_EMPTY` has no key. Use real `/api/profiles/` and `/api/agent-profiles/` writes, then activate the named agent by its ID. Pin it to QA_EMPTY and activate QA_READY; open Home and inspect the warning/pill. Invert the pinned/active profiles, reopen Home, fill `testid=chat-input` with a bounded no-tools prompt and read `browser enabled testid=submit-button`. Expected: readiness agrees with the effective LLM shown by the launch pill. Known failure (reproduced 2026-10-12, [#16951](https://github.com/OpenHands/OpenHands/issues/16951)): the readiness gate follows the named agent's pinned LLM, so a keyless pin can disable a key-present pill, and the inverse can permit a keyless pill. On current backends with per-launch LLM override, a launch uses the pill's LLM while preserving the named agent's identity, concurrency and tools (`F13.llm-pill-precedence`); that current behavior is separate from the readiness mismatch. Read only credential-presence booleans, without claiming provider authentication. Restore the default agent/active LLM and delete disposable profiles.", "raw": "**Implicit-launch readiness (`F03.composer-blocked-without-llm`).** Arrange named OpenHands agent `QA_PINNED` with `tools:[]`, concurrency7, and two LLM profiles on refused loopback `http://127.0.0.1:1/v1`: `QA_READY` has a disposable key and `QA_EMPTY` has no key. Use real `/api/profiles/` and `/api/agent-profiles/` writes, then activate the named agent by its ID. Pin it to QA_EMPTY and activate QA_READY; open Home and inspect the warning/pill. Invert the pinned/active profiles, reopen Home, fill `testid=chat-input` with a bounded no-tools prompt and read `browser enabled testid=submit-button`. Expected: readiness agrees with the effective LLM shown by the launch pill. Known failure (reproduced 2026-10-12, [#16951](https://github.com/OpenHands/OpenHands/issues/16951)): the readiness gate follows the named agent's pinned LLM, so a keyless pin can disable a key-present pill, and the inverse can permit a keyless pill. On current backends with per-launch LLM override, a launch uses the pill's LLM while preserving the named agent's identity, concurrency and tools (`F13.llm-pill-precedence`); that current behavior is separate from the readiness mismatch. Read only credential-presence booleans, without claiming provider authentication. Restore the default agent/active LLM and delete disposable profiles.", "source_line": 140, "expected": [ "with no LLM, Send is disabled and Enter creates nothing (the text stays editable)." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [#16951](https://github.com/OpenHands/OpenHands/issues/16951)): the readiness gate follows the named agent's pinned LLM, so a keyless pin can disable a key-present pill, and the inverse can permit a keyless pill. On current backends with per-launch LLM override, a launch uses the pill's LLM while preserving the named agent's identity, concurrency and tools (`F13.llm-pill-precedence`); that current behavior is separate from the readiness mismatch. Read only credential-presence booleans, without claiming provider authentication. Restore the default agent/active LLM and delete disposable profiles." ], "states": [ "failure" ] }, { "name": "F03--target-across-local-backends", "title": "Target across Local backends", "ids": [ "F03.selection-after-reload" ], "body": "In two run-owned Local backends, save distinct QA_A_ONLY and QA_B_ONLY workspaces through real `/api/workspaces` writes. Register the second backend through Add Backend with its key read from its private key file. On QA_Secondary Home, open the workspace dialog, choose QA_B_ONLY and Confirm; switch to Local through `backend-selector` without reload or `browser goto`. Read the preview, submit a bounded no-tools prompt through the real composer, and read the new Local conversation workspace. Expected: an incompatible target is cleared before submission. Known failure (reproduced 2026-10-12, [#16950](https://github.com/OpenHands/OpenHands/issues/16950)): QA_B_ONLY remains visible and its secondary-run path is submitted to Local. This is the Local variant; Cloud repository/organization recents still need an approved Cloud fixture.", "raw": "**Target across Local backends (`F03.selection-after-reload`).** In two run-owned Local backends, save distinct QA_A_ONLY and QA_B_ONLY workspaces through real `/api/workspaces` writes. Register the second backend through Add Backend with its key read from its private key file. On QA_Secondary Home, open the workspace dialog, choose QA_B_ONLY and Confirm; switch to Local through `backend-selector` without reload or `browser goto`. Read the preview, submit a bounded no-tools prompt through the real composer, and read the new Local conversation workspace. Expected: an incompatible target is cleared before submission. Known failure (reproduced 2026-10-12, [#16950](https://github.com/OpenHands/OpenHands/issues/16950)): QA_B_ONLY remains visible and its secondary-run path is submitted to Local. This is the Local variant; Cloud repository/organization recents still need an approved Cloud fixture.", "source_line": 142, "expected": [ "after a reload the preview bar is gone, but the dialog preselects the last workspace picked in its dropdown, confirmed or not: a pick closed with the X survives leaving for another page and coming back, and a reload." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [#16950](https://github.com/OpenHands/OpenHands/issues/16950)): QA_B_ONLY remains visible and its secondary-run path is submitted to Local. This is the Local variant; Cloud repository/organization recents still need an approved Cloud fixture." ], "states": [ "failure" ] }, { "name": "F03--saved-workspaces-across-local-backends", "title": "Saved workspaces across Local backends", "ids": [ "F03.workspace-dropdown-search" ], "body": "Warm the Local picker with QA_A_ONLY, close its dialog, switch through `backend-selector >> role=option[name=\"Connected QA_Secondary\"]`, then reopen the picker within60seconds, without reload or hard navigation. Expected: only QA_B_ONLY from the active server registry is offered. Known failure (reproduced 2026-10-12, [#16844](https://github.com/OpenHands/OpenHands/issues/16844)): the recorded2.7-second switch keeps QA_A_ONLY in the secondary picker while independent API readbacks show distinct registries. The same-absolute-parent directory variant requires isolated filesystems; two native runs share a host filesystem, so it and Windows remain unverified. These checks used frozen target `fa1f724ab5737a17453ad6081129eb91c5a423c6`, Agent Server/SDK1.54.0, automation1.19.3, run `backfill-18309-coordinator-checks` plus its owned `backfill-18309-local-secondary`. Retain `evidence/F03.selection-after-reload/16950-submission-readback.json`, `evidence/F03.workspace-dropdown-search/16844-local-switch.mp4` and registry readback; remove the registered backend/workspaces and stop both runs afterward.", "raw": "**Saved workspaces across Local backends (`F03.workspace-dropdown-search`).** Warm the Local picker with QA_A_ONLY, close its dialog, switch through `backend-selector >> role=option[name=\"Connected QA_Secondary\"]`, then reopen the picker within60seconds, without reload or hard navigation. Expected: only QA_B_ONLY from the active server registry is offered. Known failure (reproduced 2026-10-12, [#16844](https://github.com/OpenHands/OpenHands/issues/16844)): the recorded2.7-second switch keeps QA_A_ONLY in the secondary picker while independent API readbacks show distinct registries. The same-absolute-parent directory variant requires isolated filesystems; two native runs share a host filesystem, so it and Windows remain unverified. These checks used frozen target `fa1f724ab5737a17453ad6081129eb91c5a423c6`, Agent Server/SDK1.54.0, automation1.19.3, run `backfill-18309-coordinator-checks` plus its owned `backfill-18309-local-secondary`. Retain `evidence/F03.selection-after-reload/16950-submission-readback.json`, `evidence/F03.workspace-dropdown-search/16844-local-switch.mp4` and registry readback; remove the registered backend/workspaces and stop both runs afterward.", "source_line": 143, "expected": [ "typing filters workspaces; with more than one group, workspaces are grouped by parent with an **Other** group; no match (or no workspaces) shows `No workspaces yet`." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [#16844](https://github.com/OpenHands/OpenHands/issues/16844)): the recorded2.7-second switch keeps QA_A_ONLY in the secondary picker while independent API readbacks show distinct registries. The same-absolute-parent directory variant requires isolated filesystems; two native runs share a host filesystem, so it and Windows remain unverified. These checks used frozen target `fa1f724ab5737a17453ad6081129eb91c5a423c6`, Agent Server/SDK1.54.0, automation1.19.3, run `backfill-18309-coordinator-checks` plus its owned `backfill-18309-local-secondary`. Retain `evidence/F03.selection-after-reload/16950-submission-readback.json`, `evidence/F03.workspace-dropdown-search/16844-local-switch.mp4` and registry readback; remove the registered backend/workspaces and stop both runs afterward." ], "states": [ "failure" ] }, { "name": "F03--invalid-project-sub-agent", "title": "Invalid project sub-agent", "ids": [ "F03.launch-in-workspace" ], "body": "In an owned workspace create `.agents/agents/qa-bad-skill.md` with YAML frontmatter `name: qa-bad-skill`, `description: Bad skill`, `skills: qa-f27-missing-skill`, then body `Body`, without installing that skill. Use an explicit dummy profile pointed at a refused loopback endpoint with autotitle off for this pre-inference check. On Home use `control-openhands workspace open `, select `testid=workspace-mode-selector-option-local_repo`, fill `testid=chat-input` and click `testid=home-chat-launcher >> testid=submit-button` with `--expect-url \"/conversations/[^/]+$\"`. The invalid sub-agent is skipped with a warning; creation returns201 and opens its conversation instead of500. A later refused-loopback model error is separate from creation. Read `browser network --filter \"api/conversations($|\\?)\"`, capture `browser screenshot --feature F03.create-error-toast --name invalid-subagent`, then remove only the owned bad-agent file.", "raw": "**Invalid project sub-agent (`F03.launch-in-workspace`).** In an owned workspace create `.agents/agents/qa-bad-skill.md` with YAML frontmatter `name: qa-bad-skill`, `description: Bad skill`, `skills: qa-f27-missing-skill`, then body `Body`, without installing that skill. Use an explicit dummy profile pointed at a refused loopback endpoint with autotitle off for this pre-inference check. On Home use `control-openhands workspace open `, select `testid=workspace-mode-selector-option-local_repo`, fill `testid=chat-input` and click `testid=home-chat-launcher >> testid=submit-button` with `--expect-url \"/conversations/[^/]+$\"`. The invalid sub-agent is skipped with a warning; creation returns201 and opens its conversation instead of500. A later refused-loopback model error is separate from creation. Read `browser network --filter \"api/conversations($|\\?)\"`, capture `browser screenshot --feature F03.create-error-toast --name invalid-subagent`, then remove only the owned bad-agent file.", "source_line": 145, "expected": [ "submitting with a workspace picked shows `Creating conversation…`, opens `/conversations/`, and the agent runs in that folder." ], "notes": [ "Known issue [#15701](https://github.com/OpenHands/OpenHands/issues/15701) (`F03.launch-in-workspace`): Creating Conversation toast stuck visible. Live prerequisite: The original stuck creation request/outcome sequence; the report is intermittent and gives no deterministic trigger." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--contract-f03-open-repository-dialog", "title": "Open repository dialog", "ids": [ "F03.open-repository-dialog" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "on a Cloud backend, **Open repository** picks provider, repository and branch (blocked locally)." ], "notes": [ "Known issue [#17108](https://github.com/OpenHands/OpenHands/issues/17108) (`F03.open-repository-dialog`): The repository picker lists a search result for input the user has already cleared. Live prerequisite: An authenticated Cloud Git provider and a real repository search response that remains pending after clearing the input.", "Known issue [#18480](https://github.com/OpenHands/OpenHands/issues/18480) (`F03.open-repository-dialog`): an Azure DevOps Enterprise user belonging to several organizations can see only one organization's repositories. Live prerequisite: a real Cloud/Enterprise sign-in with two organizations and distinct repositories; the report's mocked two-organization capture is not a real-backend reproduction." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F03--contract-f03-isolated-workspace-notice", "title": "Isolated workspace notice", "ids": [ "F03.isolated-workspace-notice" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "on an isolated-runtime backend a status line explains the new isolated workspace and offers **Clear host workspace** (blocked locally)." ], "notes": [], "annotations": [], "states": [ "contract" ] }, { "name": "F03--contract-f03-workspaces-unsupported", "title": "Workspaces unsupported", "ids": [ "F03.workspaces-unsupported" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "on an Agent Server without workspace APIs the pill is disabled with a tooltip (blocked locally). Automations on home:" ], "notes": [], "annotations": [], "states": [ "contract" ] }, { "name": "F03--contract-f03-recommended-prompt-launch", "title": "Recommended prompt launch", "ids": [ "F03.recommended-prompt-launch" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "a card with no setup form and no missing integration creates a conversation that sends the automation prompt (blocked locally)." ], "notes": [], "annotations": [], "states": [ "contract" ] }, { "name": "F03--contract-f03-home-automations-view-more", "title": "Home automations view more", "ids": [ "F03.home-automations-view-more" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "past 10 rows **View more**/**View all**, past 6 pins **View more**/**View less** (not arranged)." ], "notes": [], "annotations": [], "states": [ "contract" ] } ] }, { "id": "F04", "title": "Conversation list and folders", "file": "F04-conversation-list.md", "sha256": "58cf085074029b7ced5eb8a1f81f26ad71b3253d4161c7ea7344a706669a7861", "overview": "The expanded sidebar always shows the user's conversations under a\n**Conversations** header. Each row opens its conversation and carries a pin and\na ⋮ menu (Rename, Edit tags, Stop, Download, Archive or Unarchive, Delete). The\nheader has a folder-plus picker that starts a conversation in a chosen\nworkspace and a **View** menu with layout presets, tag filters, **More\noptions** (organize, sort, scope, archived, age cutoff, automation filter, row\nmetadata) and **Delete all**. Grouped by workspace, folders collapse, preview\nfive rows and start new conversations in place. Preferences, pins, archives and\nthe workspace each conversation was started in live in this browser's\nlocalStorage.", "entry": "- Always visible in the expanded sidebar on every page (`/`, the direct URL `/conversations`, `/conversations/`, settings pages): the **Conversations** section under New Chat / Customize / Automate. At phone width it is inside the drawer opened by `sidebar-mobile-menu-toggle`.\n- Header: folder-plus (`conversation-panel-new-thread-picker`, aria \"Create thread folder — choose workspace or repository\"), **View** (`conversation-layouts-toggle`), and in grouped mode the **Conversations** label itself (`conversations-header-toggle`).\n- Row hover: pin (`conversation-pin-toggle-`) and ⋮ (`ellipsis-button`, aria \"More options\"), which opens `context-menu`. A row with more tags than fit shows a `+N` chip (`conversation-card-tag-overflow`). The Pinned section has its own More/Less (`conversation-panel-pinned-view-more`).\n- View → **More options** (`advanced-options-row`) opens `advanced-conversation-options-modal` (Organize, Sort by, Threads `scope-all`/`scope-relevant`, Additional, Automations, Metadata).\n- Grouped mode: folder heading (`thread-folder-drag-`), folder `+` (aria \"Start new conversation in \"), folder **More/Less** (`thread-folder-view-more-`).\n- No command-menu entry or keyboard shortcut exists for the list.", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done), desktop viewport, `deepseek-flash` active for the one prompted conversation.\n- `F04.empty-state` and the disabled Delete all are first checked on a run with no conversations (a fresh `launch --new`), and again after `F04.delete-all`.\n- Workspaces: `control-openhands fixture git-repo --name qa-f04-repo` and `control-openhands fixture folder --name qa-f04-folder`, then add both through the picker (F03 owns the folder browser): `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=add-workspaces-button'`, `control-openhands browser click 'testid=folder-browser-up'` twice (from `/private/home` to the run directory; check with `browser text 'testid=folder-browser-current-path'`), `control-openhands browser dblclick 'testid=folder-browser-entry-workspace'`, `control-openhands browser click 'testid=folder-browser-add-all-subdirs'`. The picker popover stays open afterwards; `control-openhands browser click 'testid=conversation-panel-new-thread-picker'` closes it.\n- A finished conversation, started right after the View menu bullet (the first two bullets need a run with no conversations): `control-openhands conversation start --prompt \"Reply with only the word pong. Do not run any tools.\" --wait --timeout 240`. The model writes the title (`✅ Reply only with pong`, `✅ Reply with pong only`, ...): select its row with `has-text=pong`.\n- `F04.load-more` and `F04.folder-preview-more` need more than 20 conversations, at least 6 in one folder. Empty conversations from the folder `+` cost no model call (see `F04.folder-new-conversation`).\n- `F04.cloud-picker` and `F04.start-task-cards` are blocked: they need an OpenHands Cloud backend with a connected git provider. The kept rows of `F04.automation-filter` (Only automation runs with real runs) and the hiding effect of `F04.hide-older` need data a fresh run cannot have (an automation-run conversation; a conversation older than one hour).", "gotchas": "- A plain `--expect-url '/conversations/'` returns at once when the browser is already on a conversation page. Start creation clicks from `/` (`browser goto /`), and never navigate away before the click returns: leaving early skips the client-side metadata write, so the new conversation lands in **No workspace** although the server has its working directory.\n- Workspace grouping, pins, archives and preferences live in this browser's localStorage (`openhands-agent-server-conversation-metadata`, `pinned-conversations`, `archived-conversations`, `conversation-panel-preferences`). `browser reset` loses them: archived rows reappear and every conversation moves to No workspace.\n- Folder test ids embed the sanitized absolute workspace path (`thread-folder-ws--tmp-...-qa-f04-repo`). Select with `[data-testid^=\"thread-folder-ws-\"][data-testid$=\"\"]` or by the `+` button's accessible name.\n- Row links carry `?backend=default-local`; anchor URL regexes without `$`.\n- Hover and ⋮ need the row on screen: `browser scroll ''` first, and `browser hover` before clicking `ellipsis-button` or the pin.\n- Escape closes the ⋮ menu and returns focus to its ⋮ button (#18044); an outside `browser mouse-click` closes it too. If Tab has moved focus to the next row's link, that row's preview tooltip opens and takes the first Escape, so press Escape twice. Inside the phone drawer the first Escape closes the menu and the next one the drawer.\n- The Stop, Delete and Delete all dialogs name their buttons `cancel-button` and `confirm-button`, like the Archive and Edit tags dialogs (#17915, fixed by #18151); `role=dialog >> role=button[name=\"...\"]` still works.\n- With the drawer open at phone width there are two `conversation-panel` elements; scope with `testid=sidebar-mobile-drawer >> ...`.\n- Recent is exactly the fresh default layout (By date, Updated, all threads, old conversations shown), so a new user sees Recent checked. Active and Compact also turn on Hide old conversations: Recent plus a manual Threads → Active only reads `More options · Custom`, not Active. Presets never change the hide-older cutoff.\n- In By workspace mode Load more often adds no visible row: later rows join existing folders behind their More button.\n- By workspace never shows `No conversations found` while any workspace is registered: empty workspace folders stay.\n- Pinned rows bypass tag and automation filters and suppress the filtered empty states; unpin before asserting them.\n- The page size is 20; filters, Active scope and archives apply only to loaded pages.\n- The ⋮ menu offers Stop for idle and finished conversations too (they count as active); it disappears only once paused.\n- `service stop agent-server` needs `control-openhands restart` and a `browser reload` before anything else works.\n- The list refreshes on an interval: a row keeps `conversation-status-working` and `Conversation ` for up to about 40 s after the conversation finished. Use `browser wait ... --timeout 60000`, not an immediate `count`.\n- Model-written titles vary from run to run; match finished conversations on a word from the prompt (`has-text=pong`), never on the whole title.\n- Opening a conversation from the sidebar never changes its tags, also right after an automation setup conversation (one tagged `automationsetup: draft`): the other conversation keeps its tags and opens as a normal chat (recipe in F07 `F07.open-by-url`). A conversation that did get the tag loses it with Edit tags (`remove-tag-automationsetup`).\n- Known issue #17919: Delete all deletes only the loaded pages, and its dialog states that smaller count (`F04.delete-all`).\n- Known issue [#18155](https://github.com/OpenHands/OpenHands/issues/18155) (`F04.show-archived-unarchive`): Missing-runtime conversations shown as Archived but never hidden, chipped, or unarchivable. Live prerequisite: A Cloud sandbox or local runtime that becomes MISSING and non-resumable; ordinary local pause remains resumable.\n- Known issue [#17497](https://github.com/OpenHands/OpenHands/issues/17497) (`F04.card`): Corrupted conversation data breaks the entire conversation history list instead of handling individual broken conversations. Live prerequisite: The original corrupted persisted conversation or a deterministic corruption fixture after model-switch failure and fork.\n- Known issue [#17257](https://github.com/OpenHands/OpenHands/issues/17257) (`F04.start-task-cards`): Cloud provisioning has no in-progress start-task cards because the list hook always returns an empty array. Live prerequisite: An authenticated Cloud backend producing an in-progress sandbox/repository start task.\n- Known issue [#16750](https://github.com/OpenHands/OpenHands/issues/16750) (`F04.card`): Conversation list does not load while automation is running. Live prerequisite: A prompt automation or native delegated task that remains running during conversation-list reload; no usable model is configured here.\n- Known issue [#15946](https://github.com/OpenHands/OpenHands/issues/15946) (`F04.card`): Show conversations hidden by active/stale agent-server leases. Live prerequisite: A copied lease-bearing persisted conversation and second owner/expired-lease state; never alter another live run's state.\n- Known issue [OpenHands/software-agent-sdk#4987](https://github.com/OpenHands/software-agent-sdk/issues/4987) (`F04.card`): Autotitle input has broken user-message separators and emoji-list formatting; verify the resulting visible title with a real provider. Live prerequisite: A working autotitle model and captured title-generation input/output; Canvas only consumes the resulting server title.\n- Known issue [OpenHands/software-agent-sdk#4537](https://github.com/OpenHands/software-agent-sdk/issues/4537) (`F04.card`): A native delegated task can freeze the sidebar conversation list and cause request timeouts until it ends. Live prerequisite: A working model with native TaskTool delegation holding the parent state lock during list reload.\n- Known issue [OpenHands/software-agent-sdk#4080](https://github.com/OpenHands/software-agent-sdk/issues/4080) (`F04.card`): One unregistered persisted tool observation can make a whole conversation disappear from the list and return404. Live prerequisite: A copied persisted conversation with its original unregistered observation kind and missing tool-module registration.", "features": [ { "id": "F04.empty-state", "description": "with no conversations (By date) the list shows `No conversations found`." }, { "id": "F04.view-menu", "description": "the View menu lists the presets By workspace, Recent, Active and Compact (a fresh profile has Recent checked), Filter by tags (`No tags yet` when empty), More options and Delete all (disabled with no conversations); Escape closes it." }, { "id": "F04.new-thread-picker", "description": "the folder-plus button lists No workspace, every known workspace, + Add Workspace and Manage Workspaces; picking a row starts a conversation there and opens it." }, { "id": "F04.card", "description": "a row shows a status dot, the title and the relative time, links to `/conversations/`, and the open conversation's row is highlighted." }, { "id": "F04.card-live-status", "description": "while the agent runs, its row shows the working indicator; once it finishes the row switches to the check mark and the generated title on the next list refresh." }, { "id": "F04.list-header", "description": "the header divider appears only while the list is scrolled." }, { "id": "F04.organize", "description": "More options → Organize switches between folders (By workspace) and a flat list (By date); the choice survives a reload." }, { "id": "F04.grouped-folders", "description": "By workspace shows one folder per known workspace (empty ones included) plus a No workspace folder while any loaded row has no workspace, in any browser." }, { "id": "F04.folder-collapse", "description": "a folder heading collapses and expands that folder." }, { "id": "F04.collapse-all", "description": "the Conversations header collapses every folder, then expands them all." }, { "id": "F04.folder-new-conversation", "description": "a folder's `+` starts a conversation in that workspace and opens it." }, { "id": "F04.folder-preview-more", "description": "a folder shows five rows; More shows every loaded row, Less returns to five." }, { "id": "F04.folder-reorder", "description": "dragging a folder heading onto another reorders the folders and the order persists." }, { "id": "F04.thread-scope", "description": "More options → Threads → Active only hides paused conversations; All threads shows them again." }, { "id": "F04.load-more", "description": "20 conversations load first; Load more fetches the rest and disappears when nothing is left." }, { "id": "F04.sort", "description": "Sort by Updated or Created reorders the list." }, { "id": "F04.presets", "description": "a preset is checked and survives a reload; Active and Compact hide paused conversations; any manual change reads `More options · Custom`." }, { "id": "F04.hide-older", "description": "Hide old conversations offers four cutoffs, persists the choice and hides conversations older than it." }, { "id": "F04.automation-filter", "description": "More options → Automations shows all, hides or keeps only automation runs, with its own empty message." }, { "id": "F04.automation-setup-tag", "description": "a conversation started from the automation setup page shows an `Automation setup: Draft` chip (lightning icon) first in its tag row; Filter by tags offers a facet and filter chip of the same name, and the hover preview lists `Automation setup` `draft`." }, { "id": "F04.tag-filter", "description": "choosing a tag facet narrows the list to rows with that tag (pins exempt) and survives a reload." }, { "id": "F04.filter-chips", "description": "active filters show as chips above the list; a chip click removes one, Clear all removes every chip." }, { "id": "F04.metadata-toggles", "description": "Repo and branch, Agent / model and Show tags add or remove row chips." }, { "id": "F04.hover-preview", "description": "hovering a row for a second shows title, directory, model and creation time, unless Details on hover is off." }, { "id": "F04.pin", "description": "the pin moves a row into a Pinned section above the list, survives a reload and unpins from there." }, { "id": "F04.pinned-preview-more", "description": "the Pinned section previews five rows; its More shows every pinned row and Less returns to five." }, { "id": "F04.card-rename", "description": "⋮ → Rename edits the title inline; Enter saves, a toast confirms, and the title persists." }, { "id": "F04.edit-tags", "description": "⋮ → Edit tags adds and removes `key: value` tags with validation; Save shows `Tags updated` and the chips persist." }, { "id": "F04.edit-tags-remove", "description": "in Edit tags each row's remove button drops that tag; Save persists the removal on the server." }, { "id": "F04.tag-overflow", "description": "tags that do not fit on a row collapse into a `+N` chip (aria `Show N more tags`) whose popover lists the hidden tags without opening the conversation; Escape closes it." }, { "id": "F04.stop", "description": "⋮ → Stop Conversation asks for confirmation and pauses the conversation; the item is gone once paused." }, { "id": "F04.download", "description": "⋮ → Download conversation data saves `conversation_.zip`." }, { "id": "F04.download-error", "description": "a failed download shows one toast and closes the ⋮ menu. When the Agent Server is down behind a plain-text 502 ingress response, the toast uses the generic error fallback rather than raw transport text." }, { "id": "F04.archive", "description": "⋮ → Archive Conversation asks for confirmation, hides and unpins the row, and leaves the conversation if it was open." }, { "id": "F04.show-archived-unarchive", "description": "Show archived brings archived rows back with an `Archived` chip; Unarchive restores a row without confirmation." }, { "id": "F04.delete", "description": "⋮ → Delete Conversation asks for confirmation, deletes it on the server and leaves it if it was open." }, { "id": "F04.delete-all", "description": "View → Delete all asks for confirmation with a count and deletes every conversation." }, { "id": "F04.collapsed-rail", "description": "the collapsed sidebar rail shows no conversation list; expanding brings it back." }, { "id": "F04.phone", "description": "at 390 px the list lives in the sidebar drawer, fits the viewport, and picking a row closes the drawer." }, { "id": "F04.cloud-picker", "description": "on a Cloud backend the folder-plus picker shows provider tabs and a repository search; a picked repository starts a conversation." }, { "id": "F04.start-task-cards", "description": "on a Cloud backend in-progress start tasks show as cards linking to `/conversations/task-`." } ], "scenarios": [ { "name": "F04--empty-list", "title": "Empty list", "ids": [ "F04.empty-state" ], "body": "On a run with no conversations, run `control-openhands browser text 'testid=conversation-panel-empty-state'`. The text is `No conversations found`.", "raw": "**Empty list (`F04.empty-state`).** On a run with no conversations, run `control-openhands browser text 'testid=conversation-panel-empty-state'`. The text is `No conversations found`.", "source_line": 80, "expected": [ "with no conversations (By date) the list shows `No conversations found`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--view-menu", "title": "View menu", "ids": [ "F04.view-menu" ], "body": "Run `control-openhands browser click 'testid=conversation-layouts-toggle'` and `control-openhands browser snapshot 'testid=conversation-layouts-menu'`: radios `By workspace`, `Recent` (`[checked]`), `Active`, `Compact`, items `Filter by tags`, `More options` and `Delete all` (`control-openhands browser attr 'testid=layout-preset-recent-activity' aria-checked` is `true`). With no conversations `control-openhands browser enabled 'testid=delete-all-conversations'` is `false`; after `control-openhands browser click 'testid=tag-filters-section'`, `control-openhands browser text 'testid=tag-filters-empty'` is `No tags yet`. `control-openhands browser press Escape` closes the menu (`browser count 'testid=conversation-layouts-menu'` is `0`).", "raw": "**View menu (`F04.view-menu`).** Run `control-openhands browser click 'testid=conversation-layouts-toggle'` and `control-openhands browser snapshot 'testid=conversation-layouts-menu'`: radios `By workspace`, `Recent` (`[checked]`), `Active`, `Compact`, items `Filter by tags`, `More options` and `Delete all` (`control-openhands browser attr 'testid=layout-preset-recent-activity' aria-checked` is `true`). With no conversations `control-openhands browser enabled 'testid=delete-all-conversations'` is `false`; after `control-openhands browser click 'testid=tag-filters-section'`, `control-openhands browser text 'testid=tag-filters-empty'` is `No tags yet`. `control-openhands browser press Escape` closes the menu (`browser count 'testid=conversation-layouts-menu'` is `0`).", "source_line": 81, "expected": [ "the View menu lists the presets By workspace, Recent, Active and Compact (a fresh profile has Recent checked), Filter by tags (`No tags yet` when empty), More options and Delete all (disabled with no conversations); Escape closes it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--start-in-a-workspace", "title": "Start in a workspace", "ids": [ "F04.new-thread-picker" ], "body": "Run `control-openhands browser goto /`, `control-openhands browser click 'testid=conversation-panel-new-thread-picker'` and `control-openhands browser snapshot 'testid=conversation-panel-new-thread-popover'`: buttons `No workspace`, `qa-f04-folder`, `qa-f04-repo`, `+ Add Workspace`, `Manage Workspaces`. Then `control-openhands browser click 'testid=conversation-panel-new-thread-popover >> testid=launch-workspace >> has-text=qa-f04-repo' --expect-url '/conversations/'` returns `/conversations/`, and `control-openhands conversation list` shows that id with `workspace` ending in `/workspace/qa-f04-repo`. From `/` again, `testid=launch-no-workspace` opens a new conversation whose row sits in the No workspace folder. `control-openhands browser click 'testid=manage-workspaces-button'` opens `testid=manage-workspaces-modal` (F03); Escape closes it. Immediately after creating that empty workspace conversation, `control-openhands conversation status ` reads `idle`; inspecting the `conversation-status-active` class shows a green dot with no animation class. Known failure (reproduced 2026-10-12, [#16752](https://github.com/OpenHands/OpenHands/issues/16752)): this empty idle row retains active green styling instead of the expected neutral state; the older report's flashing animation was not observed.", "raw": "**Start in a workspace (`F04.new-thread-picker`).** Run `control-openhands browser goto /`, `control-openhands browser click 'testid=conversation-panel-new-thread-picker'` and `control-openhands browser snapshot 'testid=conversation-panel-new-thread-popover'`: buttons `No workspace`, `qa-f04-folder`, `qa-f04-repo`, `+ Add Workspace`, `Manage Workspaces`. Then `control-openhands browser click 'testid=conversation-panel-new-thread-popover >> testid=launch-workspace >> has-text=qa-f04-repo' --expect-url '/conversations/'` returns `/conversations/`, and `control-openhands conversation list` shows that id with `workspace` ending in `/workspace/qa-f04-repo`. From `/` again, `testid=launch-no-workspace` opens a new conversation whose row sits in the No workspace folder. `control-openhands browser click 'testid=manage-workspaces-button'` opens `testid=manage-workspaces-modal` (F03); Escape closes it. Immediately after creating that empty workspace conversation, `control-openhands conversation status ` reads `idle`; inspecting the `conversation-status-active` class shows a green dot with no animation class. Known failure (reproduced 2026-10-12, [#16752](https://github.com/OpenHands/OpenHands/issues/16752)): this empty idle row retains active green styling instead of the expected neutral state; the older report's flashing animation was not observed.", "source_line": 82, "expected": [ "the folder-plus button lists No workspace, every known workspace, + Add Workspace and Manage Workspaces; picking a row starts a conversation there and opens it." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [#16752](https://github.com/OpenHands/OpenHands/issues/16752)): this empty idle row retains active green styling instead of the expected neutral state; the older report's flashing animation was not observed." ], "states": [ "failure" ] }, { "name": "F04--row-opens-its-conversation", "title": "Row opens its conversation", "ids": [ "F04.card" ], "body": "From `/` (`control-openhands browser goto /`; the previous bullet's launch left a conversation page, where `--expect-url '/conversations/'` would pass at once) run `control-openhands browser click 'testid=conversation-card >> has-text=pong' --expect-url '/conversations/'`, then `control-openhands browser attr 'testid=conversation-card >> has-text=pong' data-active` (`true`; other rows `false`). `control-openhands browser eval \"document.querySelector('[data-testid=conversation-card][data-active=true]').closest('a').getAttribute('href')\"` is `/conversations/?backend=default-local`. The finished conversation shows `testid=conversation-status-check`, idle ones `conversation-status-active` (green), paused ones `conversation-status-paused` (grey). The reopened chat has its transcript back: `control-openhands browser wait 'testid=agent-message'`, then `control-openhands browser text 'testid=agent-message >> nth=-1'` is `pong` and `control-openhands browser text 'testid=user-message >> nth=-1'` is the prompt, with `control-openhands browser count 'testid=error-message-banner'` `0` (`browser screenshot --feature F04.card --name reopened`).", "raw": "**Row opens its conversation (`F04.card`).** From `/` (`control-openhands browser goto /`; the previous bullet's launch left a conversation page, where `--expect-url '/conversations/'` would pass at once) run `control-openhands browser click 'testid=conversation-card >> has-text=pong' --expect-url '/conversations/'`, then `control-openhands browser attr 'testid=conversation-card >> has-text=pong' data-active` (`true`; other rows `false`). `control-openhands browser eval \"document.querySelector('[data-testid=conversation-card][data-active=true]').closest('a').getAttribute('href')\"` is `/conversations/?backend=default-local`. The finished conversation shows `testid=conversation-status-check`, idle ones `conversation-status-active` (green), paused ones `conversation-status-paused` (grey). The reopened chat has its transcript back: `control-openhands browser wait 'testid=agent-message'`, then `control-openhands browser text 'testid=agent-message >> nth=-1'` is `pong` and `control-openhands browser text 'testid=user-message >> nth=-1'` is the prompt, with `control-openhands browser count 'testid=error-message-banner'` `0` (`browser screenshot --feature F04.card --name reopened`).", "source_line": 83, "expected": [ "a row shows a status dot, the title and the relative time, links to `/conversations/`, and the open conversation's row is highlighted." ], "notes": [ "Known issue [#17497](https://github.com/OpenHands/OpenHands/issues/17497) (`F04.card`): Corrupted conversation data breaks the entire conversation history list instead of handling individual broken conversations. Live prerequisite: The original corrupted persisted conversation or a deterministic corruption fixture after model-switch failure and fork.", "Known issue [#16750](https://github.com/OpenHands/OpenHands/issues/16750) (`F04.card`): Conversation list does not load while automation is running. Live prerequisite: A prompt automation or native delegated task that remains running during conversation-list reload; no usable model is configured here.", "Known issue [#15946](https://github.com/OpenHands/OpenHands/issues/15946) (`F04.card`): Show conversations hidden by active/stale agent-server leases. Live prerequisite: A copied lease-bearing persisted conversation and second owner/expired-lease state; never alter another live run's state.", "Known issue [OpenHands/software-agent-sdk#4987](https://github.com/OpenHands/software-agent-sdk/issues/4987) (`F04.card`): Autotitle input has broken user-message separators and emoji-list formatting; verify the resulting visible title with a real provider. Live prerequisite: A working autotitle model and captured title-generation input/output; Canvas only consumes the resulting server title.", "Known issue [OpenHands/software-agent-sdk#4537](https://github.com/OpenHands/software-agent-sdk/issues/4537) (`F04.card`): A native delegated task can freeze the sidebar conversation list and cause request timeouts until it ends. Live prerequisite: A working model with native TaskTool delegation holding the parent state lock during list reload.", "Known issue [OpenHands/software-agent-sdk#4080](https://github.com/OpenHands/software-agent-sdk/issues/4080) (`F04.card`): One unregistered persisted tool observation can make a whole conversation disappear from the list and return404. Live prerequisite: A copied persisted conversation with its original unregistered observation kind and missing tool-module registration." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F04--live-status", "title": "Live status", "ids": [ "F04.card-live-status" ], "body": "Run `control-openhands conversation start --prompt \"Reply with only the word ping. Do not run any tools.\"` (no `--wait`; one tiny model call), then `control-openhands browser wait 'testid=conversation-panel >> testid=conversation-status-working' --timeout 20000`: the active row (`Conversation `) shows the working indicator. After `control-openhands conversation wait --timeout 180` reports `finished`, `control-openhands browser wait 'testid=conversation-panel >> [data-testid=conversation-card][data-active=true] >> testid=conversation-status-check' --timeout 60000` succeeds and `control-openhands browser text 'testid=conversation-panel >> [data-testid=conversation-card][data-active=true] >> testid=conversation-card-title'` is the generated title (it varies: `✅ Reply with only ping`, `✅ Reply only ping, no tools`). The row lags the conversation by up to about 40 s (list refresh).", "raw": "**Live status (`F04.card-live-status`).** Run `control-openhands conversation start --prompt \"Reply with only the word ping. Do not run any tools.\"` (no `--wait`; one tiny model call), then `control-openhands browser wait 'testid=conversation-panel >> testid=conversation-status-working' --timeout 20000`: the active row (`Conversation `) shows the working indicator. After `control-openhands conversation wait --timeout 180` reports `finished`, `control-openhands browser wait 'testid=conversation-panel >> [data-testid=conversation-card][data-active=true] >> testid=conversation-status-check' --timeout 60000` succeeds and `control-openhands browser text 'testid=conversation-panel >> [data-testid=conversation-card][data-active=true] >> testid=conversation-card-title'` is the generated title (it varies: `✅ Reply with only ping`, `✅ Reply only ping, no tools`). The row lags the conversation by up to about 40 s (list refresh).", "source_line": 84, "expected": [ "while the agent runs, its row shows the working indicator; once it finishes the row switches to the check mark and the generated title on the next list refresh." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--group-by-workspace", "title": "Group by workspace", "ids": [ "F04.grouped-folders" ], "body": "Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=layout-preset-by-workspace'`, then `control-openhands browser snapshot 'testid=conversation-panel'`. Regions `No workspace`, `qa-f04-repo` and `qa-f04-folder` each list their own conversations; a workspace without conversations still gets a folder. Second view in another browser: `control-openhands browser reset`, `control-openhands onboard --skip`, choose By workspace again; the workspace conversations must still be in their folders (currently they all fall into No workspace, a known failure).", "raw": "**Group by workspace (`F04.grouped-folders`).** Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=layout-preset-by-workspace'`, then `control-openhands browser snapshot 'testid=conversation-panel'`. Regions `No workspace`, `qa-f04-repo` and `qa-f04-folder` each list their own conversations; a workspace without conversations still gets a folder. Second view in another browser: `control-openhands browser reset`, `control-openhands onboard --skip`, choose By workspace again; the workspace conversations must still be in their folders (currently they all fall into No workspace, a known failure).", "source_line": 85, "expected": [ "By workspace shows one folder per known workspace (empty ones included) plus a No workspace folder while any loaded row has no workspace, in any browser." ], "notes": [], "annotations": [ "known failure)." ], "states": [ "failure" ] }, { "name": "F04--collapse-one-folder", "title": "Collapse one folder", "ids": [ "F04.folder-collapse" ], "body": "Run `control-openhands browser click '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-repo\"]'`, `control-openhands browser attr '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-repo\"]' aria-expanded` (`false`) and `control-openhands browser count '[data-testid^=\"thread-folder-ws-\"][data-testid$=\"qa-f04-repo\"] >> testid=conversation-card'` (`0`). Click again: `true` and the cards return.", "raw": "**Collapse one folder (`F04.folder-collapse`).** Run `control-openhands browser click '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-repo\"]'`, `control-openhands browser attr '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-repo\"]' aria-expanded` (`false`) and `control-openhands browser count '[data-testid^=\"thread-folder-ws-\"][data-testid$=\"qa-f04-repo\"] >> testid=conversation-card'` (`0`). Click again: `true` and the cards return.", "source_line": 86, "expected": [ "a folder heading collapses and expands that folder." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--collapse-all", "title": "Collapse all", "ids": [ "F04.collapse-all" ], "body": "Run `control-openhands browser click 'testid=conversations-header-toggle'`, `control-openhands browser attr 'testid=conversations-header-toggle' aria-expanded` (`false`) and `control-openhands browser count 'testid=conversation-panel-list-scroll >> testid=conversation-card'` (`0`). Click again: `true` and every card is back.", "raw": "**Collapse all (`F04.collapse-all`).** Run `control-openhands browser click 'testid=conversations-header-toggle'`, `control-openhands browser attr 'testid=conversations-header-toggle' aria-expanded` (`false`) and `control-openhands browser count 'testid=conversation-panel-list-scroll >> testid=conversation-card'` (`0`). Click again: `true` and every card is back.", "source_line": 87, "expected": [ "the Conversations header collapses every folder, then expands them all." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--new-conversation-in-a-folder", "title": "New conversation in a folder", "ids": [ "F04.folder-new-conversation" ], "body": "Run `control-openhands browser goto /`, then `control-openhands browser click 'role=button[name=\"Start new conversation in qa-f04-folder\"]' --expect-url '/conversations/'` (returns the new `/conversations/`), then `control-openhands browser count '[data-testid^=\"thread-folder-ws-\"][data-testid$=\"qa-f04-folder\"] >> testid=conversation-card'`; it grows by one. Repeating `browser goto /` plus the click for `qa-f04-repo` 18 times arranges the data for the next two bullets.", "raw": "**New conversation in a folder (`F04.folder-new-conversation`).** Run `control-openhands browser goto /`, then `control-openhands browser click 'role=button[name=\"Start new conversation in qa-f04-folder\"]' --expect-url '/conversations/'` (returns the new `/conversations/`), then `control-openhands browser count '[data-testid^=\"thread-folder-ws-\"][data-testid$=\"qa-f04-folder\"] >> testid=conversation-card'`; it grows by one. Repeating `browser goto /` plus the click for `qa-f04-repo` 18 times arranges the data for the next two bullets.", "source_line": 88, "expected": [ "a folder's `+` starts a conversation in that workspace and opens it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--folder-preview", "title": "Folder preview", "ids": [ "F04.folder-preview-more" ], "body": "With more than five conversations in `qa-f04-repo`, `control-openhands browser count '[data-testid^=\"thread-folder-ws-\"][data-testid$=\"qa-f04-repo\"] >> testid=conversation-card'` is `5` and `control-openhands browser text '[data-testid^=\"thread-folder-view-more-ws-\"][data-testid$=\"qa-f04-repo\"]'` is `More`. Click that button: every loaded row shows and the label is `Less`; click again: `5`.", "raw": "**Folder preview (`F04.folder-preview-more`).** With more than five conversations in `qa-f04-repo`, `control-openhands browser count '[data-testid^=\"thread-folder-ws-\"][data-testid$=\"qa-f04-repo\"] >> testid=conversation-card'` is `5` and `control-openhands browser text '[data-testid^=\"thread-folder-view-more-ws-\"][data-testid$=\"qa-f04-repo\"]'` is `More`. Click that button: every loaded row shows and the label is `Less`; click again: `5`.", "source_line": 89, "expected": [ "a folder shows five rows; More shows every loaded row, Less returns to five." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--reorder-folders", "title": "Reorder folders", "ids": [ "F04.folder-reorder" ], "body": "In By workspace mode read the order with `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=thread-folder-drag-]')].map(e=>e.textContent.trim())\"`, then run `control-openhands browser drag '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-folder\"]' '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-repo\"]' --position before`. The same eval now starts with `qa-f04-folder`, `qa-f04-repo`, also after `browser reload`, and `control-openhands browser eval \"JSON.parse(localStorage.getItem('conversation-panel-preferences')).state.groupFolderOrder\"` lists the `ws:` keys in that order.", "raw": "**Reorder folders (`F04.folder-reorder`).** In By workspace mode read the order with `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=thread-folder-drag-]')].map(e=>e.textContent.trim())\"`, then run `control-openhands browser drag '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-folder\"]' '[data-testid^=\"thread-folder-drag-ws-\"][data-testid$=\"qa-f04-repo\"]' --position before`. The same eval now starts with `qa-f04-folder`, `qa-f04-repo`, also after `browser reload`, and `control-openhands browser eval \"JSON.parse(localStorage.getItem('conversation-panel-preferences')).state.groupFolderOrder\"` lists the `ws:` keys in that order.", "source_line": 90, "expected": [ "dragging a folder heading onto another reorders the folders and the order persists." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--load-more", "title": "Load more", "ids": [ "F04.load-more" ], "body": "In By date mode (More options → `testid=organize-chronological`, or the Recent preset) with 22 or more conversations, `control-openhands browser count 'testid=conversation-panel-list-scroll >> testid=conversation-card'` is `20`. Run `control-openhands browser click 'testid=load-more-conversations'` and `control-openhands browser wait 'testid=load-more-conversations' --state detached --timeout 15000`; the count is the total (`23` with 23 conversations). In By workspace mode (`organize-grouped`, then `browser reload`) the same click adds the later rows to their own folders: with the oldest rows in No workspace and none of them loaded, `testid=thread-folder-__none_workspace` appears only after the click and `browser count 'testid=conversation-card'` grows by the number of unloaded No workspace rows (6 → 9 with three such rows, 5 → 7 with two). A folder that already shows five rows keeps the extra ones behind its More.", "raw": "**Load more (`F04.load-more`).** In By date mode (More options → `testid=organize-chronological`, or the Recent preset) with 22 or more conversations, `control-openhands browser count 'testid=conversation-panel-list-scroll >> testid=conversation-card'` is `20`. Run `control-openhands browser click 'testid=load-more-conversations'` and `control-openhands browser wait 'testid=load-more-conversations' --state detached --timeout 15000`; the count is the total (`23` with 23 conversations). In By workspace mode (`organize-grouped`, then `browser reload`) the same click adds the later rows to their own folders: with the oldest rows in No workspace and none of them loaded, `testid=thread-folder-__none_workspace` appears only after the click and `browser count 'testid=conversation-card'` grows by the number of unloaded No workspace rows (6 → 9 with three such rows, 5 → 7 with two). A folder that already shows five rows keeps the extra ones behind its More.", "source_line": 91, "expected": [ "20 conversations load first; Load more fetches the rest and disappears when nothing is left." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--organize", "title": "Organize", "ids": [ "F04.organize" ], "body": "Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=advanced-options-row'`, `control-openhands browser snapshot 'testid=advanced-conversation-options-modal'` (sections Organize, Sort by, Threads, Additional, Automations, Metadata), `control-openhands browser click 'testid=organize-chronological'` (`browser attr 'testid=organize-chronological' aria-checked` is `true`), `control-openhands browser click 'testid=advanced-options-close'`, `control-openhands browser reload`, then `control-openhands browser count '[data-testid^=thread-folder-ws-]'`. The count is `0` and the rows form one list; `organize-grouped` brings the folders back.", "raw": "**Organize (`F04.organize`).** Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=advanced-options-row'`, `control-openhands browser snapshot 'testid=advanced-conversation-options-modal'` (sections Organize, Sort by, Threads, Additional, Automations, Metadata), `control-openhands browser click 'testid=organize-chronological'` (`browser attr 'testid=organize-chronological' aria-checked` is `true`), `control-openhands browser click 'testid=advanced-options-close'`, `control-openhands browser reload`, then `control-openhands browser count '[data-testid^=thread-folder-ws-]'`. The count is `0` and the rows form one list; `organize-grouped` brings the folders back.", "source_line": 92, "expected": [ "More options → Organize switches between folders (By workspace) and a flat list (By date); the choice survives a reload." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--header-divider", "title": "Header divider", "ids": [ "F04.list-header" ], "body": "With more rows than fit (By date mode with 20 loaded rows, as Organize leaves it; By workspace mode previews too few rows to scroll), `control-openhands browser eval \"document.querySelector('[data-testid=older-conversations-summary]').parentElement.classList.contains('border-border')\"` is `false`; after `control-openhands browser scroll 'testid=conversation-panel-list-scroll' --by 300` it is `true`, and `false` again after `--by -300`.", "raw": "**Header divider (`F04.list-header`).** With more rows than fit (By date mode with 20 loaded rows, as Organize leaves it; By workspace mode previews too few rows to scroll), `control-openhands browser eval \"document.querySelector('[data-testid=older-conversations-summary]').parentElement.classList.contains('border-border')\"` is `false`; after `control-openhands browser scroll 'testid=conversation-panel-list-scroll' --by 300` it is `true`, and `false` again after `--by -300`.", "source_line": 93, "expected": [ "the header divider appears only while the list is scrolled." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--rename", "title": "Rename", "ids": [ "F04.card-rename" ], "body": "Pick the oldest untitled conversation (`b8d99...` from the picker step; the sort check below needs an old one). It is beyond the first 20 rows after a reload, so first run `control-openhands browser click 'testid=load-more-conversations'` and `control-openhands browser wait 'testid=load-more-conversations' --state detached --timeout 15000`. Then run `control-openhands browser scroll 'testid=conversation-card >> has-text=Conversation '` (the first five characters of its id), `control-openhands browser hover 'testid=conversation-card >> has-text=Conversation '`, `control-openhands browser click 'testid=conversation-card >> has-text=Conversation >> testid=ellipsis-button'`, `control-openhands browser click 'testid=context-menu >> testid=edit-button'`, `control-openhands browser fill 'input[data-testid=conversation-card-title]' 'QA_F04 renamed'`, `control-openhands browser press Enter`, then `control-openhands browser toasts` (`Conversation title updated successfully`), `control-openhands browser reload` and `control-openhands browser count 'testid=conversation-card-title >> has-text=QA_F04 renamed'` (`1`).", "raw": "**Rename (`F04.card-rename`).** Pick the oldest untitled conversation (`b8d99...` from the picker step; the sort check below needs an old one). It is beyond the first 20 rows after a reload, so first run `control-openhands browser click 'testid=load-more-conversations'` and `control-openhands browser wait 'testid=load-more-conversations' --state detached --timeout 15000`. Then run `control-openhands browser scroll 'testid=conversation-card >> has-text=Conversation '` (the first five characters of its id), `control-openhands browser hover 'testid=conversation-card >> has-text=Conversation '`, `control-openhands browser click 'testid=conversation-card >> has-text=Conversation >> testid=ellipsis-button'`, `control-openhands browser click 'testid=context-menu >> testid=edit-button'`, `control-openhands browser fill 'input[data-testid=conversation-card-title]' 'QA_F04 renamed'`, `control-openhands browser press Enter`, then `control-openhands browser toasts` (`Conversation title updated successfully`), `control-openhands browser reload` and `control-openhands browser count 'testid=conversation-card-title >> has-text=QA_F04 renamed'` (`1`).", "source_line": 94, "expected": [ "⋮ → Rename edits the title inline; Enter saves, a toast confirms, and the title persists." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--sort", "title": "Sort", "ids": [ "F04.sort" ], "body": "Under Sort by Updated (default) the just-renamed old conversation is first. Sorting applies to the loaded rows only: after a reload the first page is the 20 most recently updated. In More options run `control-openhands browser click 'testid=sort-created'` and `control-openhands browser click 'testid=advanced-options-close'`; `control-openhands browser eval \"[...document.querySelectorAll('[data-testid=conversation-panel-list-scroll] [data-testid=conversation-card-title]')].map(e=>e.textContent)\"` now ends with `QA_F04 renamed` (with every page loaded, the older `pong` row comes after it), and the View menu row reads `More options · Custom`. Restore with `testid=layout-preset-recent-activity`.", "raw": "**Sort (`F04.sort`).** Under Sort by Updated (default) the just-renamed old conversation is first. Sorting applies to the loaded rows only: after a reload the first page is the 20 most recently updated. In More options run `control-openhands browser click 'testid=sort-created'` and `control-openhands browser click 'testid=advanced-options-close'`; `control-openhands browser eval \"[...document.querySelectorAll('[data-testid=conversation-panel-list-scroll] [data-testid=conversation-card-title]')].map(e=>e.textContent)\"` now ends with `QA_F04 renamed` (with every page loaded, the older `pong` row comes after it), and the View menu row reads `More options · Custom`. Restore with `testid=layout-preset-recent-activity`.", "source_line": 95, "expected": [ "Sort by Updated or Created reorders the list." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--stop", "title": "Stop", "ids": [ "F04.stop" ], "body": "On an idle row open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=stop-button'`; `control-openhands browser snapshot 'role=dialog'` reads `Confirm Stop Conversation`, `Are you sure you want to stop this conversation?`, and `control-openhands browser testids 'role=dialog'` lists `cancel-button` (`Cancel`) and `confirm-button` (`Confirm Close`). `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps it idle (`control-openhands conversation status `). Repeat and click `'role=dialog >> testid=confirm-button'`; `control-openhands conversation wait --until paused --timeout 30` reaches `paused`, `control-openhands browser wait 'testid=conversation-card >> has-text=Conversation >> testid=conversation-status-paused'` succeeds, and the reopened ⋮ menu has no Stop item. `control-openhands browser press Escape` closes it (`browser count 'testid=context-menu'` is `0`) and puts focus back on the row's ⋮ (`control-openhands browser eval \"document.activeElement.dataset.testid\"` is `ellipsis-button`).", "raw": "**Stop (`F04.stop`).** On an idle row open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=stop-button'`; `control-openhands browser snapshot 'role=dialog'` reads `Confirm Stop Conversation`, `Are you sure you want to stop this conversation?`, and `control-openhands browser testids 'role=dialog'` lists `cancel-button` (`Cancel`) and `confirm-button` (`Confirm Close`). `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps it idle (`control-openhands conversation status `). Repeat and click `'role=dialog >> testid=confirm-button'`; `control-openhands conversation wait --until paused --timeout 30` reaches `paused`, `control-openhands browser wait 'testid=conversation-card >> has-text=Conversation >> testid=conversation-status-paused'` succeeds, and the reopened ⋮ menu has no Stop item. `control-openhands browser press Escape` closes it (`browser count 'testid=context-menu'` is `0`) and puts focus back on the row's ⋮ (`control-openhands browser eval \"document.activeElement.dataset.testid\"` is `ellipsis-button`).", "source_line": 96, "expected": [ "⋮ → Stop Conversation asks for confirmation and pauses the conversation; the item is gone once paused." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--presets", "title": "Presets", "ids": [ "F04.presets" ], "body": "Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=layout-preset-recent-activity'`, `control-openhands browser reload`, reopen the menu, then `control-openhands browser attr 'testid=layout-preset-recent-activity' aria-checked` (`true`) and `control-openhands browser text 'testid=advanced-options-row'` (`More options`). With the paused conversation from Stop (above), `testid=layout-preset-focused` (Active) and `testid=layout-preset-minimal` (Compact) make `control-openhands browser count 'testid=conversation-card >> has-text=Conversation '` `0`; Recent shows it again.", "raw": "**Presets (`F04.presets`).** Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=layout-preset-recent-activity'`, `control-openhands browser reload`, reopen the menu, then `control-openhands browser attr 'testid=layout-preset-recent-activity' aria-checked` (`true`) and `control-openhands browser text 'testid=advanced-options-row'` (`More options`). With the paused conversation from Stop (above), `testid=layout-preset-focused` (Active) and `testid=layout-preset-minimal` (Compact) make `control-openhands browser count 'testid=conversation-card >> has-text=Conversation '` `0`; Recent shows it again.", "source_line": 97, "expected": [ "a preset is checked and survives a reload; Active and Compact hide paused conversations; any manual change reads `More options · Custom`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--threads-scope", "title": "Threads scope", "ids": [ "F04.thread-scope" ], "body": "With the paused conversation from Stop and the Recent preset, open More options and run `control-openhands browser click 'testid=scope-relevant'` (`browser attr 'testid=scope-relevant' aria-checked` is `true`) and `control-openhands browser click 'testid=advanced-options-close'`: `control-openhands browser count 'testid=conversation-card >> has-text=Conversation '` is `0`, also after `browser reload`, and the View menu now checks no preset and reads `More options · Custom` (Recent shows old conversations; the Active preset also hides them). `testid=scope-all` brings the row back (`1`) and Recent is checked again.", "raw": "**Threads scope (`F04.thread-scope`).** With the paused conversation from Stop and the Recent preset, open More options and run `control-openhands browser click 'testid=scope-relevant'` (`browser attr 'testid=scope-relevant' aria-checked` is `true`) and `control-openhands browser click 'testid=advanced-options-close'`: `control-openhands browser count 'testid=conversation-card >> has-text=Conversation '` is `0`, also after `browser reload`, and the View menu now checks no preset and reads `More options · Custom` (Recent shows old conversations; the Active preset also hides them). `testid=scope-all` brings the row back (`1`) and Recent is checked again.", "source_line": 98, "expected": [ "More options → Threads → Active only hides paused conversations; All threads shows them again." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--hide-older", "title": "Hide older", "ids": [ "F04.hide-older" ], "body": "In More options, `control-openhands browser attr 'testid=toggle-older-conversations' aria-checked` is `false` (Recent shows conversations of every age) and `control-openhands browser text 'testid=older-conversation-cutoff'` is `Over 1 week`. Run `control-openhands browser click 'testid=older-conversation-cutoff'` (radios Over 1 hour, Over 1 day, Over 1 week, Over 30 days), `control-openhands browser click 'role=menuitemradio[name=\"Over 1 hour\"]'`, then `control-openhands browser click 'testid=toggle-older-conversations >> text=Hide old conversations'` (`aria-checked` turns `true`; a click on the row's center lands on the cutoff dropdown instead), `control-openhands browser click 'testid=advanced-options-close'`, `control-openhands browser reload`, then `control-openhands browser storage --values`: `conversation-panel-preferences` holds `\"showOlderConversations\":false` and `\"olderConversationCutoff\":\"1h\"`, and the View menu reads `More options · Custom`. Restore with `testid=layout-preset-recent-activity` (it turns hiding off but keeps the cutoff) and the cutoff `Over 1 week`. Hiding itself needs a conversation last updated before the cutoff (not reachable on a fresh run).", "raw": "**Hide older (`F04.hide-older`).** In More options, `control-openhands browser attr 'testid=toggle-older-conversations' aria-checked` is `false` (Recent shows conversations of every age) and `control-openhands browser text 'testid=older-conversation-cutoff'` is `Over 1 week`. Run `control-openhands browser click 'testid=older-conversation-cutoff'` (radios Over 1 hour, Over 1 day, Over 1 week, Over 30 days), `control-openhands browser click 'role=menuitemradio[name=\"Over 1 hour\"]'`, then `control-openhands browser click 'testid=toggle-older-conversations >> text=Hide old conversations'` (`aria-checked` turns `true`; a click on the row's center lands on the cutoff dropdown instead), `control-openhands browser click 'testid=advanced-options-close'`, `control-openhands browser reload`, then `control-openhands browser storage --values`: `conversation-panel-preferences` holds `\"showOlderConversations\":false` and `\"olderConversationCutoff\":\"1h\"`, and the View menu reads `More options · Custom`. Restore with `testid=layout-preset-recent-activity` (it turns hiding off but keeps the cutoff) and the cutoff `Over 1 week`. Hiding itself needs a conversation last updated before the cutoff (not reachable on a fresh run).", "source_line": 99, "expected": [ "Hide old conversations offers four cutoffs, persists the choice and hides conversations older than it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--automation-filter", "title": "Automation filter", "ids": [ "F04.automation-filter" ], "body": "In More options run `control-openhands browser click 'testid=automation-filter-only'` and `control-openhands browser click 'testid=advanced-options-close'`; with no automation runs `control-openhands browser text 'testid=conversation-panel-empty-state'` is `No conversations match the automation filter` (Load more stays). `testid=automation-filter-hide` keeps every manual conversation; `testid=automation-filter-all` restores the default. Unpin everything first: a pinned row suppresses the empty state.", "raw": "**Automation filter (`F04.automation-filter`).** In More options run `control-openhands browser click 'testid=automation-filter-only'` and `control-openhands browser click 'testid=advanced-options-close'`; with no automation runs `control-openhands browser text 'testid=conversation-panel-empty-state'` is `No conversations match the automation filter` (Load more stays). `testid=automation-filter-hide` keeps every manual conversation; `testid=automation-filter-all` restores the default. Unpin everything first: a pinned row suppresses the empty state.", "source_line": 100, "expected": [ "More options → Automations shows all, hides or keeps only automation runs, with its own empty message." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--row-metadata", "title": "Row metadata", "ids": [ "F04.metadata-toggles" ], "body": "In More options click `testid=toggle-repo-branch-metadata`, `testid=toggle-llm-profiles` and `testid=toggle-tags-metadata`, then `testid=advanced-options-close`. `control-openhands browser testids 'testid=conversation-card >> has-text=Conversation '` lists `conversation-card-workspace-folder` (`qa-f04-repo`) and `conversation-card-agent-chip` (`deepseek-flash`) and no `conversation-card-tag-chip`. Restore with `testid=layout-preset-recent-activity` (presets reset all metadata toggles).", "raw": "**Row metadata (`F04.metadata-toggles`).** In More options click `testid=toggle-repo-branch-metadata`, `testid=toggle-llm-profiles` and `testid=toggle-tags-metadata`, then `testid=advanced-options-close`. `control-openhands browser testids 'testid=conversation-card >> has-text=Conversation '` lists `conversation-card-workspace-folder` (`qa-f04-repo`) and `conversation-card-agent-chip` (`deepseek-flash`) and no `conversation-card-tag-chip`. Restore with `testid=layout-preset-recent-activity` (presets reset all metadata toggles).", "source_line": 101, "expected": [ "Repo and branch, Agent / model and Show tags add or remove row chips." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--hover-preview", "title": "Hover preview", "ids": [ "F04.hover-preview" ], "body": "Run `control-openhands browser mouse-click 900 900`, then `control-openhands browser tooltip 'testid=conversation-card >> has-text=Conversation '`. The text is the title, `Directory` with the workspace path, `Model` `deepseek/deepseek-flash` and `Created` with a date. After `testid=toggle-hover-metadata` is off (`aria-checked` `false`), `control-openhands browser tooltip ... --timeout 5000` finds no tooltip.", "raw": "**Hover preview (`F04.hover-preview`).** Run `control-openhands browser mouse-click 900 900`, then `control-openhands browser tooltip 'testid=conversation-card >> has-text=Conversation '`. The text is the title, `Directory` with the workspace path, `Model` `deepseek/deepseek-flash` and `Created` with a date. After `testid=toggle-hover-metadata` is off (`aria-checked` `false`), `control-openhands browser tooltip ... --timeout 5000` finds no tooltip.", "source_line": 102, "expected": [ "hovering a row for a second shows title, directory, model and creation time, unless Details on hover is off." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--pin", "title": "Pin", "ids": [ "F04.pin" ], "body": "Run `control-openhands browser hover 'testid=conversation-card >> has-text=QA_F04 renamed'`, `control-openhands browser click 'testid=conversation-card >> has-text=QA_F04 renamed >> [data-testid^=conversation-pin-toggle-]'`, then `control-openhands browser text 'testid=conversation-panel-pinned-section'` (`Pinned` and the row), `control-openhands browser count 'testid=conversation-card >> has-text=QA_F04 renamed'` (`1`: not duplicated) and, after `browser reload`, `control-openhands browser attr 'testid=conversation-panel-pinned-section >> [data-testid^=conversation-pin-toggle-]' aria-pressed` (`true`). Unpin with `browser hover` and `browser click` on `testid=conversation-panel-pinned-section >> testid=conversation-card` and its `[data-testid^=conversation-pin-toggle-]`; after a reload `browser count 'testid=conversation-panel-pinned-section'` is `0`.", "raw": "**Pin (`F04.pin`).** Run `control-openhands browser hover 'testid=conversation-card >> has-text=QA_F04 renamed'`, `control-openhands browser click 'testid=conversation-card >> has-text=QA_F04 renamed >> [data-testid^=conversation-pin-toggle-]'`, then `control-openhands browser text 'testid=conversation-panel-pinned-section'` (`Pinned` and the row), `control-openhands browser count 'testid=conversation-card >> has-text=QA_F04 renamed'` (`1`: not duplicated) and, after `browser reload`, `control-openhands browser attr 'testid=conversation-panel-pinned-section >> [data-testid^=conversation-pin-toggle-]' aria-pressed` (`true`). Unpin with `browser hover` and `browser click` on `testid=conversation-panel-pinned-section >> testid=conversation-card` and its `[data-testid^=conversation-pin-toggle-]`; after a reload `browser count 'testid=conversation-panel-pinned-section'` is `0`.", "source_line": 103, "expected": [ "the pin moves a row into a Pinned section above the list, survives a reload and unpins from there." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--pinned-preview", "title": "Pinned preview", "ids": [ "F04.pinned-preview-more" ], "body": "Pin six rows (`browser scroll`, `browser hover`, then click each row's `[data-testid^=conversation-pin-toggle-]`). `control-openhands browser count 'testid=conversation-panel-pinned-section >> testid=conversation-card'` is `5` and `control-openhands browser text 'testid=conversation-panel-pinned-view-more'` is `More`; `control-openhands browser click 'testid=conversation-panel-pinned-view-more'` makes the count `6` and the label `Less`; click again: `5`. Unpin all six with `browser hover` and a pin click on `'testid=conversation-panel-pinned-section >> testid=conversation-card >> nth=0'`, repeated.", "raw": "**Pinned preview (`F04.pinned-preview-more`).** Pin six rows (`browser scroll`, `browser hover`, then click each row's `[data-testid^=conversation-pin-toggle-]`). `control-openhands browser count 'testid=conversation-panel-pinned-section >> testid=conversation-card'` is `5` and `control-openhands browser text 'testid=conversation-panel-pinned-view-more'` is `More`; `control-openhands browser click 'testid=conversation-panel-pinned-view-more'` makes the count `6` and the label `Less`; click again: `5`. Unpin all six with `browser hover` and a pin click on `'testid=conversation-panel-pinned-section >> testid=conversation-card >> nth=0'`, repeated.", "source_line": 104, "expected": [ "the Pinned section previews five rows; its More shows every pinned row and Less returns to five." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--edit-tags", "title": "Edit tags", "ids": [ "F04.edit-tags" ], "body": "Open ⋮ on a row (hover, `testid=ellipsis-button`) and run `control-openhands browser click 'testid=context-menu >> testid=edit-tags-button'`. Then: `control-openhands browser fill 'testid=new-tag-key-input' 'Bad Key'` and `control-openhands browser press Enter --selector 'testid=new-tag-key-input'` → `control-openhands browser text 'testid=edit-tags-error'` is `Tag names may only contain lowercase letters and numbers` (it uses the `feedback-error` colour, #18330: `control-openhands browser eval \"getComputedStyle(document.querySelector('[data-testid=edit-tags-error]')).color\"` is `rgb(231, 106, 94)` on OpenHands-Neutral and `rgb(192, 40, 45)` on Light+, readable on the light dialog; switch themes as in `F16.color-theme` and restore OpenHands-Neutral); key `title` with `testid=add-tag-button` → `This tag name is reserved`; key `qaenv`, `browser fill 'testid=new-tag-value-input' one`, Enter → `browser count 'testid=edit-tag-row-qaenv'` is `1`; key `qaenv` again → `This tag name is already in use`; key `qalong` with a 257-character value → `Tag values must be 256 characters or fewer`; key `qabare` with an empty value left in the inputs, then `control-openhands browser click 'testid=edit-conversation-tags-modal >> testid=confirm-button'`. `control-openhands browser toasts` shows `Tags updated`; after `browser reload` the row shows chips `Qabare` and `Qaenv: one` (`conversation-card-tag-chip`), and `control-openhands api GET '/api/conversations/search?limit=3'` has tags `qaenv: one`, `qabare: \"\"`.", "raw": "**Edit tags (`F04.edit-tags`).** Open ⋮ on a row (hover, `testid=ellipsis-button`) and run `control-openhands browser click 'testid=context-menu >> testid=edit-tags-button'`. Then: `control-openhands browser fill 'testid=new-tag-key-input' 'Bad Key'` and `control-openhands browser press Enter --selector 'testid=new-tag-key-input'` → `control-openhands browser text 'testid=edit-tags-error'` is `Tag names may only contain lowercase letters and numbers` (it uses the `feedback-error` colour, #18330: `control-openhands browser eval \"getComputedStyle(document.querySelector('[data-testid=edit-tags-error]')).color\"` is `rgb(231, 106, 94)` on OpenHands-Neutral and `rgb(192, 40, 45)` on Light+, readable on the light dialog; switch themes as in `F16.color-theme` and restore OpenHands-Neutral); key `title` with `testid=add-tag-button` → `This tag name is reserved`; key `qaenv`, `browser fill 'testid=new-tag-value-input' one`, Enter → `browser count 'testid=edit-tag-row-qaenv'` is `1`; key `qaenv` again → `This tag name is already in use`; key `qalong` with a 257-character value → `Tag values must be 256 characters or fewer`; key `qabare` with an empty value left in the inputs, then `control-openhands browser click 'testid=edit-conversation-tags-modal >> testid=confirm-button'`. `control-openhands browser toasts` shows `Tags updated`; after `browser reload` the row shows chips `Qabare` and `Qaenv: one` (`conversation-card-tag-chip`), and `control-openhands api GET '/api/conversations/search?limit=3'` has tags `qaenv: one`, `qabare: \"\"`.", "source_line": 105, "expected": [ "⋮ → Edit tags adds and removes `key: value` tags with validation; Save shows `Tags updated` and the chips persist." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--tag-overflow", "title": "Tag overflow", "ids": [ "F04.tag-overflow" ], "body": "On the tagged row add three more tags in Edit tags (keys `qaregion`, `qastage`, `qaowner`, values `value-for-`, Enter after each) and save. After `browser reload` the row shows `Qabare` and `Qaenv: one` plus `control-openhands browser text 'testid=conversation-card >> has-text=Conversation >> testid=conversation-card-tag-overflow'` `+3` (aria-label `Show 3 more tags`). Clicking it keeps the URL, sets `aria-expanded` `true`, and `control-openhands browser text 'testid=conversation-card-tag-overflow-popover'` lists `Qaowner value-for-qaowner`, `Qaregion …`, `Qastage …` (3 `conversation-card-tag-overflow-row`); `control-openhands browser press Escape` closes it (count `0`).", "raw": "**Tag overflow (`F04.tag-overflow`).** On the tagged row add three more tags in Edit tags (keys `qaregion`, `qastage`, `qaowner`, values `value-for-`, Enter after each) and save. After `browser reload` the row shows `Qabare` and `Qaenv: one` plus `control-openhands browser text 'testid=conversation-card >> has-text=Conversation >> testid=conversation-card-tag-overflow'` `+3` (aria-label `Show 3 more tags`). Clicking it keeps the URL, sets `aria-expanded` `true`, and `control-openhands browser text 'testid=conversation-card-tag-overflow-popover'` lists `Qaowner value-for-qaowner`, `Qaregion …`, `Qastage …` (3 `conversation-card-tag-overflow-row`); `control-openhands browser press Escape` closes it (count `0`).", "source_line": 106, "expected": [ "tags that do not fit on a row collapse into a `+N` chip (aria `Show N more tags`) whose popover lists the hidden tags without opening the conversation; Escape closes it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--remove-tags", "title": "Remove tags", "ids": [ "F04.edit-tags-remove" ], "body": "Reopen Edit tags on that row and run `control-openhands browser click 'testid=remove-tag-qaregion'` (likewise `remove-tag-qastage`, `remove-tag-qaowner`); `control-openhands browser count 'testid=edit-tags-rows >> [data-testid^=edit-tag-row-]'` is `2`. Save with `testid=edit-conversation-tags-modal >> testid=confirm-button` (`Tags updated`); after `browser reload` the `+3` chip is gone and `control-openhands api GET '/api/conversations/search?limit=1' --pick items.0.tags` is `clientsource`, `qabare`, `qaenv` only.", "raw": "**Remove tags (`F04.edit-tags-remove`).** Reopen Edit tags on that row and run `control-openhands browser click 'testid=remove-tag-qaregion'` (likewise `remove-tag-qastage`, `remove-tag-qaowner`); `control-openhands browser count 'testid=edit-tags-rows >> [data-testid^=edit-tag-row-]'` is `2`. Save with `testid=edit-conversation-tags-modal >> testid=confirm-button` (`Tags updated`); after `browser reload` the `+3` chip is gone and `control-openhands api GET '/api/conversations/search?limit=1' --pick items.0.tags` is `clientsource`, `qabare`, `qaenv` only.", "source_line": 107, "expected": [ "in Edit tags each row's remove button drops that tag; Save persists the removal on the server." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--tag-filter", "title": "Tag filter", "ids": [ "F04.tag-filter" ], "body": "After Edit tags (above) gave one row `qaenv=one`, run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=tag-filters-section'`, `control-openhands browser click 'testid=tag-facet-row-qaenv=one'` (`aria-checked` `true`; the menu stays open), `control-openhands browser press Escape`, then `control-openhands browser count 'testid=conversation-panel-list-scroll >> testid=conversation-card'`: only the tagged row plus pinned rows. The count is the same after `browser reload`.", "raw": "**Tag filter (`F04.tag-filter`).** After Edit tags (above) gave one row `qaenv=one`, run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=tag-filters-section'`, `control-openhands browser click 'testid=tag-facet-row-qaenv=one'` (`aria-checked` `true`; the menu stays open), `control-openhands browser press Escape`, then `control-openhands browser count 'testid=conversation-panel-list-scroll >> testid=conversation-card'`: only the tagged row plus pinned rows. The count is the same after `browser reload`.", "source_line": 108, "expected": [ "choosing a tag facet narrows the list to rows with that tag (pins exempt) and survives a reload." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--filter-chips", "title": "Filter chips", "ids": [ "F04.filter-chips" ], "body": "With the facet active, `control-openhands browser text 'testid=conversation-active-tag-filters'` is `qaenv=one` and `Clear all`. `control-openhands browser click 'testid=active-tag-filter-qaenv=one'` removes it (`browser count 'testid=conversation-active-tag-filters'` is `0`). Select `tag-facet-row-qaenv=one` and `tag-facet-row-qabare=` again, then `control-openhands browser click 'testid=clear-tag-filters'`: the strip disappears and the full list returns.", "raw": "**Filter chips (`F04.filter-chips`).** With the facet active, `control-openhands browser text 'testid=conversation-active-tag-filters'` is `qaenv=one` and `Clear all`. `control-openhands browser click 'testid=active-tag-filter-qaenv=one'` removes it (`browser count 'testid=conversation-active-tag-filters'` is `0`). Select `tag-facet-row-qaenv=one` and `tag-facet-row-qabare=` again, then `control-openhands browser click 'testid=clear-tag-filters'`: the strip disappears and the full list returns.", "source_line": 109, "expected": [ "active filters show as chips above the list; a chip click removes one, Clear all removes every chip." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--automation-setup-tag", "title": "Automation setup tag", "ids": [ "F04.automation-setup-tag" ], "body": "Arrange one setup conversation through the setup page (F22): `control-openhands browser goto /automations`, `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup'`, then `control-openhands conversation start --stay --prompt 'Reply with only the word ok. Do not run any tools.' --wait --timeout 180` (one tiny model call; note ``). `control-openhands api GET /api/conversations/ --pick tags` has `automationsetup: draft`. In the sidebar, `control-openhands browser text 'testid=conversation-card >> has-text=Reply >> testid=conversation-card-tag-chip'` is `Automation setup: Draft` (the first chip; the internal draft-id keys stay hidden). Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=tag-filters-section'`: the facet `testid=tag-facet-row-automationsetup=draft` reads `Automation setup: Draft`; click it, `control-openhands browser press Escape`, and `control-openhands browser text 'testid=conversation-active-tag-filters'` is `Automation setup: Draft` and `Clear all`, with only setup conversations (and pinned rows) left; `control-openhands browser click 'testid=active-tag-filter-automationsetup=draft'` removes it. After `control-openhands browser mouse-click 900 900`, `control-openhands browser tooltip 'testid=conversation-card >> has-text=Reply'` lists `Automation setup` and the raw value `draft` (`browser screenshot --feature F04.automation-setup-tag --name facet`). End with `control-openhands browser goto /`; the setup send leaves no Home draft, and opening another conversation from the sidebar leaves that conversation's tags alone (see Gotchas).", "raw": "**Automation setup tag (`F04.automation-setup-tag`).** Arrange one setup conversation through the setup page (F22): `control-openhands browser goto /automations`, `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup'`, then `control-openhands conversation start --stay --prompt 'Reply with only the word ok. Do not run any tools.' --wait --timeout 180` (one tiny model call; note ``). `control-openhands api GET /api/conversations/ --pick tags` has `automationsetup: draft`. In the sidebar, `control-openhands browser text 'testid=conversation-card >> has-text=Reply >> testid=conversation-card-tag-chip'` is `Automation setup: Draft` (the first chip; the internal draft-id keys stay hidden). Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=tag-filters-section'`: the facet `testid=tag-facet-row-automationsetup=draft` reads `Automation setup: Draft`; click it, `control-openhands browser press Escape`, and `control-openhands browser text 'testid=conversation-active-tag-filters'` is `Automation setup: Draft` and `Clear all`, with only setup conversations (and pinned rows) left; `control-openhands browser click 'testid=active-tag-filter-automationsetup=draft'` removes it. After `control-openhands browser mouse-click 900 900`, `control-openhands browser tooltip 'testid=conversation-card >> has-text=Reply'` lists `Automation setup` and the raw value `draft` (`browser screenshot --feature F04.automation-setup-tag --name facet`). End with `control-openhands browser goto /`; the setup send leaves no Home draft, and opening another conversation from the sidebar leaves that conversation's tags alone (see Gotchas).", "source_line": 110, "expected": [ "a conversation started from the automation setup page shows an `Automation setup: Draft` chip (lightning icon) first in its tag row; Filter by tags offers a facet and filter chip of the same name, and the hover preview lists `Automation setup` `draft`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--download", "title": "Download", "ids": [ "F04.download" ], "body": "Open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=download-trajectory-button'`, then `control-openhands browser downloads`. A file `…-conversation_.zip` lands in `/private/downloads`; `control-openhands browser downloads --last 1 --inspect` lists `/base_state.json` and `meta.json` for any row, plus event folders such as `bash_events/` for a conversation that ran (the `pong` row; it is the oldest, so Load more first).", "raw": "**Download (`F04.download`).** Open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=download-trajectory-button'`, then `control-openhands browser downloads`. A file `…-conversation_.zip` lands in `/private/downloads`; `control-openhands browser downloads --last 1 --inspect` lists `/base_state.json` and `meta.json` for any row, plus event folders such as `bash_events/` for a conversation that ran (the `pong` row; it is the oldest, so Load more first).", "source_line": 111, "expected": [ "⋮ → Download conversation data saves `conversation_.zip`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--download-failure", "title": "Download failure", "ids": [ "F04.download-error" ], "body": "With ⋮ already open on a row, run `control-openhands browser errors --clear` and `control-openhands service stop agent-server`, click `testid=context-menu >> testid=download-trajectory-button`, then `control-openhands browser toasts --history` and `control-openhands browser count 'testid=context-menu'`. There is exactly one toast, `Couldn't download the conversation data: An error occurred`, and the menu is closed (`0`). The download request returned a plain-text ingress 502; `getApiErrorMessage` intentionally omits raw transport text and falls back to this generic message when the response body has no usable error field. `control-openhands browser errors --app-only` has `pageErrors` `0`; its 502 request and console error are the outage itself. Run `control-openhands restart` and `control-openhands doctor` afterwards.", "raw": "**Download failure (`F04.download-error`).** With ⋮ already open on a row, run `control-openhands browser errors --clear` and `control-openhands service stop agent-server`, click `testid=context-menu >> testid=download-trajectory-button`, then `control-openhands browser toasts --history` and `control-openhands browser count 'testid=context-menu'`. There is exactly one toast, `Couldn't download the conversation data: An error occurred`, and the menu is closed (`0`). The download request returned a plain-text ingress 502; `getApiErrorMessage` intentionally omits raw transport text and falls back to this generic message when the response body has no usable error field. `control-openhands browser errors --app-only` has `pageErrors` `0`; its 502 request and console error are the outage itself. Run `control-openhands restart` and `control-openhands doctor` afterwards.", "source_line": 112, "expected": [ "a failed download shows one toast and closes the ⋮ menu. When the Agent Server is down behind a plain-text 502 ingress response, the toast uses the generic error fallback rather than raw transport text." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--archive", "title": "Archive", "ids": [ "F04.archive" ], "body": "Open a conversation from its row, open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=archive-button'`; `control-openhands browser snapshot 'role=dialog'` reads `Archive conversation` and `Archive \"\"? The conversation stays available with its full history and can be restored from Show archived.` `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps the row. Repeat and `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations(\\?|$)'`; the row count is `0`, also after `browser reload`. Archiving a pinned row removes the Pinned section.", "raw": "**Archive (`F04.archive`).** Open a conversation from its row, open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=archive-button'`; `control-openhands browser snapshot 'role=dialog'` reads `Archive conversation` and `Archive \"<title>\"? The conversation stays available with its full history and can be restored from Show archived.` `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps the row. Repeat and `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations(\\?|$)'`; the row count is `0`, also after `browser reload`. Archiving a pinned row removes the Pinned section.", "source_line": 113, "expected": [ "⋮ → Archive Conversation asks for confirmation, hides and unpins the row, and leaves the conversation if it was open." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--show-archived-and-unarchive", "title": "Show archived and Unarchive", "ids": [ "F04.show-archived-unarchive" ], "body": "In More options click `testid=toggle-show-archived` (`aria-checked` `true`) and `testid=advanced-options-close`; `control-openhands browser count 'testid=conversation-card >> has-text=Conversation <id5> >> testid=conversation-card-archived-chip'` is `1` and the ⋮ menu offers `Unarchive Conversation`. `control-openhands browser click 'testid=context-menu >> testid=unarchive-button'` opens no dialog (`browser count 'role=dialog'` `0`) and the chip count drops to `0`. Turn Show archived off again.", "raw": "**Show archived and Unarchive (`F04.show-archived-unarchive`).** In More options click `testid=toggle-show-archived` (`aria-checked` `true`) and `testid=advanced-options-close`; `control-openhands browser count 'testid=conversation-card >> has-text=Conversation <id5> >> testid=conversation-card-archived-chip'` is `1` and the ⋮ menu offers `Unarchive Conversation`. `control-openhands browser click 'testid=context-menu >> testid=unarchive-button'` opens no dialog (`browser count 'role=dialog'` `0`) and the chip count drops to `0`. Turn Show archived off again.", "source_line": 114, "expected": [ "Show archived brings archived rows back with an `Archived` chip; Unarchive restores a row without confirmation." ], "notes": [ "Known issue [#18155](https://github.com/OpenHands/OpenHands/issues/18155) (`F04.show-archived-unarchive`): Missing-runtime conversations shown as Archived but never hidden, chipped, or unarchivable. Live prerequisite: A Cloud sandbox or local runtime that becomes MISSING and non-resumable; ordinary local pause remains resumable." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F04--delete", "title": "Delete", "ids": [ "F04.delete" ], "body": "Open a conversation from its row, open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=delete-button'`; the dialog reads `Confirm Delete` and `Are you sure you want to delete the \"<title>\" conversation? This action cannot be undone.` with `cancel-button` and `confirm-button`. `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps the row. Repeat and `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations(\\?|$)'`; after `browser reload` the row count is `0` and `control-openhands api GET /api/conversations/<id>` is `404`.", "raw": "**Delete (`F04.delete`).** Open a conversation from its row, open ⋮ and run `control-openhands browser click 'testid=context-menu >> testid=delete-button'`; the dialog reads `Confirm Delete` and `Are you sure you want to delete the \"<title>\" conversation? This action cannot be undone.` with `cancel-button` and `confirm-button`. `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps the row. Repeat and `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations(\\?|$)'`; after `browser reload` the row count is `0` and `control-openhands api GET /api/conversations/<id>` is `404`.", "source_line": 115, "expected": [ "⋮ → Delete Conversation asks for confirmation, deletes it on the server and leaves it if it was open." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--phone", "title": "Phone", "ids": [ "F04.phone" ], "body": "Run `control-openhands browser viewport phone`, `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-mobile-menu-toggle'`, then `control-openhands browser bbox 'testid=sidebar-mobile-drawer >> testid=conversation-panel'` (`insideViewport` `true`, `pageHorizontalOverflow` `false`) and `control-openhands browser screenshot --feature F04.phone --name drawer`. Hover a row in the drawer and click its `testid=ellipsis-button`; `control-openhands browser bbox 'testid=context-menu'` is inside the viewport. `control-openhands browser mouse-click 380 20` closes the menu (`browser count 'testid=context-menu'` `0`) and the drawer (`control-openhands browser wait 'testid=sidebar-mobile-drawer' --state hidden --timeout 5000`; an immediate `visible` still reads `true` during the slide-out). Reopen the drawer and `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=conversation-card >> has-text=QA_F04 renamed' --expect-url '/conversations/'`; `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state hidden --timeout 5000` succeeds. Return with `control-openhands browser viewport desktop`.", "raw": "**Phone (`F04.phone`).** Run `control-openhands browser viewport phone`, `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-mobile-menu-toggle'`, then `control-openhands browser bbox 'testid=sidebar-mobile-drawer >> testid=conversation-panel'` (`insideViewport` `true`, `pageHorizontalOverflow` `false`) and `control-openhands browser screenshot --feature F04.phone --name drawer`. Hover a row in the drawer and click its `testid=ellipsis-button`; `control-openhands browser bbox 'testid=context-menu'` is inside the viewport. `control-openhands browser mouse-click 380 20` closes the menu (`browser count 'testid=context-menu'` `0`) and the drawer (`control-openhands browser wait 'testid=sidebar-mobile-drawer' --state hidden --timeout 5000`; an immediate `visible` still reads `true` during the slide-out). Reopen the drawer and `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=conversation-card >> has-text=QA_F04 renamed' --expect-url '/conversations/'`; `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state hidden --timeout 5000` succeeds. Return with `control-openhands browser viewport desktop`.", "source_line": 116, "expected": [ "at 390 px the list lives in the sidebar drawer, fits the viewport, and picking a row closes the drawer." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--a-later-page-pin", "title": "A later-page pin", "ids": [ "F04.pin" ], "body": "With more than 20 conversations as the pagination precondition already arranges, choose Recent, reload, click `role=button[name=\"Load more\"]`, and pin a conversation from the newly loaded rows (`browser hover` its `conversation-card`, then click its `[data-testid^=conversation-pin-toggle-]`). Read the pinned section, then `control-openhands browser reload` and `control-openhands browser count 'testid=conversation-panel-pinned-section'`. Known failure (reproduced 2026-10-12, [#16399](https://github.com/OpenHands/OpenHands/issues/16399)): the later-page pin is pruned on first-page reload even though the conversation still exists. Tested with 23 empty conversations and one older pin.", "raw": "**A later-page pin (`F04.pin`).** With more than 20 conversations as the pagination precondition already arranges, choose Recent, reload, click `role=button[name=\"Load more\"]`, and pin a conversation from the newly loaded rows (`browser hover` its `conversation-card`, then click its `[data-testid^=conversation-pin-toggle-]`). Read the pinned section, then `control-openhands browser reload` and `control-openhands browser count 'testid=conversation-panel-pinned-section'`. Known failure (reproduced 2026-10-12, [#16399](https://github.com/OpenHands/OpenHands/issues/16399)): the later-page pin is pruned on first-page reload even though the conversation still exists. Tested with 23 empty conversations and one older pin.", "source_line": 117, "expected": [ "the pin moves a row into a Pinned section above the list, survives a reload and unpins from there." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [#16399](https://github.com/OpenHands/OpenHands/issues/16399)): the later-page pin is pruned on first-page reload even though the conversation still exists. Tested with 23 empty conversations and one older pin." ], "states": [ "failure" ] }, { "name": "F04--fresh-browser-grouping", "title": "Fresh-browser grouping", "ids": [ "F04.grouped-folders" ], "body": "Run this after the other list checks: select By workspace, record its groups, `control-openhands browser reset`, `control-openhands onboard --skip`, then `control-openhands browser click 'testid=conversation-layouts-toggle'` and `control-openhands browser click 'testid=layout-preset-by-workspace'`. Compare `browser snapshot 'testid=conversation-panel'` with `conversation status <id>`. Known failure (reproduced 2026-10-12, [#17386](https://github.com/OpenHands/OpenHands/issues/17386)): existing conversations move to No workspace although their server working directory still matches the registered workspace. The reset clears browser metadata; do not use it before checks that depend on the original browser's grouping.", "raw": "**Fresh-browser grouping (`F04.grouped-folders`).** Run this after the other list checks: select By workspace, record its groups, `control-openhands browser reset`, `control-openhands onboard --skip`, then `control-openhands browser click 'testid=conversation-layouts-toggle'` and `control-openhands browser click 'testid=layout-preset-by-workspace'`. Compare `browser snapshot 'testid=conversation-panel'` with `conversation status <id>`. Known failure (reproduced 2026-10-12, [#17386](https://github.com/OpenHands/OpenHands/issues/17386)): existing conversations move to No workspace although their server working directory still matches the registered workspace. The reset clears browser metadata; do not use it before checks that depend on the original browser's grouping.", "source_line": 118, "expected": [ "By workspace shows one folder per known workspace (empty ones included) plus a No workspace folder while any loaded row has no workspace, in any browser." ], "notes": [], "annotations": [ "Known failure (reproduced 2026-10-12, [#17386](https://github.com/OpenHands/OpenHands/issues/17386)): existing conversations move to No workspace although their server working directory still matches the registered workspace. The reset clears browser metadata; do not use it before checks that depend on the original browser's grouping." ], "states": [ "failure" ] }, { "name": "F04--delete-all", "title": "Delete all", "ids": [ "F04.delete-all" ], "body": "Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=delete-all-conversations'` and `control-openhands browser snapshot 'role=dialog'`: `Delete all conversations`, `Are you sure you want to delete N conversations?`. Cancel (`role=dialog >> testid=cancel-button`) keeps every row. Confirm with `control-openhands browser click 'role=dialog >> testid=confirm-button'` and `control-openhands browser wait 'role=dialog' --state detached --timeout 30000`; `control-openhands api GET '/api/conversations/search?limit=100'` has no items, Delete all is disabled again, and in By date mode the empty state returns. Expected for every conversation; today N and the deletion cover only the loaded pages (22 conversations, 20 loaded: 2 remain).", "raw": "**Delete all (`F04.delete-all`).** Run `control-openhands browser click 'testid=conversation-layouts-toggle'`, `control-openhands browser click 'testid=delete-all-conversations'` and `control-openhands browser snapshot 'role=dialog'`: `Delete all conversations`, `Are you sure you want to delete N conversations?`. Cancel (`role=dialog >> testid=cancel-button`) keeps every row. Confirm with `control-openhands browser click 'role=dialog >> testid=confirm-button'` and `control-openhands browser wait 'role=dialog' --state detached --timeout 30000`; `control-openhands api GET '/api/conversations/search?limit=100'` has no items, Delete all is disabled again, and in By date mode the empty state returns. Expected for every conversation; today N and the deletion cover only the loaded pages (22 conversations, 20 loaded: 2 remain).", "source_line": 119, "expected": [ "View → Delete all asks for confirmation with a count and deletes every conversation." ], "notes": [ "Known issue #17919: Delete all deletes only the loaded pages, and its dialog states that smaller count (`F04.delete-all`)." ], "annotations": [], "states": [ "issue" ] }, { "name": "F04--collapsed-rail", "title": "Collapsed rail", "ids": [ "F04.collapsed-rail" ], "body": "Run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, then `control-openhands browser count 'testid=conversation-panel'` (`0`). Expand with `control-openhands browser click 'aside[data-collapsed]'` (see F02); the count is `1`.", "raw": "**Collapsed rail (`F04.collapsed-rail`).** Run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, then `control-openhands browser count 'testid=conversation-panel'` (`0`). Expand with `control-openhands browser click 'aside[data-collapsed]'` (see F02); the count is `1`.", "source_line": 120, "expected": [ "the collapsed sidebar rail shows no conversation list; expanding brings it back." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F04--cloud-picker-and-start-tasks", "title": "Cloud picker and start tasks", "ids": [ "F04.cloud-picker", "F04.start-task-cards" ], "body": "Blocked without an OpenHands Cloud backend and git provider. On Cloud, the same `conversation-panel-new-thread-picker` opens `cloud-provider-tabs`, `cloud-repo-search-input` (`Search repositories`) and `launch-repository` rows; start tasks render `testid=start-task-card` linking to `/conversations/task-<id>`.", "raw": "**Cloud picker and start tasks (`F04.cloud-picker`, `F04.start-task-cards`).** Blocked without an OpenHands Cloud backend and git provider. On Cloud, the same `conversation-panel-new-thread-picker` opens `cloud-provider-tabs`, `cloud-repo-search-input` (`Search repositories`) and `launch-repository` rows; start tasks render `testid=start-task-card` linking to `/conversations/task-<id>`.", "source_line": 121, "expected": [ "on a Cloud backend the folder-plus picker shows provider tabs and a repository search; a picked repository starts a conversation.", "on a Cloud backend in-progress start tasks show as cards linking to `/conversations/task-<id>`." ], "notes": [ "Known issue [#17257](https://github.com/OpenHands/OpenHands/issues/17257) (`F04.start-task-cards`): Cloud provisioning has no in-progress start-task cards because the list hook always returns an empty array. Live prerequisite: An authenticated Cloud backend producing an in-progress sandbox/repository start task." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F04--cleanup", "title": "Cleanup", "ids": [], "body": "View → Delete all twice (the second pass removes the rows the first left behind), until `api GET '/api/conversations/search?limit=100'` is empty; re-apply the Recent preset. Fixture folders and the workspace registry vanish with the run.", "raw": "**Cleanup.** View → Delete all twice (the second pass removes the rows the first left behind), until `api GET '/api/conversations/search?limit=100'` is empty; re-apply the Recent preset. Fixture folders and the workspace registry vanish with the run.", "source_line": 123, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] } ] }, { "id": "F05", "title": "Composer, slash commands and plan mode", "file": "F05-composer.md", "sha256": "7180bf9a1dc1967d236f3354ffa88fa3cc91a469e6b8a9d5d858529a55741730", "overview": "The composer is the input box at the bottom of the home page and of every\nconversation. A user types a message (Enter sends, Shift+Enter adds a line),\nattaches files and images, picks the LLM profile for the running conversation,\nwatches and compacts the context window, stops and resumes the agent, and types\n`/` for built-in commands (`/btw`, `/model`, `/goal`, `/plan`, `/code`) and\nskills. `/plan` hands the next message to a planning agent whose PLAN.md card\ncan be viewed and built; Ctrl/Cmd+Enter builds it.", "entry": "- Home page composer (`/`) and the composer of any conversation (`/conversations/<id>`, reached from the sidebar conversation list or by URL).\n- `+` (More actions, `chat-plus-button`) inside the composer opens the tools menu; its last item **Add Files and Images** opens the native file picker (`upload-image-input`).\n- Model pill (`chat-input-llm-profile`) next to `+`; at 320 px it moves into the vertical-dots **More input actions** button. On the home page it names the default profile (the one Settings → LLM marks as default); in a conversation it names the profile that conversation runs on.\n- Right side of the composer: status text with Stop/Play, context ring (`context-window-meter`), mic (`chat-dictation-button`), Send (`submit-button`).\n- Typing `/` at the start of a word opens the slash menu (`slash-command-menu`); `/plan` and `/code` are the only mode switches on the local backend (the Code/Plan pill is Cloud only).\n- Keyboard: Enter sends, Shift+Enter newline, Ctrl+Enter / Cmd+Enter builds a plan; Escape closes the slash menu.", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done) and `control-openhands llm preset deepseek` (profiles `deepseek-flash` active and `deepseek-pro`). Every recipe except `F05.agent-profile-switch`, the home draft, `F05.image-only-send` and `F05.profile-identity` needs a model. Without a DeepSeek key, `QA_DUMMY_KEY=qa control-openhands llm set --profile qa-zeta --model openai/gpt-4o --api-key-env QA_DUMMY_KEY --no-validate` stands in for the active profile: conversations can still be created (the agent errors at once), which is all those two bullets need, and every bullet that reads a reply is `blocked`.\n- `F05.profile-identity` needs two profiles on one model: its bullet creates `qa-alpha` and `qa-zeta` with `llm set --no-validate` right before it and deletes them afterwards.\n- A conversation to type into: `control-openhands conversation start --prompt \"Reply with only: ok\" --wait --timeout 240` prints `id` (`<id>` below) and leaves the browser on `/conversations/<id>`.\n- Attachment fixtures: `control-openhands fixture image --name qa-image` and `control-openhands fixture file --name qa-note.txt --content \"QA attachment 7731\"` (both land in `$OH_VERIFY_RUN/evidence/_fixtures/`).\n- Desktop viewport unless a bullet says otherwise. Microphone dictation is blocked in the sandbox browser (no microphone); `+` > Git tools needs a conversation on a git workspace (not driven here).", "gotchas": "- Always export your own `OH_VERIFY_RUN`. `launch --new` repoints the shared `current` symlink, and another agent relying on `current` drove and then stopped this family's stack mid-run. If the URL jumps to a page you never opened or `fetch failed` appears, run `control-openhands status` before blaming the product.\n- `conversation wait <id>` right after sending returns at once with the previous `finished` status. Use `conversation wait <id> --fresh`; `--until running` then a terminal wait only works for turns longer than the poll interval (a `Reply with only` turn on deepseek-flash can finish first, and `--until running` then times out).\n- The slash menu stays open while you type a command; press Escape before Enter or Enter selects the highlighted option instead of sending.\n- Conversation drafts are saved 500 ms after the last keystroke and flushed on navigation and unmount, so text typed just before leaving comes back.\n- Escape closes the `+` tools menu (`browser count 'testid=tools-context-menu'` is `0`) and returns focus to `chat-plus-button`, whether focus was on the trigger or on a menu item. A menu left open covers attachments in screenshots. Escape also closes the narrow-width \"More input actions\" overflow menu (`browser count 'testid=chat-input-overflow-menu'` is `0`, the trigger's `aria-expanded` is `false`) and returns focus to the trigger, whether focus was on the trigger, on `overflow-model-button` or on a profile row of the open Model submenu (#18174).\n- After a client-side switch between conversations (a sidebar row, or a user-message branch in `F06.branch-from-here`) Send keeps the previous conversation's state (Known failure, [#18202](https://github.com/OpenHands/OpenHands/issues/18202); [draft fix PR #18203](https://github.com/OpenHands/OpenHands/pull/18203)). Switching from an empty composer to one with a saved draft restores the text next to a disabled Send; this is a race (Send came up enabled in 1 of 7 tries), so one enabled reading does not show the bug is gone. Switching from a composer with text to an empty one leaves Send enabled, and clicking it does nothing. Typing or `browser reload` fixes it, and Enter in the field sends the restored text anyway. Read `browser enabled 'testid=submit-button'` after a reload, or after typing.\n- Expected: a prompt that a suggestion chip (`testid=chat-suggestions`) or a `+` > **Macros** item writes into the composer enables Send at once. Known failure (reproduced 2026-10-09, [#18202](https://github.com/OpenHands/OpenHands/issues/18202)): on a new No workspace conversation, `browser click 'testid=chat-suggestions >> text=Fix README'` (or `browser click 'testid=chat-plus-button'`, `browser hover 'testid=macros-button'`, `browser click 'testid=fix-readme-button'`) fills the field, but `browser enabled 'testid=submit-button'` is `false` (5 of 5 tries at desktop; a race at phone) until a character is typed. Both paths call `setMessageToSend`, which writes the text without an input event.\n- Uploaded chips and thumbnails have no test ids. Their remove buttons are named `Remove <file name>` (file chip) and `Remove image` (thumbnail), and Send, Stop and Play read `Send message`, `Stop` and `Resume the agent task` (#18033); select by those names, or by text/alt.\n- `browser upload` feeds the hidden input directly. To drive the real path, `+` > Add Files and Images, use `control-openhands browser upload-via 'testid=add-files-and-images-button' <file>`, which answers the native file chooser.\n- The home composer has no `interactive-chat-box` wrapper: scope its chips and thumbnails with `testid=home-chat-launcher`.\n- `conversation events` rows show a message's text and its attachment count (`images: 1` for the image-only message, whose text is a blank); `--grep 'data:image/png;base64'` searches whole events and quotes the `image` content block in `excerpts`.\n- A user row with `images: 1` proves that the image reached the Agent Server, not the model. Only the reply proves that the model saw it. On agent-server 1.53.0 and older, `deepseek/*` profiles drop the image before the LLM call (`F05.image-only-send`). An image sent with no text then leaves deepseek-flash with an empty task. On 2026-10-08 it ran `env`, read the run directory and other files on the host, and decoded the picture from the event store with PIL to answer. Pause such a conversation (`control-openhands conversation pause <id>`), and use the question-plus-image send for the model check.\n- The pill resolves the conversation's stamped profile before the user-level default, so profiles that share a model are told apart by name. The stamp lives in this browser's localStorage (`openhands-agent-server-conversation-metadata`), not on the server (`api GET /api/conversations/<id>` has no profile name): read it with `browser eval`, not `api GET`.\n- The `Switched to profile` notes are client-side and gone after reload; assert the pill and `conversation status`.\n- `/plan` messages go to a separate planner conversation; the browser stays on the parent `/conversations/<id>`, so wait on the planner id from `conversation list`.\n- Only the newest goal status offers `Resume` (#18036): once a resumed goal has posted a newer status, the old `interrupted` row shows no button, also after a reload. Count `testid=goal-resume` after the resumed goal ends; while it runs the count is `0` anyway.\n- Opening the context popover logs a 404 for `/api/llm/balance` on the local backend in `browser errors --app-only`; it does not break the popover.\n- `--observe` takes the verbs' selector syntax: `testid=` and plain CSS segments are watched by an in-page MutationObserver (`observedBy` `mutation`), while `role=`, `text=` and other engines are polled every 20 ms (`poll-20ms`), which misses a state shorter than that; prefer a `testid=` or CSS form for anything brief.\n- At `narrow` (320 px) the overflow menu's Model submenu opens to the right of the menu and runs off screen: each profile row starts at x about 270 with width 210 (`browser bbox 'testid=chat-input-llm-profile-option-deepseek-flash'`, `insideViewport` `false`), so both names read `deepse…`. Pick profiles by test id: the click lands on the strip of about 50 px that stays on screen. Known issue: #18063 (filed from the follow-up noted in #18015; the submenu also runs past the bottom edge, so the LLM Profiles link cannot be clicked at `narrow`) (`F05.overflow-menu`).\n- Known issue [#17788](https://github.com/OpenHands/OpenHands/issues/17788) (`F05.attach-files`): Cloud attachment upload rejects valid runtime data when cached fields are empty strings. Live prerequisite: A Cloud sandbox whose cached runtime URL/key are blank while the fresh runtime lookup is already ready.\n- Known issue [#17621](https://github.com/OpenHands/OpenHands/issues/17621) (`F05.context-window-meter`): Compact context always fails after exactly 60s — client-side timeout aborts a condensation the server then completes. Live prerequisite: A real summarizing model and conversation whose condensation takes longer than 60 seconds.\n- Known issue [#17476](https://github.com/OpenHands/OpenHands/issues/17476) (`F05.context-window-meter`): Planning socket stats overwrite the main conversation's context meter. Live prerequisite: A real planning sub-conversation and main conversation with distinct nonzero usage stats.\n- Known issue [#17429](https://github.com/OpenHands/OpenHands/issues/17429) (`F05.slash-btw`): 500 error with /btw. Live prerequisite: The exact pending tool/streaming state that triggers the generator-pickle error, plus a working model for the side question.\n- Known issue [#17271](https://github.com/OpenHands/OpenHands/issues/17271) (`F05.context-window-meter`): New chat retains context/state from the previous conversation. Live prerequisite: A completed conversation with nonzero token usage before New Chat; a dummy-key run supplies no genuine context usage.\n- Known issue [#17263](https://github.com/OpenHands/OpenHands/issues/17263) (`F05.context-window-meter`): Context meter stays pinned after a successful compaction once the condenser has its own usage entry. Live prerequisite: A real successful compaction whose condenser usage remains larger than the default agent usage.\n- Known issue [#16851](https://github.com/OpenHands/OpenHands/issues/16851) (`F05.profile-identity`): wrong model is displayed when loading existing conversation. Live prerequisite: The original conversation/model-switch history and usage record showing Opus with a Fable selector; a dummy-key conversation has no usage evidence.\n- Known issue [#16835](https://github.com/OpenHands/OpenHands/issues/16835) (`F05.context-window-meter`): Agent Canvas sidebar shows \"Cache Hit: 0\" while backend correctly records nonzero cache_read_tokens. Live prerequisite: A working prompt-caching model/proxy and a real conversation reporting nonzero cache_read_tokens.\n- Known issue [#16430](https://github.com/OpenHands/OpenHands/issues/16430) (`F05.attach-files`): File upload does not have an indication of progress. Live prerequisite: A remote or throttled real upload that stays in flight long enough to inspect progress and composer state.\n- Known issue [#16339](https://github.com/OpenHands/OpenHands/issues/16339) (`F05.attach-files`): Agent not informed of file location for files uploaded via canvas in cloud. Live prerequisite: An authenticated Cloud upload sandbox and working model to inspect the delivered filename/path notification and read the uploaded file.\n- Known issue [#15782](https://github.com/OpenHands/OpenHands/issues/15782) (`F05.draft-persistence`): Prompt disappears when a conversation is stopped. Live prerequisite: Clarify whether prompt means unsent composer text or the already-sent history message, and capture that state before sidebar Stop.\n- Known issue [OpenHands/software-agent-sdk#5639](https://github.com/OpenHands/software-agent-sdk/issues/5639) (`F05.llm-profile-picker`): A profile switched before the first message can be absent from token/cost statistics after its successful turn. Live prerequisite: A working model, a profile switch before the first run, and nonzero usage/cost after that turn.\n- Known issue [OpenHands/software-agent-sdk#5627](https://github.com/OpenHands/software-agent-sdk/issues/5627) (`F05.slash-goal`): After backend restart the goal chip can remain Running while the conversation is paused, and Stop does not clear it. Live prerequisite: A working model goal loop active during graceful backend restart, then goal-status and resume readback.\n- Known issue [OpenHands/software-agent-sdk#5523](https://github.com/OpenHands/software-agent-sdk/issues/5523) (`F05.image-only-send`): DeepSeek V4.1 Flash images can be sent to another fallback profile because its vision capability is not detected. Live prerequisite: A working DeepSeek V4.1 Flash vision profile and another fallback profile, with an image-bearing turn.\n- Known issue [OpenHands/software-agent-sdk#5492](https://github.com/OpenHands/software-agent-sdk/issues/5492) (`F05.slash-goal`): A goal can stop as Complete when its judge replies with complete as the string false. Live prerequisite: A real goal judge response containing complete as the string false rather than a JSON boolean.\n- Known issue [OpenHands/software-agent-sdk#5330](https://github.com/OpenHands/software-agent-sdk/issues/5330) (`F05.slash-btw`): A /btw side question can prevent reuse of the normal conversation prompt cache. Live prerequisite: A working caching provider and comparable ordinary-message versus /btw prompt/cache records.\n- Known issue [OpenHands/software-agent-sdk#4498](https://github.com/OpenHands/software-agent-sdk/issues/4498) (`F05.context-window-meter`): Compaction can leave an orphaned tool call/result in the live history and make the next provider request fail. Live prerequisite: A real condensation splitting an action/result pair, followed by the next provider request.\n- Known issue [OpenHands/software-agent-sdk#4382](https://github.com/OpenHands/software-agent-sdk/issues/4382) (`F05.context-window-meter`): ACP cost can miss cache/thought usage, silently become zero, or inflate after stalled/reset cost updates. Live prerequisite: An authenticated ACP provider emitting cache/thought token usage or reset/stalled cumulative cost updates.\n\n- Known issue [#17239](https://github.com/OpenHands/OpenHands/issues/17239) (`F05.llm-profile-picker`): The reported model/profile picker can open above the viewport with unreachable rows. A frozen-target Local check with eight LLM profiles passed on Home at 1512x720, 1440x1000, 390x844 and 320x700, and in a genuine conversation at 1512x720 and 390x844: internal scrolling and selecting the first/last profile worked. The Cloud screenshot and real ACP multi-model picker remain unverified; they need an approved Cloud session or ACP account/adapter with a multi-row model catalogue. Local profile geometry is not proof of those branches. Run `backfill-18309-settings-worker`, Agent Server/SDK 1.54.0; evidence includes `F05.llm-profile-picker/17239-conversation-1512x720.png` and `17239-conversation-phone.png`.\n\n- Known issue [OpenHands/software-agent-sdk#5824](https://github.com/OpenHands/software-agent-sdk/issues/5824) (`F05.slash-goal`): resuming after a failed judge audit can show iteration 2 of a maximum of 1 and run beyond the selected goal cap. Live prerequisite: an agent/judge run whose audit fails, followed by a successful incomplete judge verdict after Resume; a controller-only test does not establish the Canvas banner outcome.\n\n- Known issue [OpenHands/software-agent-sdk#5889](https://github.com/OpenHands/software-agent-sdk/issues/5889) (`F05.attach-files`): a same-name attachment upload that fails part-way can replace the previous workspace file with partial content. Live prerequisite: an owned isolated backend with a controlled mid-upload filesystem failure (the original Linux file-size limit or an equivalent bounded child-process limit), a pre-existing target and a real Canvas overwrite. Never change resource limits on a shared runtime; a normal successful upload does not verify failure atomicity.\n\n- Known issue [OpenHands/software-agent-sdk#5820](https://github.com/OpenHands/software-agent-sdk/issues/5820) (`F05.context-window-meter`): manual compaction on a null/no-op condenser can return500. Verification is blocked by genuine populated usage/context metrics: idle and refused-loopback runs leave `usage=null`, so neither composer nor Usage panel offers Compact context. A separate direct endpoint check returned500 but is not a Canvas reproduction. Use an existing real conversation with usage and no condenser, or an authorized successful provider run; never fabricate stats to expose the action.", "features": [ { "id": "F05.send-message", "description": "Send is disabled while the field is empty; Shift+Enter inserts a newline; Enter sends, clears the field and the message persists after reload." }, { "id": "F05.draft-persistence", "description": "unsent text is restored per conversation (after navigation and reload) and on the home composer." }, { "id": "F05.composer-resize", "description": "the field grows with its content and shrinks when cleared; a top-edge grip lets the user drag the height." }, { "id": "F05.plus-menu", "description": "`+` opens a tools menu (Macros, Show skills/hooks/agent tools, Add Files and Images; Git tools in a repository); a macro fills the composer without sending." }, { "id": "F05.agent-profile-switch", "description": "before a conversation starts, `+` > Switch agent profile lists agent profiles and links to Manage agent profiles; it is absent in a started conversation." }, { "id": "F05.attach-files", "description": "picked files show as chips and images as thumbnails (with an Upload-as-file toggle), each removable; sent files land in the workspace and are named in the message." }, { "id": "F05.attach-size-limit", "description": "a file over 3 MB, or a selection over 3 MB in total, is refused with an error toast." }, { "id": "F05.image-only-send", "description": "an attached image alone, with an empty field, enables Send; the sent user message shows the thumbnail and carries the picture as an embedded data URL." }, { "id": "F05.paste", "description": "pasted clipboard text is inserted as plain text; pasted images and dropped files attach." }, { "id": "F05.dictation", "description": "the mic button dictates into the composer; without a microphone it toasts a failure and resets." }, { "id": "F05.llm-profile-picker", "description": "the model pill switches the running conversation's LLM profile; the next reply uses the new model." }, { "id": "F05.profile-identity", "description": "the pill names the profile the conversation was started with, also after a reload and when another profile uses the same model; changing the default profile in Settings changes the home pill, not an existing conversation's." }, { "id": "F05.overflow-menu", "description": "at very narrow widths the model pill collapses into a \"More input actions\" menu whose Model submenu fits the viewport and switches the profile; Escape closes the menu and returns focus to its trigger." }, { "id": "F05.context-window-meter", "description": "the ring shows context use; its popover opens the Usage tab and compacts the context." }, { "id": "F05.stop-resume", "description": "Stop pauses a running agent (status `Stopped`, Play button); Play resumes it." }, { "id": "F05.queued-message", "description": "a follow-up typed while the agent runs is accepted and answered." }, { "id": "F05.slash-menu", "description": "`/` opens a filtered command list (built-ins and skills) with arrow, Tab/Enter and Escape handling." }, { "id": "F05.slash-model", "description": "`/model` lists saved profiles; `/model <name>` switches profile." }, { "id": "F05.slash-btw", "description": "`/btw <question>` answers a side question in a dismissible card without adding to the main stream; a bare `/btw` toasts that a question is needed." }, { "id": "F05.slash-goal", "description": "`/goal [--max N] <objective>` runs a judged goal loop with a status banner, Stop and Resume; only the newest goal status offers Resume." }, { "id": "F05.slash-plan-code", "description": "`/plan` switches to plan mode (a planner sub-conversation), `/code` switches back." }, { "id": "F05.slash-plan-code-task", "description": "`/code <task>` switches to code mode and sends `<task>` to the code agent at once; `/plan <task>` switches to plan mode and sends `<task>` to the existing planner." }, { "id": "F05.plan-preview", "description": "the planner's Plan.md card shows the plan with Read more, View (Planner tab) and Build." }, { "id": "F05.build-plan-shortcut", "description": "Ctrl/Cmd+Enter in plan mode builds the plan with the code agent." }, { "id": "F05.phone", "description": "the composer fits a 390 px viewport with every control inline." } ], "scenarios": [ { "name": "F05--type-and-send", "title": "Type and send", "ids": [ "F05.send-message" ], "body": "On `/conversations/<id>` run `control-openhands browser attr 'testid=submit-button' aria-label` (`Send message`), `control-openhands browser enabled 'testid=submit-button'` (`false`), `control-openhands browser type 'testid=chat-input' 'Reply with only:'`, `control-openhands browser enabled 'testid=submit-button'` (`true`), `control-openhands browser press Shift+Enter`, `control-openhands browser type 'testid=chat-input' 'ok2'` and `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').innerText\"`: `Reply with only:\\nok2`. Run `control-openhands browser press Enter --selector 'testid=chat-input'`; the field is empty and `control-openhands browser count 'testid=stop-button'` is `1` while the agent runs. Then `control-openhands conversation wait <id> --fresh --timeout 180` (a fast model can finish before a separate `--until running` wait starts, which then times out on `finished`), `control-openhands browser reload` and `control-openhands browser count 'testid=user-message >> has-text=ok2'`: `1`; `control-openhands browser text 'testid=agent-message >> nth=-1'` is `ok2`.", "raw": "**Type and send (`F05.send-message`).** On `/conversations/<id>` run `control-openhands browser attr 'testid=submit-button' aria-label` (`Send message`), `control-openhands browser enabled 'testid=submit-button'` (`false`), `control-openhands browser type 'testid=chat-input' 'Reply with only:'`, `control-openhands browser enabled 'testid=submit-button'` (`true`), `control-openhands browser press Shift+Enter`, `control-openhands browser type 'testid=chat-input' 'ok2'` and `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').innerText\"`: `Reply with only:\\nok2`. Run `control-openhands browser press Enter --selector 'testid=chat-input'`; the field is empty and `control-openhands browser count 'testid=stop-button'` is `1` while the agent runs. Then `control-openhands conversation wait <id> --fresh --timeout 180` (a fast model can finish before a separate `--until running` wait starts, which then times out on `finished`), `control-openhands browser reload` and `control-openhands browser count 'testid=user-message >> has-text=ok2'`: `1`; `control-openhands browser text 'testid=agent-message >> nth=-1'` is `ok2`.", "source_line": 60, "expected": [ "Send is disabled while the field is empty; Shift+Enter inserts a newline; Enter sends, clears the field and the message persists after reload." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--draft-survives-navigation", "title": "Draft survives navigation", "ids": [ "F05.draft-persistence" ], "body": "Run `control-openhands browser type 'testid=chat-input' 'QA draft text'` and wait out the 500 ms save: `control-openhands browser eval \"new Promise(r=>setTimeout(()=>r(JSON.parse(localStorage.getItem('conversation-state-<id>')).draftMessage),1500))\"` returns `QA draft text`. Then `control-openhands browser click 'testid=backend-selector-settings-link' --expect-url settings`, `control-openhands browser back` and the `innerText` eval above: `QA draft text`. `control-openhands browser reload` restores it again. A draft typed just before leaving is flushed, not lost: `control-openhands browser type 'testid=chat-input' ' plus fast'` and at once `control-openhands browser click 'testid=backend-selector-settings-link' --expect-url settings`, `control-openhands browser back`: the eval reads `QA draft text plus fast`. The same holds for an immediate sidebar switch to another conversation and back (that other conversation's composer stays empty). Clear with `control-openhands browser fill 'testid=chat-input' ''` and wait a second before navigating.", "raw": "**Draft survives navigation (`F05.draft-persistence`).** Run `control-openhands browser type 'testid=chat-input' 'QA draft text'` and wait out the 500 ms save: `control-openhands browser eval \"new Promise(r=>setTimeout(()=>r(JSON.parse(localStorage.getItem('conversation-state-<id>')).draftMessage),1500))\"` returns `QA draft text`. Then `control-openhands browser click 'testid=backend-selector-settings-link' --expect-url settings`, `control-openhands browser back` and the `innerText` eval above: `QA draft text`. `control-openhands browser reload` restores it again. A draft typed just before leaving is flushed, not lost: `control-openhands browser type 'testid=chat-input' ' plus fast'` and at once `control-openhands browser click 'testid=backend-selector-settings-link' --expect-url settings`, `control-openhands browser back`: the eval reads `QA draft text plus fast`. The same holds for an immediate sidebar switch to another conversation and back (that other conversation's composer stays empty). Clear with `control-openhands browser fill 'testid=chat-input' ''` and wait a second before navigating.", "source_line": 61, "expected": [ "unsent text is restored per conversation (after navigation and reload) and on the home composer." ], "notes": [ "Known issue [#15782](https://github.com/OpenHands/OpenHands/issues/15782) (`F05.draft-persistence`): Prompt disappears when a conversation is stopped. Live prerequisite: Clarify whether prompt means unsent composer text or the already-sent history message, and capture that state before sidebar Stop." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--home-draft", "title": "Home draft", "ids": [ "F05.draft-persistence" ], "body": "Run `control-openhands browser goto /`, `control-openhands browser type 'testid=chat-input' 'QA home draft'`, `control-openhands browser goto /settings/secrets`, `control-openhands browser goto /`, then the `innerText` eval: `QA home draft`; `control-openhands browser storage --session` lists `oh:home-prompt-draft`. The automation setup page's docked composer (`/automations/setup`, F22) keeps no draft and never touches the Home one. With `QA home draft` still in the Home composer, run `control-openhands browser goto /automations`, `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup'` and `control-openhands browser wait 'testid=chat-input'`: the `innerText` eval is empty. `control-openhands browser type 'testid=chat-input' 'Reply with only the word ok2. Do not run any tools.'`, then `control-openhands browser eval \"new Promise(r=>setTimeout(()=>r(sessionStorage.getItem('oh:home-prompt-draft')),1500))\"` still returns `QA home draft`. `control-openhands browser click 'testid=submit-button' --expect-url '/conversations/'` and `control-openhands browser click 'testid=sidebar-conversations-link' --expect-url '/conversations(\\?|$)'`: the Home composer shows `QA home draft`. Clear it (`fill 'testid=chat-input' ''`, wait a second) and repeat the setup send: New Chat is empty and `browser storage --session --values` has no `oh:home-prompt-draft`. Unsent text in the setup composer is lost when the user leaves that page (by design).", "raw": "**Home draft (`F05.draft-persistence`).** Run `control-openhands browser goto /`, `control-openhands browser type 'testid=chat-input' 'QA home draft'`, `control-openhands browser goto /settings/secrets`, `control-openhands browser goto /`, then the `innerText` eval: `QA home draft`; `control-openhands browser storage --session` lists `oh:home-prompt-draft`. The automation setup page's docked composer (`/automations/setup`, F22) keeps no draft and never touches the Home one. With `QA home draft` still in the Home composer, run `control-openhands browser goto /automations`, `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup'` and `control-openhands browser wait 'testid=chat-input'`: the `innerText` eval is empty. `control-openhands browser type 'testid=chat-input' 'Reply with only the word ok2. Do not run any tools.'`, then `control-openhands browser eval \"new Promise(r=>setTimeout(()=>r(sessionStorage.getItem('oh:home-prompt-draft')),1500))\"` still returns `QA home draft`. `control-openhands browser click 'testid=submit-button' --expect-url '/conversations/'` and `control-openhands browser click 'testid=sidebar-conversations-link' --expect-url '/conversations(\\?|$)'`: the Home composer shows `QA home draft`. Clear it (`fill 'testid=chat-input' ''`, wait a second) and repeat the setup send: New Chat is empty and `browser storage --session --values` has no `oh:home-prompt-draft`. Unsent text in the setup composer is lost when the user leaves that page (by design).", "source_line": 62, "expected": [ "unsent text is restored per conversation (after navigation and reload) and on the home composer." ], "notes": [ "Known issue [#15782](https://github.com/OpenHands/OpenHands/issues/15782) (`F05.draft-persistence`): Prompt disappears when a conversation is stopped. Live prerequisite: Clarify whether prompt means unsent composer text or the already-sent history message, and capture that state before sidebar Stop." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--auto-resize", "title": "Auto-resize", "ids": [ "F05.composer-resize" ], "body": "On `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`; Home draft left `/`) type `line`, then eight times `control-openhands browser press Shift+Enter` and `control-openhands browser type 'testid=chat-input' 'lineN'` (`line2`…`line9`): `control-openhands browser bbox 'testid=chat-input'` grows by 20 px per line, from `height` `20` to `180`; after `control-openhands browser fill 'testid=chat-input' ''` it is `20`. `control-openhands browser count '#resize-grip'` is `1`. `control-openhands browser drag '#resize-grip' --by 0,-200` makes the empty field `220` high and `control-openhands browser drag '#resize-grip' --by 0,150` brings it to `70`; after `control-openhands browser reload` it is `20` again (the dragged height is not kept).", "raw": "**Auto-resize (`F05.composer-resize`).** On `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`; Home draft left `/`) type `line`, then eight times `control-openhands browser press Shift+Enter` and `control-openhands browser type 'testid=chat-input' 'lineN'` (`line2`…`line9`): `control-openhands browser bbox 'testid=chat-input'` grows by 20 px per line, from `height` `20` to `180`; after `control-openhands browser fill 'testid=chat-input' ''` it is `20`. `control-openhands browser count '#resize-grip'` is `1`. `control-openhands browser drag '#resize-grip' --by 0,-200` makes the empty field `220` high and `control-openhands browser drag '#resize-grip' --by 0,150` brings it to `70`; after `control-openhands browser reload` it is `20` again (the dragged height is not kept).", "source_line": 63, "expected": [ "the field grows with its content and shrinks when cleared; a top-edge grip lets the user drag the height." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--tools-menu-and-macros", "title": "Tools menu and macros", "ids": [ "F05.plus-menu" ], "body": "On `/conversations/<id>` run `control-openhands browser click 'testid=chat-plus-button'` and `control-openhands browser testids 'testid=tools-context-menu'`: `macros-button`, `show-skills-button`, `show-hooks-button`, `show-agent-tools-button`, `add-files-and-images-button` (no `git-tools-button` outside a repository). `control-openhands browser hover 'testid=macros-button'` shows `macros-submenu` with `increase-test-coverage-button`, `fix-readme-button`, `auto-merge-prs-button`, `clean-dependencies-button`. `control-openhands browser click 'testid=fix-readme-button'` closes the menu and fills the composer with `Please look at the README and make the following improvements…`; the `testid=user-message` count is unchanged. Clear with `control-openhands browser fill 'testid=chat-input' ''`. Open the menu again and `control-openhands browser press Escape`: `browser count 'testid=tools-context-menu'` is `0` and `control-openhands browser eval \"document.activeElement.dataset.testid\"` is `chat-plus-button`.", "raw": "**Tools menu and macros (`F05.plus-menu`).** On `/conversations/<id>` run `control-openhands browser click 'testid=chat-plus-button'` and `control-openhands browser testids 'testid=tools-context-menu'`: `macros-button`, `show-skills-button`, `show-hooks-button`, `show-agent-tools-button`, `add-files-and-images-button` (no `git-tools-button` outside a repository). `control-openhands browser hover 'testid=macros-button'` shows `macros-submenu` with `increase-test-coverage-button`, `fix-readme-button`, `auto-merge-prs-button`, `clean-dependencies-button`. `control-openhands browser click 'testid=fix-readme-button'` closes the menu and fills the composer with `Please look at the README and make the following improvements…`; the `testid=user-message` count is unchanged. Clear with `control-openhands browser fill 'testid=chat-input' ''`. Open the menu again and `control-openhands browser press Escape`: `browser count 'testid=tools-context-menu'` is `0` and `control-openhands browser eval \"document.activeElement.dataset.testid\"` is `chat-plus-button`.", "source_line": 64, "expected": [ "`+` opens a tools menu (Macros, Show skills/hooks/agent tools, Add Files and Images; Git tools in a repository); a macro fills the composer without sending." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--show-dialogs", "title": "Show dialogs", "ids": [ "F05.plus-menu" ], "body": "For each of `show-skills-button`, `show-hooks-button`, `show-agent-tools-button`: `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser click 'testid=show-hooks-button'` (etc.), `control-openhands browser text 'role=dialog'` (starts `Available Skills`, `Available Hooks`, `Agent Tools & Metadata`), then `control-openhands browser press Escape` and `control-openhands browser count 'role=dialog'`: `0`.", "raw": "**Show dialogs (`F05.plus-menu`).** For each of `show-skills-button`, `show-hooks-button`, `show-agent-tools-button`: `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser click 'testid=show-hooks-button'` (etc.), `control-openhands browser text 'role=dialog'` (starts `Available Skills`, `Available Hooks`, `Agent Tools & Metadata`), then `control-openhands browser press Escape` and `control-openhands browser count 'role=dialog'`: `0`.", "source_line": 65, "expected": [ "`+` opens a tools menu (Macros, Show skills/hooks/agent tools, Add Files and Images; Git tools in a repository); a macro fills the composer without sending." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--agent-profile-before-start", "title": "Agent profile before start", "ids": [ "F05.agent-profile-switch" ], "body": "Run `control-openhands browser goto /`, `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser hover 'testid=switch-agent-profile-button'` and `control-openhands browser text 'testid=agent-profile-submenu'`: `AVAILABLE PROFILES`, `default` (checked; `chat-input-agent-profile-option-default`), `Manage agent profiles`. `control-openhands browser click 'testid=agent-profile-submenu >> text=Manage agent profiles' --expect-url 'settings/agents'` lands on `/settings/agents`. In a started conversation (`control-openhands browser goto /conversations/<id>`) `control-openhands browser click 'testid=chat-plus-button'` then `control-openhands browser count 'testid=switch-agent-profile-button'` is `0`.", "raw": "**Agent profile before start (`F05.agent-profile-switch`).** Run `control-openhands browser goto /`, `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser hover 'testid=switch-agent-profile-button'` and `control-openhands browser text 'testid=agent-profile-submenu'`: `AVAILABLE PROFILES`, `default` (checked; `chat-input-agent-profile-option-default`), `Manage agent profiles`. `control-openhands browser click 'testid=agent-profile-submenu >> text=Manage agent profiles' --expect-url 'settings/agents'` lands on `/settings/agents`. In a started conversation (`control-openhands browser goto /conversations/<id>`) `control-openhands browser click 'testid=chat-plus-button'` then `control-openhands browser count 'testid=switch-agent-profile-button'` is `0`.", "source_line": 66, "expected": [ "before a conversation starts, `+` > Switch agent profile lists agent profiles and links to Manage agent profiles; it is absent in a started conversation." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--attach-a-file-and-an-image", "title": "Attach a file and an image", "ids": [ "F05.attach-files" ], "body": "On `/conversations/<id>` run `control-openhands browser upload 'testid=upload-image-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-image.png\" \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"` and `control-openhands browser screenshot 'testid=interactive-chat-box' --feature F05.attach-files --name attached`: a `qa-note.txt TXT` chip and a thumbnail. `control-openhands browser count 'testid=interactive-chat-box >> alt=qa-image.png'` is `1`. `control-openhands browser click 'role=button[name=\"Upload as file\"]'` turns the image toggle into `Do not upload as file` (`control-openhands browser attr 'role=button[name=\"Do not upload as file\"]' aria-pressed` is `true`); click it again to restore. The remove buttons are named: `control-openhands browser snapshot 'testid=interactive-chat-box'` lists `button \"Remove qa-note.txt\"` and `button \"Remove image\"`. Remove the image with `control-openhands browser click 'testid=interactive-chat-box >> role=button[name=\"Remove image\"]'`; the `alt=qa-image.png` count is `0`. Then `control-openhands browser type 'testid=chat-input' 'Reply with only the content of the attached file.'`, `control-openhands browser press Enter --selector 'testid=chat-input'`, wait as in Type and send, and `control-openhands browser text 'testid=user-message >> nth=-1'`: it ends `NEW FILES ADDED: qa-note.txt`. `control-openhands conversation events <id> --kinds MessageEvent --last 3` shows the reply `QA attachment 7731`, and `qa-note.txt` sits in the conversation `workspace` printed by `control-openhands conversation status <id>`. Attachments added on Home before the conversation exists upload after it is created: with no workspace or a Local Repo workspace they land in it (`browser network --filter upload` shows `200`). Expected: the same in New Worktree mode. Known failure (reproduced 2026-10-16, SDK 1.55.0, [#18421](https://github.com/OpenHands/OpenHands/issues/18421)): run `control-openhands fixture git-repo --name qa-repo`, `control-openhands browser goto /`, `control-openhands workspace open qa-repo`, `control-openhands browser click 'testid=workspace-mode-selector'`, `control-openhands browser click 'testid=workspace-mode-selector-option-new_worktree'`, `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser upload-via 'testid=add-files-and-images-button' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"`, type `Reply with only the word received. Do not run any tools.` and `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --expect-url '/conversations/[^/]+$'`: `control-openhands browser toasts --history` shows `HTTP request failed (422 Unprocessable Entity): {\"detail\":\"path must be inside the conversation workspace\"}` and `browser network --filter upload` a `422` for `?path=<run>/workspace/qa-repo/qa-note.txt`, while `conversation status <id>` reports the worktree `/tmp/conversation-worktrees/<id>/qa-repo`; the message is sent without the file. The same file attached inside that conversation afterwards uploads to the worktree (`200`). Restore with `testid=workspace-mode-selector-option-local_repo`.\n ```sh known-bug F05.attach-files home-new-worktree\n control-openhands browser viewport desktop\n control-openhands fixture git-repo --name qa-repo\n control-openhands fixture file --name qa-note.txt --content \"QA attachment 7731\"\n control-openhands browser goto /\n control-openhands browser fill 'testid=chat-input' ''\n control-openhands workspace open qa-repo\n control-openhands browser click 'testid=workspace-mode-selector'\n control-openhands browser click 'testid=workspace-mode-selector-option-new_worktree'\n control-openhands assert --field text --matches 'New Worktree' -- browser text 'testid=home-git-control-bar-preview'\n control-openhands browser click 'testid=chat-plus-button'\n control-openhands browser upload-via 'testid=add-files-and-images-button' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"\n control-openhands browser type 'testid=chat-input' 'Reply with only the word received. Do not run any tools.'\n control-openhands browser network --clear\n control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --expect-url '/conversations/[^/]+$'\n control-openhands browser wait 'testid=user-message' --timeout 30000\n control-openhands browser eval \"new Promise(r=>setTimeout(r,5000))\"\n control-openhands browser screenshot --feature F05.attach-files --name home-new-worktree\n control-openhands assert --field recent.0.status --equals 200 --bug-shows 422 -- browser network --filter file/upload # bug\n ```", "raw": "**Attach a file and an image (`F05.attach-files`).** On `/conversations/<id>` run `control-openhands browser upload 'testid=upload-image-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-image.png\" \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"` and `control-openhands browser screenshot 'testid=interactive-chat-box' --feature F05.attach-files --name attached`: a `qa-note.txt TXT` chip and a thumbnail. `control-openhands browser count 'testid=interactive-chat-box >> alt=qa-image.png'` is `1`. `control-openhands browser click 'role=button[name=\"Upload as file\"]'` turns the image toggle into `Do not upload as file` (`control-openhands browser attr 'role=button[name=\"Do not upload as file\"]' aria-pressed` is `true`); click it again to restore. The remove buttons are named: `control-openhands browser snapshot 'testid=interactive-chat-box'` lists `button \"Remove qa-note.txt\"` and `button \"Remove image\"`. Remove the image with `control-openhands browser click 'testid=interactive-chat-box >> role=button[name=\"Remove image\"]'`; the `alt=qa-image.png` count is `0`. Then `control-openhands browser type 'testid=chat-input' 'Reply with only the content of the attached file.'`, `control-openhands browser press Enter --selector 'testid=chat-input'`, wait as in Type and send, and `control-openhands browser text 'testid=user-message >> nth=-1'`: it ends `NEW FILES ADDED: qa-note.txt`. `control-openhands conversation events <id> --kinds MessageEvent --last 3` shows the reply `QA attachment 7731`, and `qa-note.txt` sits in the conversation `workspace` printed by `control-openhands conversation status <id>`. Attachments added on Home before the conversation exists upload after it is created: with no workspace or a Local Repo workspace they land in it (`browser network --filter upload` shows `200`). Expected: the same in New Worktree mode. Known failure (reproduced 2026-10-16, SDK 1.55.0, [#18421](https://github.com/OpenHands/OpenHands/issues/18421)): run `control-openhands fixture git-repo --name qa-repo`, `control-openhands browser goto /`, `control-openhands workspace open qa-repo`, `control-openhands browser click 'testid=workspace-mode-selector'`, `control-openhands browser click 'testid=workspace-mode-selector-option-new_worktree'`, `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser upload-via 'testid=add-files-and-images-button' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"`, type `Reply with only the word received. Do not run any tools.` and `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --expect-url '/conversations/[^/]+$'`: `control-openhands browser toasts --history` shows `HTTP request failed (422 Unprocessable Entity): {\"detail\":\"path must be inside the conversation workspace\"}` and `browser network --filter upload` a `422` for `?path=<run>/workspace/qa-repo/qa-note.txt`, while `conversation status <id>` reports the worktree `/tmp/conversation-worktrees/<id>/qa-repo`; the message is sent without the file. The same file attached inside that conversation afterwards uploads to the worktree (`200`). Restore with `testid=workspace-mode-selector-option-local_repo`.\n ```sh known-bug F05.attach-files home-new-worktree\n control-openhands browser viewport desktop\n control-openhands fixture git-repo --name qa-repo\n control-openhands fixture file --name qa-note.txt --content \"QA attachment 7731\"\n control-openhands browser goto /\n control-openhands browser fill 'testid=chat-input' ''\n control-openhands workspace open qa-repo\n control-openhands browser click 'testid=workspace-mode-selector'\n control-openhands browser click 'testid=workspace-mode-selector-option-new_worktree'\n control-openhands assert --field text --matches 'New Worktree' -- browser text 'testid=home-git-control-bar-preview'\n control-openhands browser click 'testid=chat-plus-button'\n control-openhands browser upload-via 'testid=add-files-and-images-button' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"\n control-openhands browser type 'testid=chat-input' 'Reply with only the word received. Do not run any tools.'\n control-openhands browser network --clear\n control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --expect-url '/conversations/[^/]+$'\n control-openhands browser wait 'testid=user-message' --timeout 30000\n control-openhands browser eval \"new Promise(r=>setTimeout(r,5000))\"\n control-openhands browser screenshot --feature F05.attach-files --name home-new-worktree\n control-openhands assert --field recent.0.status --equals 200 --bug-shows 422 -- browser network --filter file/upload # bug\n ```", "source_line": 67, "expected": [ "picked files show as chips and images as thumbnails (with an Upload-as-file toggle), each removable; sent files land in the workspace and are named in the message." ], "notes": [ "Known issue [#17788](https://github.com/OpenHands/OpenHands/issues/17788) (`F05.attach-files`): Cloud attachment upload rejects valid runtime data when cached fields are empty strings. Live prerequisite: A Cloud sandbox whose cached runtime URL/key are blank while the fresh runtime lookup is already ready.", "Known issue [#16430](https://github.com/OpenHands/OpenHands/issues/16430) (`F05.attach-files`): File upload does not have an indication of progress. Live prerequisite: A remote or throttled real upload that stays in flight long enough to inspect progress and composer state.", "Known issue [#16339](https://github.com/OpenHands/OpenHands/issues/16339) (`F05.attach-files`): Agent not informed of file location for files uploaded via canvas in cloud. Live prerequisite: An authenticated Cloud upload sandbox and working model to inspect the delivered filename/path notification and read the uploaded file.", "Known issue [OpenHands/software-agent-sdk#5889](https://github.com/OpenHands/software-agent-sdk/issues/5889) (`F05.attach-files`): a same-name attachment upload that fails part-way can replace the previous workspace file with partial content. Live prerequisite: an owned isolated backend with a controlled mid-upload filesystem failure (the original Linux file-size limit or an equivalent bounded child-process limit), a pre-existing target and a real Canvas overwrite. Never change resource limits on a shared runtime; a normal successful upload does not verify failure atomicity." ], "annotations": [ "Known failure (reproduced 2026-10-16, SDK 1.55.0, [#18421](https://github.com/OpenHands/OpenHands/issues/18421)): run `control-openhands fixture git-repo --name qa-repo`, `control-openhands browser goto /`, `control-openhands workspace open qa-repo`, `control-openhands browser click 'testid=workspace-mode-selector'`, `control-openhands browser click 'testid=workspace-mode-selector-option-new_worktree'`, `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser upload-via 'testid=add-files-and-images-button' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"`, type `Reply with only the word received. Do not run any tools.` and `control-openhands browser click 'testid=home-chat-launcher >> testid=submit-button' --expect-url '/conversations/[^/]+$'`: `control-openhands browser toasts --history` shows `HTTP request failed (422 Unprocessable Entity): {\"detail\":\"path must be inside the conversation workspace\"}` and `browser network --filter upload` a `422` for `?path=<run>/workspace/qa-repo/qa-note.txt`, while `conversation status <id>` reports the worktree `/tmp/conversation-worktrees/<id>/qa-repo`; the message is sent without the file. The same file attached inside that conversation afterwards uploads to the worktree (`200`). Restore with `testid=workspace-mode-selector-option-local_repo`." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F05--size-limit", "title": "Size limit", "ids": [ "F05.attach-size-limit" ], "body": "Arrange `head -c 4194304 /dev/zero > \"$OH_VERIFY_RUN/evidence/_fixtures/qa-big.bin\"` (and `qa-two-a.bin`, `qa-two-b.bin` with `head -c 2097152`). Run `control-openhands browser upload 'testid=upload-image-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-big.bin\"` and `control-openhands browser toasts`: `Error: Files exceeding 3MB are not allowed: qa-big.bin`. Uploading `qa-two-a.bin` and `qa-two-b.bin` together toasts `Error: Total file size would be 4.0MB, exceeding the 3MB limit. Please select fewer or smaller files.` `control-openhands browser count 'testid=interactive-chat-box >> text=qa-big.bin'` is `0`.", "raw": "**Size limit (`F05.attach-size-limit`).** Arrange `head -c 4194304 /dev/zero > \"$OH_VERIFY_RUN/evidence/_fixtures/qa-big.bin\"` (and `qa-two-a.bin`, `qa-two-b.bin` with `head -c 2097152`). Run `control-openhands browser upload 'testid=upload-image-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-big.bin\"` and `control-openhands browser toasts`: `Error: Files exceeding 3MB are not allowed: qa-big.bin`. Uploading `qa-two-a.bin` and `qa-two-b.bin` together toasts `Error: Total file size would be 4.0MB, exceeding the 3MB limit. Please select fewer or smaller files.` `control-openhands browser count 'testid=interactive-chat-box >> text=qa-big.bin'` is `0`.", "source_line": 88, "expected": [ "a file over 3 MB, or a selection over 3 MB in total, is refused with an error toast." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--image-alone", "title": "Image alone", "ids": [ "F05.image-only-send" ], "body": "From the home composer: `control-openhands browser goto /`, `control-openhands browser enabled 'testid=submit-button'` (`false`, empty field), `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser upload-via 'testid=add-files-and-images-button' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-image.png\"`, then `control-openhands browser count 'testid=home-chat-launcher >> alt=qa-image.png'` (`1`) and `control-openhands browser enabled 'testid=submit-button'`: `true` while `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').innerText\"` is still empty (`control-openhands browser screenshot 'testid=home-chat-launcher' --feature F05.image-only-send --name image-alone-send-enabled`). Run `control-openhands browser click 'testid=submit-button' --expect-url '/conversations/'` and read `<image-id>` from `control-openhands browser url`. After `control-openhands browser wait 'testid=user-message'` and `control-openhands browser wait 'testid=chat-message-sending' --state detached --timeout 5000` (the pending bubble carries `testid=user-message` too, so wait for its `Sending...` to go, not only for a bubble; the wait returns `\"state\": \"detached\"` at once when the indicator has already gone, the usual case on a local stack), `control-openhands browser count 'testid=user-message >> testid=image-preview'` is `1`, and `control-openhands browser count 'testid=chat-message-sending'` and `control-openhands browser count 'testid=chat-message-retry'` are `0`. Server side, `control-openhands conversation events <image-id> --kinds MessageEvent --grep 'data:image/png;base64'` matches the one user row (`count` `1`; its `text` is a blank `\" \"`, the row carries `images: 1`, and the excerpt shows the `\"type\":\"image\"` block with `image_urls`), and `control-openhands conversation events <image-id> --kinds MessageEvent` (default `--last`) lists the same row with `images: 1` and no excerpt. `--kinds` filters before `--last` counts, so `--kinds MessageEvent --last 1` is that same row. To check that the model is handed the picture, send the image from Home together with `Which two colours are the bars in this image? Answer with the colour names only. Do not run any tools.` and read `control-openhands conversation events <id> --kinds MessageEvent`. Expect `orange` and `blue` in the agent row and `--kinds ActionEvent` `count` `0` (`fixture image` draws those bars).", "raw": "**Image alone (`F05.image-only-send`).** From the home composer: `control-openhands browser goto /`, `control-openhands browser enabled 'testid=submit-button'` (`false`, empty field), `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser upload-via 'testid=add-files-and-images-button' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-image.png\"`, then `control-openhands browser count 'testid=home-chat-launcher >> alt=qa-image.png'` (`1`) and `control-openhands browser enabled 'testid=submit-button'`: `true` while `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').innerText\"` is still empty (`control-openhands browser screenshot 'testid=home-chat-launcher' --feature F05.image-only-send --name image-alone-send-enabled`). Run `control-openhands browser click 'testid=submit-button' --expect-url '/conversations/'` and read `<image-id>` from `control-openhands browser url`. After `control-openhands browser wait 'testid=user-message'` and `control-openhands browser wait 'testid=chat-message-sending' --state detached --timeout 5000` (the pending bubble carries `testid=user-message` too, so wait for its `Sending...` to go, not only for a bubble; the wait returns `\"state\": \"detached\"` at once when the indicator has already gone, the usual case on a local stack), `control-openhands browser count 'testid=user-message >> testid=image-preview'` is `1`, and `control-openhands browser count 'testid=chat-message-sending'` and `control-openhands browser count 'testid=chat-message-retry'` are `0`. Server side, `control-openhands conversation events <image-id> --kinds MessageEvent --grep 'data:image/png;base64'` matches the one user row (`count` `1`; its `text` is a blank `\" \"`, the row carries `images: 1`, and the excerpt shows the `\"type\":\"image\"` block with `image_urls`), and `control-openhands conversation events <image-id> --kinds MessageEvent` (default `--last`) lists the same row with `images: 1` and no excerpt. `--kinds` filters before `--last` counts, so `--kinds MessageEvent --last 1` is that same row. To check that the model is handed the picture, send the image from Home together with `Which two colours are the bars in this image? Answer with the colour names only. Do not run any tools.` and read `control-openhands conversation events <id> --kinds MessageEvent`. Expect `orange` and `blue` in the agent row and `--kinds ActionEvent` `count` `0` (`fixture image` draws those bars).", "source_line": 89, "expected": [ "an attached image alone, with an empty field, enables Send; the sent user message shows the thumbnail and carries the picture as an embedded data URL." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5523](https://github.com/OpenHands/software-agent-sdk/issues/5523) (`F05.image-only-send`): DeepSeek V4.1 Flash images can be sent to another fallback profile because its vision capability is not detected. Live prerequisite: A working DeepSeek V4.1 Flash vision profile and another fallback profile, with an image-bearing turn." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--paste-text", "title": "Paste text", "ids": [ "F05.paste" ], "body": "Run `control-openhands browser type 'testid=chat-input' 'QA paste text'`, `control-openhands browser press Control+a`, `control-openhands browser press Control+c`, `control-openhands browser press Delete`, `control-openhands browser press Control+v`, then `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').innerHTML\"`: `QA paste text`, no markup. On a cleared field `control-openhands browser paste 'testid=chat-input' --file \"$OH_VERIFY_RUN/evidence/_fixtures/qa-image.png\"` attaches the image (`control-openhands browser count 'testid=interactive-chat-box >> alt=qa-image.png'` is `1`, with an `Upload as file` toggle) and `control-openhands browser paste 'testid=chat-input' --text 'QA pasted plain'` inserts plain text (`innerHTML` is `QA pasted plain`). `control-openhands browser drop-files 'testid=chat-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\" --stage over` makes `control-openhands browser text 'testid=interactive-chat-box'` start `Drop your files here`; the same command without `--stage` drops it: a `qa-note.txt` `TXT` chip appears. Remove the chip with `control-openhands browser click 'testid=interactive-chat-box >> role=button[name=\"Remove qa-note.txt\"]'` and the image as in Attach; clear the field.", "raw": "**Paste text (`F05.paste`).** Run `control-openhands browser type 'testid=chat-input' 'QA paste text'`, `control-openhands browser press Control+a`, `control-openhands browser press Control+c`, `control-openhands browser press Delete`, `control-openhands browser press Control+v`, then `control-openhands browser eval \"document.querySelector('[data-testid=chat-input]').innerHTML\"`: `QA paste text`, no markup. On a cleared field `control-openhands browser paste 'testid=chat-input' --file \"$OH_VERIFY_RUN/evidence/_fixtures/qa-image.png\"` attaches the image (`control-openhands browser count 'testid=interactive-chat-box >> alt=qa-image.png'` is `1`, with an `Upload as file` toggle) and `control-openhands browser paste 'testid=chat-input' --text 'QA pasted plain'` inserts plain text (`innerHTML` is `QA pasted plain`). `control-openhands browser drop-files 'testid=chat-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\" --stage over` makes `control-openhands browser text 'testid=interactive-chat-box'` start `Drop your files here`; the same command without `--stage` drops it: a `qa-note.txt` `TXT` chip appears. Remove the chip with `control-openhands browser click 'testid=interactive-chat-box >> role=button[name=\"Remove qa-note.txt\"]'` and the image as in Attach; clear the field.", "source_line": 90, "expected": [ "pasted clipboard text is inserted as plain text; pasted images and dropped files attach." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--dictation-failure", "title": "Dictation failure", "ids": [ "F05.dictation" ], "body": "Run `control-openhands browser click 'testid=chat-dictation-button' --observe '[role=status]' --observe-ms 3000`: the observed toasts include `Dictation failed. Check microphone access and your voice input settings.` and `control-openhands browser attr 'testid=chat-dictation-button' aria-label` is back to `Start dictation`. Real dictation is blocked without a microphone.", "raw": "**Dictation failure (`F05.dictation`).** Run `control-openhands browser click 'testid=chat-dictation-button' --observe '[role=status]' --observe-ms 3000`: the observed toasts include `Dictation failed. Check microphone access and your voice input settings.` and `control-openhands browser attr 'testid=chat-dictation-button' aria-label` is back to `Start dictation`. Real dictation is blocked without a microphone.", "source_line": 91, "expected": [ "the mic button dictates into the composer; without a microphone it toasts a failure and resets." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F05--switch-llm-profile", "title": "Switch LLM profile", "ids": [ "F05.llm-profile-picker" ], "body": "Run `control-openhands browser click 'testid=chat-input-llm-profile'` and `control-openhands browser text 'testid=chat-input-llm-profile-popover'`: `AVAILABLE PROFILES`, `deepseek-flash` / `deepseek/deepseek-flash`, `deepseek-pro` / `deepseek/deepseek-v4-pro`, `LLM Profiles`. Run `control-openhands browser click 'testid=chat-input-llm-profile-option-deepseek-pro'`, `control-openhands browser wait-text 'Switched to profile' --timeout 15000`, `control-openhands browser text 'testid=chat-input-llm-profile'` (`deepseek-pro`) and `control-openhands conversation status <id>` (`model` is `deepseek/deepseek-v4-pro`). Send `Reply with only: pro-ok` as in Type and send; the reply is `pro-ok`. After `control-openhands browser reload` the pill still reads `deepseek-pro`. Switch back with `control-openhands browser click 'testid=chat-input-llm-profile'` and `control-openhands browser click 'testid=chat-input-llm-profile-option-deepseek-flash'`.", "raw": "**Switch LLM profile (`F05.llm-profile-picker`).** Run `control-openhands browser click 'testid=chat-input-llm-profile'` and `control-openhands browser text 'testid=chat-input-llm-profile-popover'`: `AVAILABLE PROFILES`, `deepseek-flash` / `deepseek/deepseek-flash`, `deepseek-pro` / `deepseek/deepseek-v4-pro`, `LLM Profiles`. Run `control-openhands browser click 'testid=chat-input-llm-profile-option-deepseek-pro'`, `control-openhands browser wait-text 'Switched to profile' --timeout 15000`, `control-openhands browser text 'testid=chat-input-llm-profile'` (`deepseek-pro`) and `control-openhands conversation status <id>` (`model` is `deepseek/deepseek-v4-pro`). Send `Reply with only: pro-ok` as in Type and send; the reply is `pro-ok`. After `control-openhands browser reload` the pill still reads `deepseek-pro`. Switch back with `control-openhands browser click 'testid=chat-input-llm-profile'` and `control-openhands browser click 'testid=chat-input-llm-profile-option-deepseek-flash'`.", "source_line": 92, "expected": [ "the model pill switches the running conversation's LLM profile; the next reply uses the new model." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5639](https://github.com/OpenHands/software-agent-sdk/issues/5639) (`F05.llm-profile-picker`): A profile switched before the first message can be absent from token/cost statistics after its successful turn. Live prerequisite: A working model, a profile switch before the first run, and nonzero usage/cost after that turn.", "Known issue [#17239](https://github.com/OpenHands/OpenHands/issues/17239) (`F05.llm-profile-picker`): The reported model/profile picker can open above the viewport with unreachable rows. A frozen-target Local check with eight LLM profiles passed on Home at 1512x720, 1440x1000, 390x844 and 320x700, and in a genuine conversation at 1512x720 and 390x844: internal scrolling and selecting the first/last profile worked. The Cloud screenshot and real ACP multi-model picker remain unverified; they need an approved Cloud session or ACP account/adapter with a multi-row model catalogue. Local profile geometry is not proof of those branches. Run `backfill-18309-settings-worker`, Agent Server/SDK 1.54.0; evidence includes `F05.llm-profile-picker/17239-conversation-1512x720.png` and `17239-conversation-phone.png`." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--profile-identity", "title": "Profile identity", "ids": [ "F05.profile-identity" ], "body": "Arrange two profiles on one model right before this bullet: `QA_DUMMY_KEY=qa control-openhands llm set --profile qa-alpha --model openai/gpt-4o --api-key-env QA_DUMMY_KEY --no-validate --no-activate` and `QA_DUMMY_KEY=qa control-openhands llm set --profile qa-zeta --model openai/gpt-4o --api-key-env QA_DUMMY_KEY --no-validate` (`qa-zeta` becomes the default: `control-openhands api GET /api/profiles --pick active_profile` is `qa-zeta`). Run `control-openhands conversation start --prompt \"Reply with only: identity-ok\"` (no `--wait`; with the dummy key the agent errors at once, which does not matter here) and note its `id` as `<zeta-id>`. `control-openhands browser wait 'testid=chat-input-llm-profile'`, then `control-openhands browser text 'testid=chat-input-llm-profile'` is `qa-zeta`, not the alphabetically first profile on that model. `control-openhands browser reload`, the wait and the text again: still `qa-zeta`. With a DeepSeek key, arrange both profiles on `deepseek/deepseek-flash` with `--api-key-env DEEPSEEK_API_KEY` and without `--no-validate`, and start with `--prompt \"Reply with only: identity-ok. Do not run any tools.\" --wait --timeout 180`. Then the reply also survives the reload: `control-openhands browser text 'testid=agent-message >> nth=-1'` is `identity-ok` before and after it (driven 2026-10-08). The `default` agent profile still names `deepseek-flash` here, so the launch runs the pill's profile through agent settings (`F13.llm-pill-precedence`). The name is stamped on the conversation in this browser: `control-openhands browser eval \"JSON.parse(localStorage.getItem('openhands-agent-server-conversation-metadata'))['<zeta-id>'].active_profile\"` is `qa-zeta`. Change the default in Settings: `control-openhands browser goto /settings/llm`, `control-openhands browser wait 'testid=add-llm-profile'`, `control-openhands browser click '[data-testid=profile-row]:has([title=\"qa-alpha\"]) >> testid=profile-menu-trigger'`, `control-openhands browser click 'testid=profile-actions-menu >> testid=profile-set-active'`, `control-openhands browser wait-text 'Switched to profile \"qa-alpha\"'` (`control-openhands api GET /api/profiles --pick active_profile` is `qa-alpha`). The home pill follows the default: `control-openhands browser goto /`, the wait, and `control-openhands browser text 'testid=chat-input-llm-profile'` is `qa-alpha`; the started conversation keeps its own: `control-openhands browser goto /conversations/<zeta-id>`, the wait, and the text is `qa-zeta` (`control-openhands browser screenshot 'testid=interactive-chat-box' --feature F05.profile-identity --name conversation-keeps-qa-zeta`). Restore: make `deepseek-flash` the default again the same way (its row's `profile-menu-trigger`, then `profile-set-active`), then `control-openhands api DELETE /api/profiles/qa-alpha --write` and `control-openhands api DELETE /api/profiles/qa-zeta --write` (arrange, not proof) so `/model` below still counts two profiles. In a model-free run make `qa-zeta` the default instead and delete only `qa-alpha`. Return with `control-openhands browser goto /conversations/<id>`.", "raw": "**Profile identity (`F05.profile-identity`).** Arrange two profiles on one model right before this bullet: `QA_DUMMY_KEY=qa control-openhands llm set --profile qa-alpha --model openai/gpt-4o --api-key-env QA_DUMMY_KEY --no-validate --no-activate` and `QA_DUMMY_KEY=qa control-openhands llm set --profile qa-zeta --model openai/gpt-4o --api-key-env QA_DUMMY_KEY --no-validate` (`qa-zeta` becomes the default: `control-openhands api GET /api/profiles --pick active_profile` is `qa-zeta`). Run `control-openhands conversation start --prompt \"Reply with only: identity-ok\"` (no `--wait`; with the dummy key the agent errors at once, which does not matter here) and note its `id` as `<zeta-id>`. `control-openhands browser wait 'testid=chat-input-llm-profile'`, then `control-openhands browser text 'testid=chat-input-llm-profile'` is `qa-zeta`, not the alphabetically first profile on that model. `control-openhands browser reload`, the wait and the text again: still `qa-zeta`. With a DeepSeek key, arrange both profiles on `deepseek/deepseek-flash` with `--api-key-env DEEPSEEK_API_KEY` and without `--no-validate`, and start with `--prompt \"Reply with only: identity-ok. Do not run any tools.\" --wait --timeout 180`. Then the reply also survives the reload: `control-openhands browser text 'testid=agent-message >> nth=-1'` is `identity-ok` before and after it (driven 2026-10-08). The `default` agent profile still names `deepseek-flash` here, so the launch runs the pill's profile through agent settings (`F13.llm-pill-precedence`). The name is stamped on the conversation in this browser: `control-openhands browser eval \"JSON.parse(localStorage.getItem('openhands-agent-server-conversation-metadata'))['<zeta-id>'].active_profile\"` is `qa-zeta`. Change the default in Settings: `control-openhands browser goto /settings/llm`, `control-openhands browser wait 'testid=add-llm-profile'`, `control-openhands browser click '[data-testid=profile-row]:has([title=\"qa-alpha\"]) >> testid=profile-menu-trigger'`, `control-openhands browser click 'testid=profile-actions-menu >> testid=profile-set-active'`, `control-openhands browser wait-text 'Switched to profile \"qa-alpha\"'` (`control-openhands api GET /api/profiles --pick active_profile` is `qa-alpha`). The home pill follows the default: `control-openhands browser goto /`, the wait, and `control-openhands browser text 'testid=chat-input-llm-profile'` is `qa-alpha`; the started conversation keeps its own: `control-openhands browser goto /conversations/<zeta-id>`, the wait, and the text is `qa-zeta` (`control-openhands browser screenshot 'testid=interactive-chat-box' --feature F05.profile-identity --name conversation-keeps-qa-zeta`). Restore: make `deepseek-flash` the default again the same way (its row's `profile-menu-trigger`, then `profile-set-active`), then `control-openhands api DELETE /api/profiles/qa-alpha --write` and `control-openhands api DELETE /api/profiles/qa-zeta --write` (arrange, not proof) so `/model` below still counts two profiles. In a model-free run make `qa-zeta` the default instead and delete only `qa-alpha`. Return with `control-openhands browser goto /conversations/<id>`.", "source_line": 93, "expected": [ "the pill names the profile the conversation was started with, also after a reload and when another profile uses the same model; changing the default profile in Settings changes the home pill, not an existing conversation's." ], "notes": [ "Known issue [#16851](https://github.com/OpenHands/OpenHands/issues/16851) (`F05.profile-identity`): wrong model is displayed when loading existing conversation. Live prerequisite: The original conversation/model-switch history and usage record showing Opus with a Fable selector; a dummy-key conversation has no usage evidence." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--overflow-menu", "title": "Overflow menu", "ids": [ "F05.overflow-menu" ], "body": "Run `control-openhands browser viewport narrow` and `control-openhands browser snapshot 'testid=interactive-chat-box'`: the model pill is gone and `button \"More input actions\"` appears (at `phone` and `350x800` the pill stays inline). `control-openhands browser click 'role=button[name=\"More input actions\"]' --observe '[data-testid=chat-input-overflow-menu]' --observe-ms 1000`: `observed` goes from `<absent>` to `Model` (the menu appears in the observed sequence; its delay varies from run to run, so do not assert a time) and has no later `<absent>`. `control-openhands browser count 'testid=chat-input-overflow-menu'` is `1` and `control-openhands browser attr 'role=button[name=\"More input actions\"]' aria-expanded` is `true`. Run `control-openhands browser click 'testid=overflow-model-button'`; `control-openhands browser text 'testid=chat-input-overflow-menu'` reads `Model`, `AVAILABLE PROFILES`, `deepseek-flash` / `deepseek/deepseek-flash`, `deepseek-pro` / `deepseek/deepseek-v4-pro`, `LLM Profiles`. Run `control-openhands browser bbox 'testid=chat-input-llm-profile-option-deepseek-flash'` and `control-openhands browser bbox 'role=link[name=\"LLM Profiles\"]'`. Expected: both `insideViewport` `true`. Known failure (#18063): the submenu opens to the right of the menu, so the profile row starts at x about 270 with width 210 and the `LLM Profiles` link sits at y about 725, below the 700 px viewport (both `insideViewport` `false`); the profile clicks below still work because the first two rows keep a strip of about 50 px on screen (from x about 270 to the 320 px edge), while a click on the `LLM Profiles` link times out (`browser screenshot --feature F05.overflow-menu --name model-submenu-narrow`). `control-openhands browser click 'testid=chat-input-llm-profile-option-deepseek-pro'` and `control-openhands browser wait-text 'Switched to profile' --timeout 15000` close the menu (count `0`), and `control-openhands conversation status <id>` shows `deepseek/deepseek-v4-pro`. Switch back the same way: the trigger, `testid=overflow-model-button`, then `testid=chat-input-llm-profile-option-deepseek-flash` (`conversation status <id>` shows `deepseek/deepseek-flash`). Open the menu once more; a second click on the trigger closes it (`aria-expanded` `false`). Escape closes it too: record it with `control-openhands browser record start --feature F05.overflow-menu --name escape-narrow`, open the menu, `control-openhands browser press Escape` and `control-openhands browser wait 'testid=chat-input-overflow-menu' --state detached --timeout 3000`; `browser attr 'role=button[name=\"More input actions\"]' aria-expanded` is `false` and `control-openhands browser eval \"document.activeElement.getAttribute('aria-label')\"` is `More input actions`. Repeat with focus on a menu item (`control-openhands browser focus 'testid=overflow-model-button'` before Escape) and with the Model submenu open (`browser focus 'testid=chat-input-llm-profile-option-deepseek-flash'`): the same result. `control-openhands browser record stop --gif`. Return with `control-openhands browser viewport desktop`.", "raw": "**Overflow menu (`F05.overflow-menu`).** Run `control-openhands browser viewport narrow` and `control-openhands browser snapshot 'testid=interactive-chat-box'`: the model pill is gone and `button \"More input actions\"` appears (at `phone` and `350x800` the pill stays inline). `control-openhands browser click 'role=button[name=\"More input actions\"]' --observe '[data-testid=chat-input-overflow-menu]' --observe-ms 1000`: `observed` goes from `<absent>` to `Model` (the menu appears in the observed sequence; its delay varies from run to run, so do not assert a time) and has no later `<absent>`. `control-openhands browser count 'testid=chat-input-overflow-menu'` is `1` and `control-openhands browser attr 'role=button[name=\"More input actions\"]' aria-expanded` is `true`. Run `control-openhands browser click 'testid=overflow-model-button'`; `control-openhands browser text 'testid=chat-input-overflow-menu'` reads `Model`, `AVAILABLE PROFILES`, `deepseek-flash` / `deepseek/deepseek-flash`, `deepseek-pro` / `deepseek/deepseek-v4-pro`, `LLM Profiles`. Run `control-openhands browser bbox 'testid=chat-input-llm-profile-option-deepseek-flash'` and `control-openhands browser bbox 'role=link[name=\"LLM Profiles\"]'`. Expected: both `insideViewport` `true`. Known failure (#18063): the submenu opens to the right of the menu, so the profile row starts at x about 270 with width 210 and the `LLM Profiles` link sits at y about 725, below the 700 px viewport (both `insideViewport` `false`); the profile clicks below still work because the first two rows keep a strip of about 50 px on screen (from x about 270 to the 320 px edge), while a click on the `LLM Profiles` link times out (`browser screenshot --feature F05.overflow-menu --name model-submenu-narrow`). `control-openhands browser click 'testid=chat-input-llm-profile-option-deepseek-pro'` and `control-openhands browser wait-text 'Switched to profile' --timeout 15000` close the menu (count `0`), and `control-openhands conversation status <id>` shows `deepseek/deepseek-v4-pro`. Switch back the same way: the trigger, `testid=overflow-model-button`, then `testid=chat-input-llm-profile-option-deepseek-flash` (`conversation status <id>` shows `deepseek/deepseek-flash`). Open the menu once more; a second click on the trigger closes it (`aria-expanded` `false`). Escape closes it too: record it with `control-openhands browser record start --feature F05.overflow-menu --name escape-narrow`, open the menu, `control-openhands browser press Escape` and `control-openhands browser wait 'testid=chat-input-overflow-menu' --state detached --timeout 3000`; `browser attr 'role=button[name=\"More input actions\"]' aria-expanded` is `false` and `control-openhands browser eval \"document.activeElement.getAttribute('aria-label')\"` is `More input actions`. Repeat with focus on a menu item (`control-openhands browser focus 'testid=overflow-model-button'` before Escape) and with the Model submenu open (`browser focus 'testid=chat-input-llm-profile-option-deepseek-flash'`): the same result. `control-openhands browser record stop --gif`. Return with `control-openhands browser viewport desktop`.", "source_line": 94, "expected": [ "at very narrow widths the model pill collapses into a \"More input actions\" menu whose Model submenu fits the viewport and switches the profile; Escape closes the menu and returns focus to its trigger." ], "notes": [ "At `narrow` (320 px) the overflow menu's Model submenu opens to the right of the menu and runs off screen: each profile row starts at x about 270 with width 210 (`browser bbox 'testid=chat-input-llm-profile-option-deepseek-flash'`, `insideViewport` `false`), so both names read `deepse…`. Pick profiles by test id: the click lands on the strip of about 50 px that stays on screen. Known issue: #18063 (filed from the follow-up noted in #18015; the submenu also runs past the bottom edge, so the LLM Profiles link cannot be clicked at `narrow`) (`F05.overflow-menu`)." ], "annotations": [ "Known failure (#18063): the submenu opens to the right of the menu, so the profile row starts at x about 270 with width 210 and the `LLM Profiles` link sits at y about 725, below the 700 px viewport (both `insideViewport` `false`); the profile clicks below still work because the first two rows keep a strip of about 50 px on screen (from x about 270 to the 320 px edge), while a click on the `LLM Profiles` link times out (`browser screenshot --feature F05.overflow-menu --name model-submenu-narrow`). `control-openhands browser click 'testid=chat-input-llm-profile-option-deepseek-pro'` and `control-openhands browser wait-text 'Switched to profile' --timeout 15000` close the menu (count `0`), and `control-openhands conversation status <id>` shows `deepseek/deepseek-v4-pro`. Switch back the same way: the trigger, `testid=overflow-model-button`, then `testid=chat-input-llm-profile-option-deepseek-flash` (`conversation status <id>` shows `deepseek/deepseek-flash`). Open the menu once more; a second click on the trigger closes it (`aria-expanded` `false`). Escape closes it too: record it with `control-openhands browser record start --feature F05.overflow-menu --name escape-narrow`, open the menu, `control-openhands browser press Escape` and `control-openhands browser wait 'testid=chat-input-overflow-menu' --state detached --timeout 3000`; `browser attr 'role=button[name=\"More input actions\"]' aria-expanded` is `false` and `control-openhands browser eval \"document.activeElement.getAttribute('aria-label')\"` is `More input actions`. Repeat with focus on a menu item (`control-openhands browser focus 'testid=overflow-model-button'` before Escape) and with the Model submenu open (`browser focus 'testid=chat-input-llm-profile-option-deepseek-flash'`): the same result. `control-openhands browser record stop --gif`. Return with `control-openhands browser viewport desktop`." ], "states": [ "failure", "issue" ] }, { "name": "F05--context-window", "title": "Context window", "ids": [ "F05.context-window-meter" ], "body": "Run `control-openhands browser attr 'testid=context-window-meter' aria-label` (`Context window usage: 1% used (99% left)`), `control-openhands browser click 'testid=context-window-meter'` and `control-openhands browser text 'testid=context-window-meter-popover'` (`Context Window`, the percentage, `Compact context`, tokens such as `11.7k / 1.0M`, `Usage`). `control-openhands browser click 'testid=context-window-meter-bar-button'` (the progress bar) closes the popover and opens the Usage tab (`control-openhands browser testids --filter usage` lists `conversation-tab-usage`, `usage-panel`). The popover's `Usage` row does the same: from another tab (`control-openhands browser click 'testid=conversation-tab-files'`), reopen the ring, `control-openhands browser click 'testid=context-window-plan-usage'` and `control-openhands browser wait 'testid=usage-panel'` (the panel mounts a moment later; an immediate `count` is `0`). Reopen the ring, then `control-openhands browser click 'testid=context-window-compact-button' --observe '[role=status]' --observe-ms 25000`: `Context compaction started`, then `Context compacted`; `control-openhands conversation events <id> --kinds CondensationRequest,Condensation --last 5` lists both kinds.", "raw": "**Context window (`F05.context-window-meter`).** Run `control-openhands browser attr 'testid=context-window-meter' aria-label` (`Context window usage: 1% used (99% left)`), `control-openhands browser click 'testid=context-window-meter'` and `control-openhands browser text 'testid=context-window-meter-popover'` (`Context Window`, the percentage, `Compact context`, tokens such as `11.7k / 1.0M`, `Usage`). `control-openhands browser click 'testid=context-window-meter-bar-button'` (the progress bar) closes the popover and opens the Usage tab (`control-openhands browser testids --filter usage` lists `conversation-tab-usage`, `usage-panel`). The popover's `Usage` row does the same: from another tab (`control-openhands browser click 'testid=conversation-tab-files'`), reopen the ring, `control-openhands browser click 'testid=context-window-plan-usage'` and `control-openhands browser wait 'testid=usage-panel'` (the panel mounts a moment later; an immediate `count` is `0`). Reopen the ring, then `control-openhands browser click 'testid=context-window-compact-button' --observe '[role=status]' --observe-ms 25000`: `Context compaction started`, then `Context compacted`; `control-openhands conversation events <id> --kinds CondensationRequest,Condensation --last 5` lists both kinds.", "source_line": 95, "expected": [ "the ring shows context use; its popover opens the Usage tab and compacts the context." ], "notes": [ "Known issue [#17621](https://github.com/OpenHands/OpenHands/issues/17621) (`F05.context-window-meter`): Compact context always fails after exactly 60s — client-side timeout aborts a condensation the server then completes. Live prerequisite: A real summarizing model and conversation whose condensation takes longer than 60 seconds.", "Known issue [#17476](https://github.com/OpenHands/OpenHands/issues/17476) (`F05.context-window-meter`): Planning socket stats overwrite the main conversation's context meter. Live prerequisite: A real planning sub-conversation and main conversation with distinct nonzero usage stats.", "Known issue [#17271](https://github.com/OpenHands/OpenHands/issues/17271) (`F05.context-window-meter`): New chat retains context/state from the previous conversation. Live prerequisite: A completed conversation with nonzero token usage before New Chat; a dummy-key run supplies no genuine context usage.", "Known issue [#17263](https://github.com/OpenHands/OpenHands/issues/17263) (`F05.context-window-meter`): Context meter stays pinned after a successful compaction once the condenser has its own usage entry. Live prerequisite: A real successful compaction whose condenser usage remains larger than the default agent usage.", "Known issue [#16835](https://github.com/OpenHands/OpenHands/issues/16835) (`F05.context-window-meter`): Agent Canvas sidebar shows \"Cache Hit: 0\" while backend correctly records nonzero cache_read_tokens. Live prerequisite: A working prompt-caching model/proxy and a real conversation reporting nonzero cache_read_tokens.", "Known issue [OpenHands/software-agent-sdk#4498](https://github.com/OpenHands/software-agent-sdk/issues/4498) (`F05.context-window-meter`): Compaction can leave an orphaned tool call/result in the live history and make the next provider request fail. Live prerequisite: A real condensation splitting an action/result pair, followed by the next provider request.", "Known issue [OpenHands/software-agent-sdk#4382](https://github.com/OpenHands/software-agent-sdk/issues/4382) (`F05.context-window-meter`): ACP cost can miss cache/thought usage, silently become zero, or inflate after stalled/reset cost updates. Live prerequisite: An authenticated ACP provider emitting cache/thought token usage or reset/stalled cumulative cost updates.", "Known issue [OpenHands/software-agent-sdk#5820](https://github.com/OpenHands/software-agent-sdk/issues/5820) (`F05.context-window-meter`): manual compaction on a null/no-op condenser can return500. Verification is blocked by genuine populated usage/context metrics: idle and refused-loopback runs leave `usage=null`, so neither composer nor Usage panel offers Compact context. A separate direct endpoint check returned500 but is not a Canvas reproduction. Use an existing real conversation with usage and no condenser, or an authorized successful provider run; never fabricate stats to expose the action." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--stop-and-resume-with-a-queued-follow-up", "title": "Stop and resume with a queued follow-up", "ids": [ "F05.stop-resume", "F05.queued-message" ], "body": "Send `Run this shell command: sleep 25 && echo slept. Then reply with only: done` (type + Enter as above), `control-openhands conversation wait <id> --until running --timeout 30` and `control-openhands browser wait 'testid=stop-button'`; `control-openhands browser attr 'testid=stop-button' aria-label` is `Stop`. While it runs, `control-openhands browser enabled 'testid=submit-button'` is `false` with an empty field; type `Also reply with: queued-ok` (enabled turns `true`) and press Enter. Run `control-openhands browser click 'testid=stop-button'` and `control-openhands browser wait 'testid=play-button' --timeout 30000`; `control-openhands browser text 'testid=interactive-chat-box'` ends `Stopped`, `control-openhands browser attr 'testid=play-button' aria-label` is `Resume the agent task` and `control-openhands conversation status <id>` is `paused`. Run `control-openhands browser click 'testid=play-button'`, wait for `running` then the terminal state: the last agent message is `done queued-ok` and `testid=user-message >> has-text=queued-ok` counts `1`.", "raw": "**Stop and resume (`F05.stop-resume`) with a queued follow-up (`F05.queued-message`).** Send `Run this shell command: sleep 25 && echo slept. Then reply with only: done` (type + Enter as above), `control-openhands conversation wait <id> --until running --timeout 30` and `control-openhands browser wait 'testid=stop-button'`; `control-openhands browser attr 'testid=stop-button' aria-label` is `Stop`. While it runs, `control-openhands browser enabled 'testid=submit-button'` is `false` with an empty field; type `Also reply with: queued-ok` (enabled turns `true`) and press Enter. Run `control-openhands browser click 'testid=stop-button'` and `control-openhands browser wait 'testid=play-button' --timeout 30000`; `control-openhands browser text 'testid=interactive-chat-box'` ends `Stopped`, `control-openhands browser attr 'testid=play-button' aria-label` is `Resume the agent task` and `control-openhands conversation status <id>` is `paused`. Run `control-openhands browser click 'testid=play-button'`, wait for `running` then the terminal state: the last agent message is `done queued-ok` and `testid=user-message >> has-text=queued-ok` counts `1`.", "source_line": 96, "expected": [ "Stop pauses a running agent (status `Stopped`, Play button); Play resumes it.", "a follow-up typed while the agent runs is accepted and answered." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--slash-menu", "title": "Slash menu", "ids": [ "F05.slash-menu" ], "body": "On a cleared composer run `control-openhands browser type 'testid=chat-input' '/'`, `control-openhands browser count 'testid=slash-command-menu >> role=option'` (37 with the bundled skills) and `control-openhands browser text 'testid=slash-command-menu'`: `Commands`, then `/btw`, `/model`, `/goal`, `/plan`, `/code` with descriptions, then skills (`/agent-creator`, …); no `/new` on the local backend. Type `mo`: the count drops (descriptions match too). `control-openhands browser attr 'testid=slash-command-menu >> role=option >> nth=0' aria-selected` is `true`; after `control-openhands browser press ArrowDown` the selected option is `/plan` (read it with `control-openhands browser text 'testid=slash-command-menu >> [aria-selected=true]'`). `control-openhands browser press Escape` closes the menu and keeps `/mo`. On `/btw`, `control-openhands browser press Tab` leaves `/btw ` in the field; on `/model ` (trailing space) the menu lists `/model deepseek-flash` and `/model deepseek-pro`, and `control-openhands browser press Enter` inserts `/model deepseek-flash ` instead of sending.", "raw": "**Slash menu (`F05.slash-menu`).** On a cleared composer run `control-openhands browser type 'testid=chat-input' '/'`, `control-openhands browser count 'testid=slash-command-menu >> role=option'` (37 with the bundled skills) and `control-openhands browser text 'testid=slash-command-menu'`: `Commands`, then `/btw`, `/model`, `/goal`, `/plan`, `/code` with descriptions, then skills (`/agent-creator`, …); no `/new` on the local backend. Type `mo`: the count drops (descriptions match too). `control-openhands browser attr 'testid=slash-command-menu >> role=option >> nth=0' aria-selected` is `true`; after `control-openhands browser press ArrowDown` the selected option is `/plan` (read it with `control-openhands browser text 'testid=slash-command-menu >> [aria-selected=true]'`). `control-openhands browser press Escape` closes the menu and keeps `/mo`. On `/btw`, `control-openhands browser press Tab` leaves `/btw ` in the field; on `/model ` (trailing space) the menu lists `/model deepseek-flash` and `/model deepseek-pro`, and `control-openhands browser press Enter` inserts `/model deepseek-flash ` instead of sending.", "source_line": 97, "expected": [ "`/` opens a filtered command list (built-ins and skills) with arrow, Tab/Enter and Escape handling." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--model", "title": "`/model`", "ids": [ "F05.slash-model" ], "body": "Type `/model deepseek-pro`, `control-openhands browser press Escape` (close the menu), `control-openhands browser press Enter --selector 'testid=chat-input'`: `Switched to profile` appears, the pill reads `deepseek-pro` and `conversation status <id>` shows `deepseek/deepseek-v4-pro`. Bare `/model` (Escape, Enter) adds `Available profiles (2)` to `testid=model-messages`; `control-openhands browser click 'testid=model-messages >> role=button[name=\"Expand\"]'` and `control-openhands browser click 'role=button[name=\"Toggle details for deepseek-pro\"]'` show `model: deepseek/deepseek-v4-pro`, `base_url: —`, `api_key: set`. `/model qa-nope` (Escape, Enter) toasts `Profile 'qa-nope' not found` (`control-openhands browser toasts`) and the pill keeps `deepseek-pro`. Restore with `/model deepseek-flash`.", "raw": "**`/model` (`F05.slash-model`).** Type `/model deepseek-pro`, `control-openhands browser press Escape` (close the menu), `control-openhands browser press Enter --selector 'testid=chat-input'`: `Switched to profile` appears, the pill reads `deepseek-pro` and `conversation status <id>` shows `deepseek/deepseek-v4-pro`. Bare `/model` (Escape, Enter) adds `Available profiles (2)` to `testid=model-messages`; `control-openhands browser click 'testid=model-messages >> role=button[name=\"Expand\"]'` and `control-openhands browser click 'role=button[name=\"Toggle details for deepseek-pro\"]'` show `model: deepseek/deepseek-v4-pro`, `base_url: —`, `api_key: set`. `/model qa-nope` (Escape, Enter) toasts `Profile 'qa-nope' not found` (`control-openhands browser toasts`) and the pill keeps `deepseek-pro`. Restore with `/model deepseek-flash`.", "source_line": 98, "expected": [ "`/model` lists saved profiles; `/model <name>` switches profile." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--btw", "title": "`/btw`", "ids": [ "F05.slash-btw" ], "body": "Note `control-openhands conversation events <id> --kinds MessageEvent` count, type `/btw Reply with only: side-ok`, Escape, Enter. `control-openhands browser text 'testid=btw-messages'` is `BTW:` + the question and `testid=btw-spinner` counts `1`; after the answer the card shows `Got it`. `control-openhands browser click 'testid=btw-messages >> role=button[name=\"Expand\"]'` reveals `side-ok`; `control-openhands browser click 'testid=btw-messages >> role=button[name=\"Got it\"]'` removes the card (count `0`). The MessageEvent count is unchanged. Bare `/btw` (Escape, Enter) clears the field and `control-openhands browser toasts` shows `Please provide a question — e.g. /btw <question>`; `control-openhands browser count 'testid=btw-messages'` stays `0`.", "raw": "**`/btw` (`F05.slash-btw`).** Note `control-openhands conversation events <id> --kinds MessageEvent` count, type `/btw Reply with only: side-ok`, Escape, Enter. `control-openhands browser text 'testid=btw-messages'` is `BTW:` + the question and `testid=btw-spinner` counts `1`; after the answer the card shows `Got it`. `control-openhands browser click 'testid=btw-messages >> role=button[name=\"Expand\"]'` reveals `side-ok`; `control-openhands browser click 'testid=btw-messages >> role=button[name=\"Got it\"]'` removes the card (count `0`). The MessageEvent count is unchanged. Bare `/btw` (Escape, Enter) clears the field and `control-openhands browser toasts` shows `Please provide a question — e.g. /btw <question>`; `control-openhands browser count 'testid=btw-messages'` stays `0`.", "source_line": 99, "expected": [ "`/btw <question>` answers a side question in a dismissible card without adding to the main stream; a bare `/btw` toasts that a question is needed." ], "notes": [ "Known issue [#17429](https://github.com/OpenHands/OpenHands/issues/17429) (`F05.slash-btw`): 500 error with /btw. Live prerequisite: The exact pending tool/streaming state that triggers the generator-pickle error, plus a working model for the side question.", "Known issue [OpenHands/software-agent-sdk#5330](https://github.com/OpenHands/software-agent-sdk/issues/5330) (`F05.slash-btw`): A /btw side question can prevent reuse of the normal conversation prompt cache. Live prerequisite: A working caching provider and comparable ordinary-message versus /btw prompt/cache records." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--goal", "title": "`/goal`", "ids": [ "F05.slash-goal" ], "body": "Bare `/goal` + Enter: `control-openhands browser toasts` shows `Please provide an objective — e.g. /goal <objective>`. Send `/goal --max 1 Reply with only: goal-ok`: `control-openhands browser text 'testid=goal-status'` reads `Goal: … Round 0/1 running Stop`; `control-openhands browser wait 'testid=goal-done' --timeout 150000`, then `Round 1/1 complete score 100%`. Send `/goal --max 2 Reply with only: goal-two` and at once `control-openhands browser click 'testid=goal-status >> nth=-1 >> testid=goal-stop'`; `control-openhands browser wait 'testid=goal-resume'` and the banner reads `Round 0/2 interrupted Resume`. `control-openhands browser click 'testid=goal-resume'` sets it `running`; poll `control-openhands browser text 'testid=goal-status >> nth=-1'` until a new row ends `Round 1/2 complete score 100%`. The old `Round 0/2 interrupted` row no longer offers Resume: `control-openhands browser count 'testid=goal-resume'` is `0`. After `control-openhands browser reload`, `control-openhands browser count 'testid=goal-status'` is `3` and the `goal-resume` count is still `0`.", "raw": "**`/goal` (`F05.slash-goal`).** Bare `/goal` + Enter: `control-openhands browser toasts` shows `Please provide an objective — e.g. /goal <objective>`. Send `/goal --max 1 Reply with only: goal-ok`: `control-openhands browser text 'testid=goal-status'` reads `Goal: … Round 0/1 running Stop`; `control-openhands browser wait 'testid=goal-done' --timeout 150000`, then `Round 1/1 complete score 100%`. Send `/goal --max 2 Reply with only: goal-two` and at once `control-openhands browser click 'testid=goal-status >> nth=-1 >> testid=goal-stop'`; `control-openhands browser wait 'testid=goal-resume'` and the banner reads `Round 0/2 interrupted Resume`. `control-openhands browser click 'testid=goal-resume'` sets it `running`; poll `control-openhands browser text 'testid=goal-status >> nth=-1'` until a new row ends `Round 1/2 complete score 100%`. The old `Round 0/2 interrupted` row no longer offers Resume: `control-openhands browser count 'testid=goal-resume'` is `0`. After `control-openhands browser reload`, `control-openhands browser count 'testid=goal-status'` is `3` and the `goal-resume` count is still `0`.", "source_line": 100, "expected": [ "`/goal [--max N] <objective>` runs a judged goal loop with a status banner, Stop and Resume; only the newest goal status offers Resume." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5627](https://github.com/OpenHands/software-agent-sdk/issues/5627) (`F05.slash-goal`): After backend restart the goal chip can remain Running while the conversation is paused, and Stop does not clear it. Live prerequisite: A working model goal loop active during graceful backend restart, then goal-status and resume readback.", "Known issue [OpenHands/software-agent-sdk#5492](https://github.com/OpenHands/software-agent-sdk/issues/5492) (`F05.slash-goal`): A goal can stop as Complete when its judge replies with complete as the string false. Live prerequisite: A real goal judge response containing complete as the string false rather than a JSON boolean.", "Known issue [OpenHands/software-agent-sdk#5824](https://github.com/OpenHands/software-agent-sdk/issues/5824) (`F05.slash-goal`): resuming after a failed judge audit can show iteration 2 of a maximum of 1 and run beyond the selected goal cap. Live prerequisite: an agent/judge run whose audit fails, followed by a successful incomplete judge verdict after Resume; a controller-only test does not establish the Canvas banner outcome." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F05--plan-and-code", "title": "`/plan` and `/code`", "ids": [ "F05.slash-plan-code" ], "body": "Send bare `/plan` (Escape, Enter): `control-openhands browser attr 'testid=chat-input' data-placeholder` turns `Let’s work on a plan`; `control-openhands conversation list` gains a planner conversation (title `null` at first, `<planner-id>`). Bare `/code` turns it back to `What do you want to build?`. `control-openhands browser count 'role=button[name=\"Code\"]'` is `0` on local (the pill is Cloud only). `/plan` sent while the agent runs should be refused with feedback; today it is dropped silently: the field clears, no toast, the placeholder stays `What do you want to build?` (fail row; `use-plan-mode-interceptor.ts` swallows it by design but tells the user nothing).", "raw": "**`/plan` and `/code` (`F05.slash-plan-code`).** Send bare `/plan` (Escape, Enter): `control-openhands browser attr 'testid=chat-input' data-placeholder` turns `Let’s work on a plan`; `control-openhands conversation list` gains a planner conversation (title `null` at first, `<planner-id>`). Bare `/code` turns it back to `What do you want to build?`. `control-openhands browser count 'role=button[name=\"Code\"]'` is `0` on local (the pill is Cloud only). `/plan` sent while the agent runs should be refused with feedback; today it is dropped silently: the field clears, no toast, the placeholder stays `What do you want to build?` (fail row; `use-plan-mode-interceptor.ts` swallows it by design but tells the user nothing).", "source_line": 101, "expected": [ "`/plan` switches to plan mode (a planner sub-conversation), `/code` switches back." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--plan-card", "title": "Plan card", "ids": [ "F05.plan-preview" ], "body": "In plan mode send `Plan creating a file hello.txt containing hi. Keep the plan to three short steps.`, then `control-openhands conversation wait <planner-id> --until running --timeout 60`, `control-openhands conversation wait <planner-id> --timeout 240` and `control-openhands browser wait 'testid=plan-preview-content' --timeout 60000`. `control-openhands browser text 'testid=plan-preview-content'` starts `1. OBJECTIVE` and ends `Read more`; `control-openhands browser enabled 'testid=plan-preview-build-button'` is `true`. Use `--fresh` instead of the `--until running` wait if the planner is fast. `control-openhands browser click 'testid=plan-preview-view-button'` (Read more does the same) opens the Planner tab: `control-openhands browser testids --filter planner` lists `conversation-tab-planner` and `planner-tab-build-button` (`Build ⌘↩`).", "raw": "**Plan card (`F05.plan-preview`).** In plan mode send `Plan creating a file hello.txt containing hi. Keep the plan to three short steps.`, then `control-openhands conversation wait <planner-id> --until running --timeout 60`, `control-openhands conversation wait <planner-id> --timeout 240` and `control-openhands browser wait 'testid=plan-preview-content' --timeout 60000`. `control-openhands browser text 'testid=plan-preview-content'` starts `1. OBJECTIVE` and ends `Read more`; `control-openhands browser enabled 'testid=plan-preview-build-button'` is `true`. Use `--fresh` instead of the `--until running` wait if the planner is fast. `control-openhands browser click 'testid=plan-preview-view-button'` (Read more does the same) opens the Planner tab: `control-openhands browser testids --filter planner` lists `conversation-tab-planner` and `planner-tab-build-button` (`Build ⌘↩`).", "source_line": 102, "expected": [ "the planner's Plan.md card shows the plan with Read more, View (Planner tab) and Build." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--build-shortcut", "title": "Build shortcut", "ids": [ "F05.build-plan-shortcut" ], "body": "With the plan card shown and the agent idle, `control-openhands browser press Control+Enter`: the placeholder returns to `What do you want to build?`, `control-openhands conversation wait <id> --until running --timeout 60` then `--timeout 240` finishes, `control-openhands browser text 'testid=user-message >> nth=-1'` is `Execute the plan based on the .agents_tmp/PLAN.md file.` (the agent's reply wording varies, e.g. `Executed the plan in .agents_tmp/PLAN.md`), and `hello.txt` exists in the `workspace` from `control-openhands conversation status <id>`.", "raw": "**Build shortcut (`F05.build-plan-shortcut`).** With the plan card shown and the agent idle, `control-openhands browser press Control+Enter`: the placeholder returns to `What do you want to build?`, `control-openhands conversation wait <id> --until running --timeout 60` then `--timeout 240` finishes, `control-openhands browser text 'testid=user-message >> nth=-1'` is `Execute the plan based on the .agents_tmp/PLAN.md file.` (the agent's reply wording varies, e.g. `Executed the plan in .agents_tmp/PLAN.md`), and `hello.txt` exists in the `workspace` from `control-openhands conversation status <id>`.", "source_line": 103, "expected": [ "Ctrl/Cmd+Enter in plan mode builds the plan with the code agent." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--plan-task-and-code-task", "title": "`/plan <task>` and `/code <task>`", "ids": [ "F05.slash-plan-code-task" ], "body": "With the agent idle, send bare `/plan`, then `/code Reply with only: code-ok` (Escape, Enter): the placeholder turns `What do you want to build?` at once, `control-openhands conversation wait <id> --fresh --timeout 180`, `control-openhands browser text 'testid=user-message >> nth=-1'` is `Reply with only: code-ok` (no `/code` prefix) and the last agent message is `code-ok`. Then send `/plan Plan writing bye.txt containing bye. One step only.`: the placeholder turns `Let’s work on a plan`, `control-openhands conversation wait <planner-id> --fresh --timeout 240`, `control-openhands conversation events <planner-id> --kinds MessageEvent --last 100` ends with the user text `Plan writing bye.txt containing bye. One step only.` and a planner reply, and `control-openhands conversation list` still has only the two conversations (the existing planner is reused).", "raw": "**`/plan <task>` and `/code <task>` (`F05.slash-plan-code-task`).** With the agent idle, send bare `/plan`, then `/code Reply with only: code-ok` (Escape, Enter): the placeholder turns `What do you want to build?` at once, `control-openhands conversation wait <id> --fresh --timeout 180`, `control-openhands browser text 'testid=user-message >> nth=-1'` is `Reply with only: code-ok` (no `/code` prefix) and the last agent message is `code-ok`. Then send `/plan Plan writing bye.txt containing bye. One step only.`: the placeholder turns `Let’s work on a plan`, `control-openhands conversation wait <planner-id> --fresh --timeout 240`, `control-openhands conversation events <planner-id> --kinds MessageEvent --last 100` ends with the user text `Plan writing bye.txt containing bye. One step only.` and a planner reply, and `control-openhands conversation list` still has only the two conversations (the existing planner is reused).", "source_line": 104, "expected": [ "`/code <task>` switches to code mode and sends `<task>` to the code agent at once; `/plan <task>` switches to plan mode and sends `<task>` to the existing planner." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--build-button", "title": "Build button", "ids": [ "F05.plan-preview" ], "body": "With the newer plan card from `/plan Plan writing bye.txt containing bye. One step only.` (above) shown and the agent idle, `control-openhands browser click 'testid=plan-preview-build-button >> nth=-1'` does the same as the shortcut: placeholder `What do you want to build?`, `control-openhands conversation wait <id> --fresh --timeout 240`, the same last user message, and `bye.txt` (content `bye`) in the workspace.", "raw": "**Build button (`F05.plan-preview`).** With the newer plan card from `/plan Plan writing bye.txt containing bye. One step only.` (above) shown and the agent idle, `control-openhands browser click 'testid=plan-preview-build-button >> nth=-1'` does the same as the shortcut: placeholder `What do you want to build?`, `control-openhands conversation wait <id> --fresh --timeout 240`, the same last user message, and `bye.txt` (content `bye`) in the workspace.", "source_line": 105, "expected": [ "the planner's Plan.md card shows the plan with Read more, View (Planner tab) and Build." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--phone", "title": "Phone", "ids": [ "F05.phone" ], "body": "On `/conversations/<id>` run `control-openhands browser viewport phone`, `control-openhands browser bbox 'testid=interactive-chat-box'` (`insideViewport` `true`, `pageHorizontalOverflow` `false`) and `control-openhands browser screenshot --feature F05.phone --name composer` (`+`, model pill, ring, mic and Send on one row). Return with `control-openhands browser viewport desktop`.", "raw": "**Phone (`F05.phone`).** On `/conversations/<id>` run `control-openhands browser viewport phone`, `control-openhands browser bbox 'testid=interactive-chat-box'` (`insideViewport` `true`, `pageHorizontalOverflow` `false`) and `control-openhands browser screenshot --feature F05.phone --name composer` (`+`, model pill, ring, mic and Send on one row). Return with `control-openhands browser viewport desktop`.", "source_line": 106, "expected": [ "the composer fits a 390 px viewport with every control inline." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F05--home-file-re-pick", "title": "Home file re-pick", "ids": [ "F05.attach-files" ], "body": "As a separate check after the conversation recipes, run `control-openhands browser goto /`, upload the file fixture with `control-openhands browser upload 'testid=upload-image-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"`, remove it with `control-openhands browser click 'role=button[name=\"Remove qa-note.txt\"]'`, upload that same path again, then `control-openhands browser count 'role=button[name=\"Remove qa-note.txt\"]'`. Expected: `1`. Known failure (reproduced 2026-10-12, [#17471](https://github.com/OpenHands/OpenHands/issues/17471)): the second pick is ignored and the count is `0`. End with `browser reload` to clear the file input.", "raw": "**Home file re-pick (`F05.attach-files`).** As a separate check after the conversation recipes, run `control-openhands browser goto /`, upload the file fixture with `control-openhands browser upload 'testid=upload-image-input' \"$OH_VERIFY_RUN/evidence/_fixtures/qa-note.txt\"`, remove it with `control-openhands browser click 'role=button[name=\"Remove qa-note.txt\"]'`, upload that same path again, then `control-openhands browser count 'role=button[name=\"Remove qa-note.txt\"]'`. Expected: `1`. Known failure (reproduced 2026-10-12, [#17471](https://github.com/OpenHands/OpenHands/issues/17471)): the second pick is ignored and the count is `0`. End with `browser reload` to clear the file input.", "source_line": 108, "expected": [ "picked files show as chips and images as thumbnails (with an Upload-as-file toggle), each removable; sent files land in the workspace and are named in the message." ], "notes": [ "Known issue [#17788](https://github.com/OpenHands/OpenHands/issues/17788) (`F05.attach-files`): Cloud attachment upload rejects valid runtime data when cached fields are empty strings. Live prerequisite: A Cloud sandbox whose cached runtime URL/key are blank while the fresh runtime lookup is already ready.", "Known issue [#16430](https://github.com/OpenHands/OpenHands/issues/16430) (`F05.attach-files`): File upload does not have an indication of progress. Live prerequisite: A remote or throttled real upload that stays in flight long enough to inspect progress and composer state.", "Known issue [#16339](https://github.com/OpenHands/OpenHands/issues/16339) (`F05.attach-files`): Agent not informed of file location for files uploaded via canvas in cloud. Live prerequisite: An authenticated Cloud upload sandbox and working model to inspect the delivered filename/path notification and read the uploaded file.", "Known issue [OpenHands/software-agent-sdk#5889](https://github.com/OpenHands/software-agent-sdk/issues/5889) (`F05.attach-files`): a same-name attachment upload that fails part-way can replace the previous workspace file with partial content. Live prerequisite: an owned isolated backend with a controlled mid-upload filesystem failure (the original Linux file-size limit or an equivalent bounded child-process limit), a pre-existing target and a real Canvas overwrite. Never change resource limits on a shared runtime; a normal successful upload does not verify failure atomicity." ], "annotations": [ "Known failure (reproduced 2026-10-12, [#17471](https://github.com/OpenHands/OpenHands/issues/17471)): the second pick is ignored and the count is `0`. End with `browser reload` to clear the file input." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F05--unmatched-failed-conversation-profile", "title": "Unmatched failed-conversation profile", "ids": [ "F05.profile-identity" ], "body": "Separate disposable check: record the original active profile with `control-openhands llm show`, then create an empty Local conversation with a new dummy profile pointing to closed loopback `http://127.0.0.1:9` (`llm set --no-validate`, as F06's dummy-profile precondition does); give it model `openai/gpt-4o`. After `browser reset` and `onboard --skip`, reopen it and send `Reply with only QA. Do not run any tools.`. Change the same saved profile to `openai/gpt-4o-mini` with `llm set --no-validate`, then reload and read `browser text 'testid=chat-input-llm-profile'` and `conversation status <id>`. Known failure (reproduced 2026-10-12, [#16263](https://github.com/OpenHands/OpenHands/issues/16263)): once the loopback turn errors, the pill reads Select a model despite a loaded active profile and the persisted conversation model. This check covers the missing browser stamp and unmatched-model case, not successful provider output. Reactivate the original profile with `control-openhands api POST /api/profiles/<original-profile>/activate --write`, then delete the disposable conversation and profile afterwards.", "raw": "**Unmatched failed-conversation profile (`F05.profile-identity`).** Separate disposable check: record the original active profile with `control-openhands llm show`, then create an empty Local conversation with a new dummy profile pointing to closed loopback `http://127.0.0.1:9` (`llm set --no-validate`, as F06's dummy-profile precondition does); give it model `openai/gpt-4o`. After `browser reset` and `onboard --skip`, reopen it and send `Reply with only QA. Do not run any tools.`. Change the same saved profile to `openai/gpt-4o-mini` with `llm set --no-validate`, then reload and read `browser text 'testid=chat-input-llm-profile'` and `conversation status <id>`. Known failure (reproduced 2026-10-12, [#16263](https://github.com/OpenHands/OpenHands/issues/16263)): once the loopback turn errors, the pill reads Select a model despite a loaded active profile and the persisted conversation model. This check covers the missing browser stamp and unmatched-model case, not successful provider output. Reactivate the original profile with `control-openhands api POST /api/profiles/<original-profile>/activate --write`, then delete the disposable conversation and profile afterwards.", "source_line": 109, "expected": [ "the pill names the profile the conversation was started with, also after a reload and when another profile uses the same model; changing the default profile in Settings changes the home pill, not an existing conversation's." ], "notes": [ "Known issue [#16851](https://github.com/OpenHands/OpenHands/issues/16851) (`F05.profile-identity`): wrong model is displayed when loading existing conversation. Live prerequisite: The original conversation/model-switch history and usage record showing Opus with a Fable selector; a dummy-key conversation has no usage evidence." ], "annotations": [ "Known failure (reproduced 2026-10-12, [#16263](https://github.com/OpenHands/OpenHands/issues/16263)): once the loopback turn errors, the pill reads Select a model despite a loaded active profile and the persisted conversation model. This check covers the missing browser stamp and unmatched-model case, not successful provider output. Reactivate the original profile with `control-openhands api POST /api/profiles/<original-profile>/activate --write`, then delete the disposable conversation and profile afterwards." ], "states": [ "failure", "issue", "blocked" ] } ] }, { "id": "F06", "title": "Agent activity: messages, events and lifecycle", "file": "F06-agent-activity.md", "sha256": "4896fe659802fd4efea2135c5eefeab2e0fe0a69030249a46cb898bf94b99aae", "overview": "Inside a conversation the user watches the agent work: their own messages appear\nat once (optimistically), the agent's replies render as markdown, tool calls are\ngrouped into collapsible \"N actions completed\" blocks whose cards show commands,\noutputs, diffs and task lists, and a live chip says what the agent is doing right\nnow. Around the transcript sit lifecycle controls and states: the starting pill,\nstop and resume, the confirmation prompt when confirmation mode is on, error and\nreconnect banners, the no-LLM banner and the skill-installed banner. History\npages in from the server, survives reloads and backend restarts, and any message\ncan be branched into a new conversation.", "entry": "- Open any conversation: a sidebar conversation card, or the direct URL `/conversations/<id>`. Leaving one for the home page: the sidebar's **New Chat** link; coming back: its card again.\n- Start one from the home composer (**New Chat**, `/`): type and press Send. `control-openhands conversation start` drives exactly this path.\n- An empty conversation: the sidebar's create-thread button (`conversation-panel-new-thread-picker`), then **No workspace** (or a workspace).\n- Composer status control (right side of the composer): Stop while running, Play while stopped.\n- Hover a bubble: Branch from here and Copy to clipboard icons; hover a tool title for its timestamp.\n- Confirmation mode and the critic are switched on under Settings → Verification (`/settings/verification`; the page itself belongs to the settings families).", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done) and `control-openhands llm preset deepseek` (deepseek-flash active). Every prompt below is tiny and stays in the conversation's own workspace.\n- Conversation ids are printed as `\"id\"` by `control-openhands conversation start`, or read from `control-openhands browser url` (`/conversations/<id>`). `<id>` below is always the conversation the step just created or opened.\n- `F06.llm-not-configured-banner` needs a second, fresh `control-openhands launch --new --build never` with no `llm preset` (export its run dir as `OH_VERIFY_RUN` for those commands, then `control-openhands stop` it).\n- `F06.critic-result` is blocked: it needs a model and an OpenHands Cloud critic API key (Settings → Verification → Critic API Key); its bullet says what a scored run shows. The automatic-fetch half of `F06.load-older-history` (its 16-command conversation) needs a model; the scroll-to-top half is driven without one, on the dummy profile below, with thirty refused user messages. `F06.hook-events` needs a configured hook; none is set in a fresh run.\n- `F06.clock-skew-send` and `F06.failed-send-persists` need no model reply: a conversation that exists is enough (without a key the agent errors at once, and the user rows still land). Without a key, create the family's conversations with a dummy profile (`QA_DUMMY_KEY=qa control-openhands llm set --profile qa-zeta --model openai/gpt-4o --api-key-env QA_DUMMY_KEY --no-validate`) and `control-openhands conversation start --prompt ...` without `--wait`. `F06.corrective-nudge` is blocked: the nudge needs a model turn with neither text nor a tool call, which a normal prompt to deepseek-flash does not produce. When one happens, `control-openhands browser count 'testid=corrective-nudge-message'` is `1` and `browser count 'testid=user-message >> has-text=Your last response did not include'` is `0`.\n- The harness browser grants clipboard access: `control-openhands browser clipboard` reads what a Copy button wrote and `browser clipboard --write qa-empty` resets it first.\n- A terminal call can come back as \"The terminal session was reset...\" after this run's agent server restarts; the model reruns it, which turns one action into a group of two (see Gotchas).", "gotchas": "- `control-openhands conversation events <id>` returns the last 25 events unless you pass `--last N`; it reads the server's 100-event pages until `N` rows are in hand (`pages` says how many it read, `more: true` that older pages remain), so `--last 500` covers a 150-event conversation in two pages (`total` `152`, `pages` `2`, `more` `false` on the Older history conversation; `--last 25` there reads one page, `more` `true`) and `--from-start` reads from the oldest. `--kinds` and `--grep` filter before `--last` counts, and pages are read until `--last` matching rows are in hand: `--kinds MessageEvent --last 3` is the three newest messages, `--kinds MessageEvent --last 500` all thirty, and `--grep 'qa older 1'` reaches `qa older 1` on the oldest page with the default `--last` (`count` `11`, `pages` `2`). Event texts are cut at about 160 characters.\n- `--observe` takes the same selector syntax as the verbs: `testid=` and plain CSS segments (joined with ` >> `) are watched by an in-page MutationObserver (`observedBy` `mutation`; the Older history bullet's `--observe 'testid=loading-older-events'` is one), while other engines (`role=`, `text=`, `nth=`) are polled every 20 ms instead (not driven in this family).\n- `role=button[name=\"Expand\"]` also matches \"Expand thinking\" and \"Expand actions\"; use `[exact]` to target tool rows. Each click flips the row to Collapse, so `nth=0` walks to the next collapsed row.\n- Tool rows show no success icon; the only indicators are the `exit N` badge for non-zero exits and a clock icon (`status-icon`) for timeouts. A single action is not grouped: it renders as a bare row without `event-group`.\n- The model writes the tool titles and may append to commands (it turned `ls /missing` into `ls ...; echo \"EXIT:$?\"`, exit 0). Ask for the command \"verbatim, with nothing appended\" when the exit code matters.\n- The agent itself may open files with `canvas_ui_control` (a third action in the echo/file run). Close the file tab before proving that a path link opens it.\n- On Settings → Verification the Security Analyzer field (All tab) is shown only while Confirmation Mode is on: switch Confirmation Mode on first, then pick the analyzer. Check `api GET /api/settings` after saving.\n- Confirmation mode with the default Security Analyzer `LLM` uses ConfirmRisky(HIGH): harmless commands run without a prompt. Set the analyzer to `None` to confirm every action, and restore `LLM` afterwards. The composer stays editable while waiting (status \"User needed\"); the buttons read Cancel / Continue.\n- The confirm and cancel shortcuts accept Meta or Ctrl and ignore a held key's auto-repeat; once a response is sent the listener is detached, so a second press sends nothing (#18127). Hints read `Ctrl+↩` / `Ctrl+⇧+⌫` off Apple platforms.\n- Known failure, Canvas UI ([#18202](https://github.com/OpenHands/OpenHands/issues/18202), reproduced 2026-10-09; [draft fix PR #18203](https://github.com/OpenHands/OpenHands/pull/18203)): after a user-message branch the composer holds the prefilled text but Send stays disabled (`browser enabled 'testid=submit-button'` is `false`, and a click on it times out with `element is not enabled`). The prefill is a saved draft that `src/hooks/chat/use-draft-persistence.ts` writes into the field without an input event, and the composer is reused across the client-side switch, so its Send state keeps the source conversation's empty field. Editing the text, a reload, or Enter in the field (`browser press Enter --selector 'testid=chat-input'`) all work. A draft restored by a sidebar switch behaves the same (F05 Gotchas).\n- Known failure, likely Agent Server: every message in a branched conversation ends with `ConversationErrorEvent` `LLMAuthenticationError` (\"Your LLM API key appears to be invalid or has expired.\"), while the source conversation keeps working (OpenHands/software-agent-sdk#5465). Do not use branches as fixtures for other checks.\n- \"The terminal session was reset because the underlying tmux server/session disappeared\" (clock `status-icon` on the row) appears after a `service stop agent-server` / `restart` cycle. Since `launch` gives each run its own short `TMUX_TMPDIR`, other runs no longer cause it. The underlying product bug is still open (OpenHands/OpenHands#17946) for plain `agent-canvas` instances: `bin/agent-canvas.mjs` → `scripts/dev-with-automation.mjs` passes `TMUX_TMPDIR=<state>/tmux` to the Agent Server but `ensureDirectories` never creates it (`dev-safe.mjs` does), so tmux 3.4 falls back to the shared `/tmp/tmux-<uid>/openhands` socket and another instance's restart or stop kills this run's windows. The model usually reruns the command; expect an extra action.\n- The `ConversationErrorEvent` banner is not restored after navigating away and back (or a reload); read the error from the events API instead.\n- After `restart`, reconnection takes about 20 s without a click; click `error-message-banner-retry` to reconnect at once. The error banner does not come back after a page reload once the stack is healthy.\n- \"Sending...\" lasts about 100 to 650 ms on a local stack, too short to hover and click `chat-message-stop`; that stop path is not driven here. The `F06.pending-messages` failure path needs `service stop agent-server`.\n- `browser clock` installs a fake clock that stays with the page across reloads and client-side navigation. Install it before the `goto` whose page should see it, and put it back with `browser clock --offset-ms 0` plus `browser reload`; compare `browser eval \"new Date().toISOString()\"` with `date -u` when in doubt.\n- The pending-message queue lives in page memory, so a failed bubble survives sidebar navigation and a backend `restart` but not a `browser reload` or `browser goto` of the conversation (`src/stores/optimistic-user-message-store.ts`). Leave and return through the sidebar when proving `F06.failed-send-persists`. The sidebar card's title is the first message or model-written: click its link by id, `'a[href*=\"/conversations/<id>\"] >> nth=0'`.\n- Both the pending bubble and the echoed message carry `testid=user-message`, so `count 'testid=user-message >> has-text=...'` is one too high while a failed bubble is still shown; count attempts with `conversation events <id> --kinds MessageEvent` (user rows: `\"source\": \"user\"`) and the bubbles with `chat-message-retry`.\n- Local `Stop` calls the Agent Server's interrupt (Cloud pauses the sandbox instead, see `pauseConversation`): a command already running in the terminal is cut off, recorded as `AgentErrorEvent` + `InterruptEvent`, and after Play the model only reports the interruption.\n- The no-LLM banner reuses the home banner's test ids (`home-llm-not-configured-banner`). It exists only in a run where no LLM was ever configured: use a separate `launch --new`, never remove the profile from the shared run.\n- Read the clipboard with `control-openhands browser clipboard`, never `browser eval \"navigator.clipboard.readText()\"`. An agent bubble that contains a code block holds two `copy-to-clipboard` buttons (the code block's is hidden until hovered): scope the bubble's with `>> nth=0`.\n- Plan-mode previews (`plan-preview-*`) and the BTW side messages render in this stream but are reached from the composer's mode controls; they are not mapped here.\n- Known issue [#18117](https://github.com/OpenHands/OpenHands/issues/18117) (`F06.thinking`): Inline thinking leaks into the agent chat bubble. Live prerequisite: A live model that emits a leading inline think block on a tool-call turn, plus its finalized ActionEvent.\n- Known issue [#17559](https://github.com/OpenHands/OpenHands/issues/17559) (`F06.load-older-history`): History pagination can show both an action and its observation (ready-for-dev companion to #16947). Live prerequisite: A real persisted history page boundary splitting an action/observation or ACP started/completed pair; dummy-key errors do not produce these tool pairs.\n- Known issue [#17492](https://github.com/OpenHands/OpenHands/issues/17492) (`F06.tool-visualizers`): Failed browser, glob, grep, and file editor tool calls render as successful in chat. Live prerequisite: A working model emitting failed browser, glob, grep, planning-editor and file-editor observations with is_error=true.\n- Known issue [#17490](https://github.com/OpenHands/OpenHands/issues/17490) (`F06.tool-visualizers`): Terminal commands that fail with an ordinary non-zero exit render as successful in chat. Live prerequisite: A working model producing a completed terminal command with nonzero exit_code and is_error=false.\n- Known issue [#17161](https://github.com/OpenHands/OpenHands/issues/17161) (`F06.markdown-rendering`): markdown sanitizer lets untrusted content set rel=\"opener\" and any target on links. Live prerequisite: A real reasoning, activated-skill, tool-description or error-detail body rendered without includeStandard. The attempted skills dialog exposed public skills only and did not reach the affected chat renderer.\n- Known issue [#17026](https://github.com/OpenHands/OpenHands/issues/17026) (`F06.thinking`): agent-canvas: final-reply thinking block missing for text-only turns (reasoning_content not read for MessageEvents). Live prerequisite: A reasoning-capable live model emitting nonempty MessageEvent.llm_message.reasoning_content on a text-only turn, followed by reload.\n- Known issue [#16842](https://github.com/OpenHands/OpenHands/issues/16842) (`F06.reconnect`): A late close from a replaced WebSocket marks the active connection disconnected. Live prerequisite: A transport fixture delaying an old socket close until its replacement is open; ordinary server restart cannot select that ordering.\n- Known issue [#16719](https://github.com/OpenHands/OpenHands/issues/16719) (`F06.live-activity`): Agent status always says \"thinking\" even when nothing is happening. Live prerequisite: The event/status trace and deterministic stalled or errored state from the original report; its trigger is unspecified.\n- Known issue [#16686](https://github.com/OpenHands/OpenHands/issues/16686) (`F06.error-events`): LLM errors not propagated to the frontend. Live prerequisite: A Nebius credential and its rejected DeepSeek-V3 request, or the original complete emitted error fixture.\n- Known issue [#16582](https://github.com/OpenHands/OpenHands/issues/16582) (`F06.stop-resume`): After reconnect, a maximum-iteration error can fail to pause the agent through the legacy event handler. Live prerequisite: A maximum-iteration legacy error delivered after socket replacement, with a real model run reaching that limit.\n- Known issue [#16455](https://github.com/OpenHands/OpenHands/issues/16455) (`F06.events-match`): Duplicate messages from the agent in 1.12.0. Live prerequisite: The original duplicated event/stream trace or deterministic prompt and timing; the report provides no reproducible trigger.\n- Known issue [#16341](https://github.com/OpenHands/OpenHands/issues/16341) (`F06.tool-visualizers`): Code-explorer subagent frequently errors. Live prerequisite: A working model with native code-explorer delegation and the original repository task; no delegated-tool output can be generated without credentials.\n- Known issue [#15626](https://github.com/OpenHands/OpenHands/issues/15626) (`F06.error-events`): Conversation can fail with LiteLLM auth error when database server at 127.0.0.1:5432 is unreachable. Live prerequisite: The original LiteLLM proxy/database-failure event or a disposable proxy with its PostgreSQL dependency unavailable.\n- Known issue [#15606](https://github.com/OpenHands/OpenHands/issues/15606) (`F06.reload-mid-run`): When re-loading ACP conversations, messages disappear or move. Live prerequisite: An authenticated Codex ACP session emitting intermediate messages while another window reloads history.\n- Known issue [#15527](https://github.com/OpenHands/OpenHands/issues/15527) (`F06.reconnect`): Canvas briefly shows \"disconnected\" when coming back to an idle tab. Live prerequisite: The original custom-domain deployment, idle duration and tab-suspension/network transition; those details are missing.\n- Known failure [OpenHands/software-agent-sdk#5635](https://github.com/OpenHands/software-agent-sdk/issues/5635) (`F06.reconnect`), reproduced 2026-10-12 at `fa1f724ab573` with SDK 1.54.0 and Chromium 153.0.8010.12: sandbox pause returned 204 but an already communicating session socket remained open for more than 10 seconds. A new Canvas tab received a subsequent `run:false` marker while the original tab did not. DELETE returned 200 and GET returned 404, yet both held sockets remained open for more than 10 seconds; Canvas later redirected by metadata polling. Recipe: open an idle dummy-profile conversation and confirm its marker/socket; POST `/api/conversations/prepare-for-sandbox-pause`, retain the original tab for 10 seconds, open a second tab, POST a new unique user event with `run:false`, then compare the two transcripts. DELETE the owned conversation and read lifecycle metadata for another 10 seconds. Use `--run <run>` and `browser events --kinds websocket-created,websocket-open,websocket-close --last 100`; no frame bodies or auth data are recorded. Evidence: `backfill-18309-surface-rechecks/F06.reconnect/pause-delete-sockets-5635.json`, `pause-new-tab-5635.png`, `pause-stale-tab-5635.png`. The expected socket shutdown/reconnect remains unchanged; polling recovery must not be mistaken for socket closure.\n- Known issue [OpenHands/software-agent-sdk#5465](https://github.com/OpenHands/software-agent-sdk/issues/5465) (`F06.branch-from-here`): Branch from here can lose encrypted LLM credentials, so the fork fails its first turn with LLMAuthenticationError. Live prerequisite: A working cipher-backed model conversation, then fork and a successful turn on the fork.\n- Known issue [OpenHands/software-agent-sdk#5447](https://github.com/OpenHands/software-agent-sdk/issues/5447) (`F06.error-events`): An Anthropic Claude Opus5.5 conversation can fail with400 when its next request ends in an assistant prefill. Live prerequisite: Anthropic Claude Opus5.5 credentials and a real trailing-assistant-prefill request.\n- Known issue [OpenHands/software-agent-sdk#5354](https://github.com/OpenHands/software-agent-sdk/issues/5354) (`F06.event-groups`): A mixed valid/invalid tool batch can be split into separate assistant messages, breaking call/result pairing on the next provider request. Live prerequisite: A live model turn mixing valid, invalid and valid tool calls in one response.\n- Known issue [OpenHands/software-agent-sdk#5225](https://github.com/OpenHands/software-agent-sdk/issues/5225) (`F06.error-events`): A provider-rejected image tool result can leave the conversation permanently failing on every resume. Live prerequisite: A real provider-rejected poisoned image tool_result in a disposable persisted history, followed by retry.\n- Known issue [OpenHands/software-agent-sdk#5108](https://github.com/OpenHands/software-agent-sdk/issues/5108) (`F06.tool-visualizers`): The native SDK Gemini editor can report success while corrupting unrelated invalid-UTF8 bytes in the file. Live prerequisite: An Agent Server configured with the native Gemini edit tool set and a real tool edit of an invalid-UTF8 fixture; Gemini ACP uses a different external executor.\n- Known issue [OpenHands/software-agent-sdk#5096](https://github.com/OpenHands/software-agent-sdk/issues/5096) (`F06.tool-visualizers`): The canonical agent file editor can replace a symlink instead of editing its target; Undo does not restore the link. Live prerequisite: A working model invoking canonical file-editor str_replace/insert through a real symlink, then undo.\n- Known issue [OpenHands/software-agent-sdk#5092](https://github.com/OpenHands/software-agent-sdk/issues/5092) (`F06.confirmation-mode`): A revocation message sent while waiting for confirmation can implicitly approve the pending action. Live prerequisite: A working model proposing a real confirmation-gated marker action, then a revocation user message.\n- Known issue [OpenHands/software-agent-sdk#5074](https://github.com/OpenHands/software-agent-sdk/issues/5074) (`F06.tool-visualizers`): After server restart a native delegated child can remain Running while the parent loses its task mapping/result. Live prerequisite: A native delegated child still running during an owned server crash and parent recovery.\n- Known issue [OpenHands/software-agent-sdk#5038](https://github.com/OpenHands/software-agent-sdk/issues/5038) (`F06.tool-visualizers`): The agent file editor can refuse valid Chinese UTF8 text as binary when its sample splits a character; Canvas Files uses a separate preview detector. Live prerequisite: A live canonical file-editor view/edit on the specific UTF8 sample alignment; Canvas Files preview uses a separate NUL sniff, not this validator.\n- Known issue [OpenHands/software-agent-sdk#4990](https://github.com/OpenHands/software-agent-sdk/issues/4990) (`F06.stop-resume`): An ACP turn can keep executing tool calls while the UI already reports Paused. Live prerequisite: An authenticated Claude Code ACP turn executing separate marker-writing tools while Pause is pressed.\n- Known issue [OpenHands/software-agent-sdk#4983](https://github.com/OpenHands/software-agent-sdk/issues/4983) (`F06.tool-visualizers`): Agent grep can silently miss extended-regex matches when its runtime falls back to system grep. Live prerequisite: A runtime without ripgrep and a live agent grep call using extended regex and escaped literal punctuation.\n- Known issue [OpenHands/software-agent-sdk#4965](https://github.com/OpenHands/software-agent-sdk/issues/4965) (`F06.error-events`): A strict OpenAI-compatible provider can reject blank content in the saved history, preventing later turns. Live prerequisite: A strict OpenAI-compatible provider and a real history containing blank text or empty content arrays.\n- Known issue [OpenHands/software-agent-sdk#4777](https://github.com/OpenHands/software-agent-sdk/issues/4777) (`F06.stop-resume`): Stop can still start a queued tool after it finishes waiting on a resource lock. Live prerequisite: A live batch with a second real tool waiting on a declared-resource lock when Stop is pressed.\n- Known issue [OpenHands/software-agent-sdk#4668](https://github.com/OpenHands/software-agent-sdk/issues/4668) (`F06.tool-visualizers`): A planning editor card can lose old/new content and fail to show the complete edit diff. Live prerequisite: A live planner create/edit yielding PlanningFileEditorObservation with old/new contents for the chat diff.\n- Known issue [OpenHands/software-agent-sdk#4663](https://github.com/OpenHands/software-agent-sdk/issues/4663) (`F06.tool-visualizers`): Parallel Python-fallback glob calls can change process cwd and return files from the wrong workspace. Live prerequisite: Concurrent real glob/tool calls in distinct workspaces with the Python fallback active.\n- Known issue [OpenHands/software-agent-sdk#4661](https://github.com/OpenHands/software-agent-sdk/issues/4661) (`F06.tool-visualizers`): Same-name project subagents can resolve to another conversation workspace definition. Live prerequisite: Two native task-enabled conversations with same-name project subagent definitions and a working model for delegation.\n- Known issue [OpenHands/software-agent-sdk#4583](https://github.com/OpenHands/software-agent-sdk/issues/4583) (`F06.tool-visualizers`): An agent editor insertion at EOF can join onto the last line when the file has no trailing newline. Live prerequisite: A working model invoking file-editor insert at EOF on a file with no trailing newline.\n- Known issue [OpenHands/software-agent-sdk#4558](https://github.com/OpenHands/software-agent-sdk/issues/4558) (`F06.error-events`): A saved encrypted fallback profile can send ciphertext as its key and fail authentication, leaving only the primary error visible. Live prerequisite: A working encrypted saved fallback profile and real primary-model failure that triggers fallback resolution.\n- Known issue [OpenHands/software-agent-sdk#4540](https://github.com/OpenHands/software-agent-sdk/issues/4540) (`F06.tool-visualizers`): Synthetic GLM/Kimi tool calls can render as raw tool_call syntax instead of executing. Live prerequisite: Synthetic provider credentials and a GLM/Kimi turn emitting the raw tool_call wrapper.\n- Known issue [OpenHands/software-agent-sdk#4532](https://github.com/OpenHands/software-agent-sdk/issues/4532) (`F06.confirmation-mode`): After restarting while awaiting confirmation, approval can execute the tool but the next model request omits its assistant call. Live prerequisite: A real confirmation-paused action persisted across backend restart, then approval and a working provider call.\n- Known issue [OpenHands/software-agent-sdk#4077](https://github.com/OpenHands/software-agent-sdk/issues/4077) (`F06.reload-mid-run`): Retried or late streaming deltas can duplicate live text; a stalled subscriber can delay Stop and finalization. Live prerequisite: A working stream interrupted/retried or delivered to a controlled stalled subscriber, with delta/state-order capture.\n- Known issue [#16947](https://github.com/OpenHands/OpenHands/issues/16947) (`F06.load-older-history`): Loading older history can leave both the action and its newer observation visible because UI projection depends on arrival order. Live prerequisite: A real long conversation whose REST boundary splits an action/observation pair; keep companion #17559 linked separately.\n\n- Known issue [#18445](https://github.com/OpenHands/OpenHands/issues/18445) (`F06.load-older-history`, `F06.reconnect`): a failed initial Cloud history search can be swallowed into an empty page while the runtime count succeeds, advancing `after_seq` past history that was never displayed. Live prerequisite: a Cloud account and owned conversation with a genuinely failing App API `events/search` and a successful runtime `events/count`. The Local failure path throws instead and is not an equivalent check.\n\n- Known issue [OpenHands/software-agent-sdk#5969](https://github.com/OpenHands/software-agent-sdk/issues/5969) (`F06.stop-resume`): a conversation can report finished but reject Play with already-running, or delay a follow-up, while a slow event webhook is still pending. Live prerequisite: a genuinely successful model run on an owned backend with a bounded slow webhook receiver, then Play or send during that completion window.\n\n- Known issue [OpenHands/software-agent-sdk#5740](https://github.com/OpenHands/software-agent-sdk/issues/5740) (`F06.error-events`): a Responses display delta without `item_id` can terminate the turn with an AttributeError. Live prerequisite: the genuine provider/transformer streaming path producing that incomplete event; the report did not retain its original raw SSE. A credential-free parser diagnostic proves the SDK exception, not a Canvas model turn.\n\n- Known issue [OpenHands/software-agent-sdk#5806](https://github.com/OpenHands/software-agent-sdk/issues/5806) (`F06.branch-from-here`): editing a displayed user message first reads its parent event; if that event has become unreadable/missing on the server, the action can fail with 500 instead of a missing-event response. Live prerequisite: an owned persisted conversation with a still-displayed user message whose event storage is missing, followed by the actual Edit action. A random unknown-ID API call proves neither this UI path nor the batch-get contract, which Canvas does not call.\n\n- Session reconnects use the latest received `after_seq` cursor (`F06.reconnect`). For a local metadata check, retain the open page, send genuine `run:false` user events, record `/events/count`, restart the owned backend and inspect finite WebSocket lifecycle metadata: the reconnect cursor is the previous count minus one and a subsequent recovery marker appears once. This does not measure frame-body replay or Cloud behavior.\n- Stop persists `paused` before an in-flight model request completes (`F06.stop-resume`); the persisted state and reloaded Stopped UI survive an Agent Server restart. A bounded credential-free check can use the tracked accept-only endpoint and an exact listener PID proven to descend from the owned launcher; never kill by pattern or resume the paused conversation.", "features": [ { "id": "F06.empty-state-suggestions", "description": "an empty conversation shows \"Let's start building!\" with four suggestion chips; a chip fills the composer with its prompt." }, { "id": "F06.user-and-agent-messages", "description": "user and agent bubbles; a long user message is clipped with a gradient and expands on click (\"View More\")." }, { "id": "F06.message-copy", "description": "hovering a bubble shows Copy to clipboard; it copies the message's markdown source and the label flips to Copied to clipboard for 2 s." }, { "id": "F06.code-block-copy", "description": "hovering a fenced code block in an agent reply shows its own Copy button, which copies only the code." }, { "id": "F06.timestamps", "description": "hovering a bubble or tool title shows the event's local date and time." }, { "id": "F06.pending-messages", "description": "a sent message shows \"Sending...\" until the server echoes it; a failed send shows Failed to send with Retry and Dismiss; Retry delivers it exactly once." }, { "id": "F06.clock-skew-send", "description": "a message sent while the browser clock runs ahead of the server is still confirmed by the server echo: no lingering \"Sending...\" and no Retry." }, { "id": "F06.failed-send-persists", "description": "a failed send keeps its Retry row after the backend returns and after leaving and reopening the conversation from the sidebar; sending the same text again succeeds without clearing it, and Retry then delivers that attempt exactly once." }, { "id": "F06.markdown-rendering", "description": "agent replies render markdown: tables inside a horizontal scroller with edge fades, fenced code, inline code." }, { "id": "F06.workspace-path-links", "description": "a workspace file path in an agent reply, and the path chip of a file-editor card, open that file in the Files tab." }, { "id": "F06.thinking", "description": "model reasoning appears as a collapsed \"Thinking\" row that expands." }, { "id": "F06.event-groups", "description": "consecutive tool calls collapse into \"N actions completed\" (or \"k/N actions completed\" with a spinner while one is pending); expanding lists one titled row per action." }, { "id": "F06.tool-visualizers", "description": "expanding a tool row shows a rich body: terminal command and output (non-zero exit badge `exit N`), file-editor path chip and content." }, { "id": "F06.markdown-file-preview", "description": "when the agent creates a Markdown file, its row stays expanded and ungrouped and shows a rendered, height-limited preview card with the file name and View, which opens the file in the Files tab." }, { "id": "F06.task-list", "description": "task-tracker calls render a \"Tasks\" card with done / todo icons." }, { "id": "F06.events-match", "description": "the stream shows one bubble per message event and one row per action, with no duplicates (compare with `conversation events`)." }, { "id": "F06.load-older-history", "description": "only the newest 50 events load first; the previous 50 are fetched when the user scrolls the transcript to its top (a \"Fetching older messages…\" row shows until they arrive) and automatically when the content is too short to scroll, so the first prompt and all actions appear." }, { "id": "F06.scroll-to-bottom", "description": "scrolling up shows a scroll-to-bottom button; clicking it returns to the bottom and hides it." }, { "id": "F06.live-activity", "description": "while the agent runs, a chip above the composer names the current action (\"Thinking\", then the action title) and the composer status reads Running." }, { "id": "F06.status-indicator", "description": "while a new conversation starts, a \"Starting\" pill shows above the composer." }, { "id": "F06.stop-resume", "description": "Stop in the composer interrupts the agent, including a running tool call (status Stopped, play button, an Agent error row for the interrupted call); Play resumes it to completion." }, { "id": "F06.reload-mid-run", "description": "reloading while the agent runs reattaches to the live run; it finishes without duplicated events." }, { "id": "F06.reconnect", "description": "when the Agent Server goes away the composer shows Disconnected; after it returns the page reconnects without a reload." }, { "id": "F06.error-banner", "description": "losing the server shows \"Unable to connect to server\" with Retry, Copy and Close; Retry reconnects once the server is back." }, { "id": "F06.error-events", "description": "a conversation error (ConversationErrorEvent) surfaces as a warning banner and the composer status Error." }, { "id": "F06.confirmation-mode", "description": "with confirmation mode on, a lock chip sits above the composer and each action waits for \"Do you want to continue with this action?\" with Cancel and Continue (hints `⇧⌘⌫` / `⌘↩` on macOS, `Ctrl+⇧+⌫` / `Ctrl+↩` elsewhere, hidden below 640 px); a cancelled action is marked Rejected and counts as resolved in its group." }, { "id": "F06.confirmation-shortcuts", "description": "Cmd+Enter or Ctrl+Enter continues and Shift+Cmd+Backspace or Shift+Ctrl+Backspace cancels a pending action." }, { "id": "F06.branch-from-here", "description": "hovering a message shows Branch from here; on an agent message it forks inclusively; on a user message it forks without that message and pre-fills its text in the new composer, ready to send. The branch is titled \"<title> (branch)\" and keeps working." }, { "id": "F06.image-attachments", "description": "images sent with a message show as thumbnails in the bubble; expand opens a lightbox that closes with X or Escape." }, { "id": "F06.skill-install-banner", "description": "after the agent installs a skill into the workspace, a banner offers \"Start new conversation with this skill\" (same workspace) and Close." }, { "id": "F06.llm-not-configured-banner", "description": "with no usable LLM, a conversation shows \"Your LLM isn't set up yet...\" with Set up LLM; suggestions are hidden and sending is disabled." }, { "id": "F06.critic-result", "description": "with the critic enabled, agent messages and the finishing action each carry a \"Critic: agent success likelihood\" block (stars labelled `Score: <N.N>%`, one decimal, with the percentage beside them); Expand details lists the issue categories (Potential Issues, Infrastructure, Likely Follow-up)." }, { "id": "F06.hook-events", "description": "hook executions render as their own rows in the stream." }, { "id": "F06.corrective-nudge", "description": "when the model answers with neither a message nor a tool call, the SDK's nudge (`Your last response did not include a function call or a message. …`) shows as a muted, italic `role=note` line with an info icon (`corrective-nudge-message`), not as a user bubble (#17864)." }, { "id": "F06.phone", "description": "at 390 px the transcript fits the viewport; wide tables scroll inside their own container." } ], "scenarios": [ { "name": "F06--one-run-that-exercises-the-stream", "title": "One run that exercises the stream", "ids": [ "F06.markdown-rendering", "F06.event-groups", "F06.thinking", "F06.events-match" ], "body": "Run `control-openhands conversation start --prompt \"Run the terminal command 'echo qa-f06-hello', then create a file qa_f06.txt containing the word hi. Finally reply with a markdown table with columns Step and Result (two rows) followed by a python code block that prints 1.\" --wait --timeout 300`, then `control-openhands browser testids 'testid=chat-interface'` and `control-openhands browser snapshot 'testid=agent-message'`. The testids include `user-message`, `event-group` (\"3 actions completed\"), `agent-message`, `markdown-table-scroll` with `markdown-table-scroll-fade-left`/`-right`; the snapshot shows a `table` with column headers Step and Result and a `code` block `print(1)`. Compare with `control-openhands conversation events <id>`: one user and one agent `MessageEvent`, three `ActionEvent`s (`terminal`, `file_editor`, `canvas_ui_control`), matching one `user-message`, one group of three rows and one `agent-message`. The thinking row (`collapsible-thinking`) lives inside the group, so it is listed only after the next bullet expands it.", "raw": "**One run that exercises the stream (`F06.markdown-rendering`, `F06.event-groups`, `F06.thinking`, `F06.events-match`).** Run `control-openhands conversation start --prompt \"Run the terminal command 'echo qa-f06-hello', then create a file qa_f06.txt containing the word hi. Finally reply with a markdown table with columns Step and Result (two rows) followed by a python code block that prints 1.\" --wait --timeout 300`, then `control-openhands browser testids 'testid=chat-interface'` and `control-openhands browser snapshot 'testid=agent-message'`. The testids include `user-message`, `event-group` (\"3 actions completed\"), `agent-message`, `markdown-table-scroll` with `markdown-table-scroll-fade-left`/`-right`; the snapshot shows a `table` with column headers Step and Result and a `code` block `print(1)`. Compare with `control-openhands conversation events <id>`: one user and one agent `MessageEvent`, three `ActionEvent`s (`terminal`, `file_editor`, `canvas_ui_control`), matching one `user-message`, one group of three rows and one `agent-message`. The thinking row (`collapsible-thinking`) lives inside the group, so it is listed only after the next bullet expands it.", "source_line": 74, "expected": [ "agent replies render markdown: tables inside a horizontal scroller with edge fades, fenced code, inline code.", "consecutive tool calls collapse into \"N actions completed\" (or \"k/N actions completed\" with a spinner while one is pending); expanding lists one titled row per action.", "model reasoning appears as a collapsed \"Thinking\" row that expands.", "the stream shows one bubble per message event and one row per action, with no duplicates (compare with `conversation events`)." ], "notes": [ "Known issue [#18117](https://github.com/OpenHands/OpenHands/issues/18117) (`F06.thinking`): Inline thinking leaks into the agent chat bubble. Live prerequisite: A live model that emits a leading inline think block on a tool-call turn, plus its finalized ActionEvent.", "Known issue [#17161](https://github.com/OpenHands/OpenHands/issues/17161) (`F06.markdown-rendering`): markdown sanitizer lets untrusted content set rel=\"opener\" and any target on links. Live prerequisite: A real reasoning, activated-skill, tool-description or error-detail body rendered without includeStandard. The attempted skills dialog exposed public skills only and did not reach the affected chat renderer.", "Known issue [#17026](https://github.com/OpenHands/OpenHands/issues/17026) (`F06.thinking`): agent-canvas: final-reply thinking block missing for text-only turns (reasoning_content not read for MessageEvents). Live prerequisite: A reasoning-capable live model emitting nonempty MessageEvent.llm_message.reasoning_content on a text-only turn, followed by reload.", "Known issue [#16455](https://github.com/OpenHands/OpenHands/issues/16455) (`F06.events-match`): Duplicate messages from the agent in 1.12.0. Live prerequisite: The original duplicated event/stream trace or deterministic prompt and timing; the report provides no reproducible trigger.", "Known issue [OpenHands/software-agent-sdk#5354](https://github.com/OpenHands/software-agent-sdk/issues/5354) (`F06.event-groups`): A mixed valid/invalid tool batch can be split into separate assistant messages, breaking call/result pairing on the next provider request. Live prerequisite: A live model turn mixing valid, invalid and valid tool calls in one response." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--expand-a-group-and-its-rows", "title": "Expand a group and its rows", "ids": [ "F06.event-groups", "F06.thinking", "F06.tool-visualizers" ], "body": "Run `control-openhands browser click 'testid=event-group-toggle'` (label becomes Collapse actions), `control-openhands browser text 'testid=event-group'` (three titles, e.g. \"Echo test string in terminal\", \"Create qa_f06.txt containing hi\"; titles are model-written), then `control-openhands browser click 'testid=event-group-content >> role=button[name=\"Expand\"][exact] >> nth=0'` twice (each click expands the next collapsed row). The group text now contains `echo qa-f06-hello` and its output `qa-f06-hello`, the file path chip (`testid=file-path-chip`) and the content `hi`. `control-openhands browser click 'testid=collapsible-thinking-toggle >> nth=0'` reveals `collapsible-thinking-content`. Take `control-openhands browser screenshot 'testid=chat-interface' --feature F06.tool-visualizers --name bash-and-file`.", "raw": "**Expand a group and its rows (`F06.event-groups`, `F06.thinking`, `F06.tool-visualizers`).** Run `control-openhands browser click 'testid=event-group-toggle'` (label becomes Collapse actions), `control-openhands browser text 'testid=event-group'` (three titles, e.g. \"Echo test string in terminal\", \"Create qa_f06.txt containing hi\"; titles are model-written), then `control-openhands browser click 'testid=event-group-content >> role=button[name=\"Expand\"][exact] >> nth=0'` twice (each click expands the next collapsed row). The group text now contains `echo qa-f06-hello` and its output `qa-f06-hello`, the file path chip (`testid=file-path-chip`) and the content `hi`. `control-openhands browser click 'testid=collapsible-thinking-toggle >> nth=0'` reveals `collapsible-thinking-content`. Take `control-openhands browser screenshot 'testid=chat-interface' --feature F06.tool-visualizers --name bash-and-file`.", "source_line": 75, "expected": [ "consecutive tool calls collapse into \"N actions completed\" (or \"k/N actions completed\" with a spinner while one is pending); expanding lists one titled row per action.", "model reasoning appears as a collapsed \"Thinking\" row that expands.", "expanding a tool row shows a rich body: terminal command and output (non-zero exit badge `exit N`), file-editor path chip and content." ], "notes": [ "Known issue [#18117](https://github.com/OpenHands/OpenHands/issues/18117) (`F06.thinking`): Inline thinking leaks into the agent chat bubble. Live prerequisite: A live model that emits a leading inline think block on a tool-call turn, plus its finalized ActionEvent.", "Known issue [#17492](https://github.com/OpenHands/OpenHands/issues/17492) (`F06.tool-visualizers`): Failed browser, glob, grep, and file editor tool calls render as successful in chat. Live prerequisite: A working model emitting failed browser, glob, grep, planning-editor and file-editor observations with is_error=true.", "Known issue [#17490](https://github.com/OpenHands/OpenHands/issues/17490) (`F06.tool-visualizers`): Terminal commands that fail with an ordinary non-zero exit render as successful in chat. Live prerequisite: A working model producing a completed terminal command with nonzero exit_code and is_error=false.", "Known issue [#17026](https://github.com/OpenHands/OpenHands/issues/17026) (`F06.thinking`): agent-canvas: final-reply thinking block missing for text-only turns (reasoning_content not read for MessageEvents). Live prerequisite: A reasoning-capable live model emitting nonempty MessageEvent.llm_message.reasoning_content on a text-only turn, followed by reload.", "Known issue [#16341](https://github.com/OpenHands/OpenHands/issues/16341) (`F06.tool-visualizers`): Code-explorer subagent frequently errors. Live prerequisite: A working model with native code-explorer delegation and the original repository task; no delegated-tool output can be generated without credentials.", "Known issue [OpenHands/software-agent-sdk#5354](https://github.com/OpenHands/software-agent-sdk/issues/5354) (`F06.event-groups`): A mixed valid/invalid tool batch can be split into separate assistant messages, breaking call/result pairing on the next provider request. Live prerequisite: A live model turn mixing valid, invalid and valid tool calls in one response.", "Known issue [OpenHands/software-agent-sdk#5108](https://github.com/OpenHands/software-agent-sdk/issues/5108) (`F06.tool-visualizers`): The native SDK Gemini editor can report success while corrupting unrelated invalid-UTF8 bytes in the file. Live prerequisite: An Agent Server configured with the native Gemini edit tool set and a real tool edit of an invalid-UTF8 fixture; Gemini ACP uses a different external executor.", "Known issue [OpenHands/software-agent-sdk#5096](https://github.com/OpenHands/software-agent-sdk/issues/5096) (`F06.tool-visualizers`): The canonical agent file editor can replace a symlink instead of editing its target; Undo does not restore the link. Live prerequisite: A working model invoking canonical file-editor str_replace/insert through a real symlink, then undo.", "Known issue [OpenHands/software-agent-sdk#5074](https://github.com/OpenHands/software-agent-sdk/issues/5074) (`F06.tool-visualizers`): After server restart a native delegated child can remain Running while the parent loses its task mapping/result. Live prerequisite: A native delegated child still running during an owned server crash and parent recovery.", "Known issue [OpenHands/software-agent-sdk#5038](https://github.com/OpenHands/software-agent-sdk/issues/5038) (`F06.tool-visualizers`): The agent file editor can refuse valid Chinese UTF8 text as binary when its sample splits a character; Canvas Files uses a separate preview detector. Live prerequisite: A live canonical file-editor view/edit on the specific UTF8 sample alignment; Canvas Files preview uses a separate NUL sniff, not this validator.", "Known issue [OpenHands/software-agent-sdk#4983](https://github.com/OpenHands/software-agent-sdk/issues/4983) (`F06.tool-visualizers`): Agent grep can silently miss extended-regex matches when its runtime falls back to system grep. Live prerequisite: A runtime without ripgrep and a live agent grep call using extended regex and escaped literal punctuation.", "Known issue [OpenHands/software-agent-sdk#4668](https://github.com/OpenHands/software-agent-sdk/issues/4668) (`F06.tool-visualizers`): A planning editor card can lose old/new content and fail to show the complete edit diff. Live prerequisite: A live planner create/edit yielding PlanningFileEditorObservation with old/new contents for the chat diff.", "Known issue [OpenHands/software-agent-sdk#4663](https://github.com/OpenHands/software-agent-sdk/issues/4663) (`F06.tool-visualizers`): Parallel Python-fallback glob calls can change process cwd and return files from the wrong workspace. Live prerequisite: Concurrent real glob/tool calls in distinct workspaces with the Python fallback active.", "Known issue [OpenHands/software-agent-sdk#4661](https://github.com/OpenHands/software-agent-sdk/issues/4661) (`F06.tool-visualizers`): Same-name project subagents can resolve to another conversation workspace definition. Live prerequisite: Two native task-enabled conversations with same-name project subagent definitions and a working model for delegation.", "Known issue [OpenHands/software-agent-sdk#4583](https://github.com/OpenHands/software-agent-sdk/issues/4583) (`F06.tool-visualizers`): An agent editor insertion at EOF can join onto the last line when the file has no trailing newline. Live prerequisite: A working model invoking file-editor insert at EOF on a file with no trailing newline.", "Known issue [OpenHands/software-agent-sdk#4540](https://github.com/OpenHands/software-agent-sdk/issues/4540) (`F06.tool-visualizers`): Synthetic GLM/Kimi tool calls can render as raw tool_call syntax instead of executing. Live prerequisite: Synthetic provider credentials and a GLM/Kimi turn emitting the raw tool_call wrapper." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--open-files-from-the-stream", "title": "Open files from the stream", "ids": [ "F06.workspace-path-links" ], "body": "On the same page the agent may already have opened the file; run `control-openhands browser click 'testid=file-quick-row-close-qa_f06.txt'` and check `control-openhands browser count 'testid=file-quick-row-item-qa_f06.txt'` is `0`. Run `control-openhands browser click 'testid=agent-message >> testid=markdown-file-path-link'`: the count is `1` and `control-openhands browser text 'testid=file-content-viewer-plain'` is `hi`. Close it again and `control-openhands browser click 'testid=file-path-chip'` (inside the expanded file-editor row): the count is `1` again.", "raw": "**Open files from the stream (`F06.workspace-path-links`).** On the same page the agent may already have opened the file; run `control-openhands browser click 'testid=file-quick-row-close-qa_f06.txt'` and check `control-openhands browser count 'testid=file-quick-row-item-qa_f06.txt'` is `0`. Run `control-openhands browser click 'testid=agent-message >> testid=markdown-file-path-link'`: the count is `1` and `control-openhands browser text 'testid=file-content-viewer-plain'` is `hi`. Close it again and `control-openhands browser click 'testid=file-path-chip'` (inside the expanded file-editor row): the count is `1` again.", "source_line": 76, "expected": [ "a workspace file path in an agent reply, and the path chip of a file-editor card, open that file in the Files tab." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--markdown-file-preview", "title": "Markdown file preview", "ids": [ "F06.markdown-file-preview" ], "body": "Run `control-openhands conversation start --prompt \"Use the file_editor tool to create the file qa_notes.md containing a heading '# QA Notes' and one bullet '- first'. Do nothing else and reply DONE.\" --wait --timeout 150`. Without any click, `control-openhands browser text 'testid=markdown-file-preview'` reads `QA Notes first qa_notes.md View` (rendered heading and bullet, then the file name) and `count 'testid=event-group'` is `0`. `control-openhands browser count 'testid=file-quick-row-item-qa_notes.md'` is `0`; `control-openhands browser click 'testid=markdown-file-preview-view'` makes it `1` and `control-openhands browser text 'testid=file-content-viewer-markdown'` reads `QA Notes first`. Then `control-openhands browser goto /conversations/<stream id>` to return to the stream conversation for the next bullets.", "raw": "**Markdown file preview (`F06.markdown-file-preview`).** Run `control-openhands conversation start --prompt \"Use the file_editor tool to create the file qa_notes.md containing a heading '# QA Notes' and one bullet '- first'. Do nothing else and reply DONE.\" --wait --timeout 150`. Without any click, `control-openhands browser text 'testid=markdown-file-preview'` reads `QA Notes first qa_notes.md View` (rendered heading and bullet, then the file name) and `count 'testid=event-group'` is `0`. `control-openhands browser count 'testid=file-quick-row-item-qa_notes.md'` is `0`; `control-openhands browser click 'testid=markdown-file-preview-view'` makes it `1` and `control-openhands browser text 'testid=file-content-viewer-markdown'` reads `QA Notes first`. Then `control-openhands browser goto /conversations/<stream id>` to return to the stream conversation for the next bullets.", "source_line": 77, "expected": [ "when the agent creates a Markdown file, its row stays expanded and ungrouped and shows a rendered, height-limited preview card with the file name and View, which opens the file in the Files tab." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--timestamps", "title": "Timestamps", "ids": [ "F06.timestamps" ], "body": "Run `control-openhands browser tooltip 'testid=user-message'` and `control-openhands browser tooltip 'testid=event-group-toggle'`. Both return the local date and time, e.g. `Oct 4, 2026, 11:08 PM`.", "raw": "**Timestamps (`F06.timestamps`).** Run `control-openhands browser tooltip 'testid=user-message'` and `control-openhands browser tooltip 'testid=event-group-toggle'`. Both return the local date and time, e.g. `Oct 4, 2026, 11:08 PM`.", "source_line": 78, "expected": [ "hovering a bubble or tool title shows the event's local date and time." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--copy-a-message-and-a-code-block", "title": "Copy a message and a code block", "ids": [ "F06.message-copy", "F06.code-block-copy" ], "body": "On the same page run `control-openhands browser clipboard --write qa-empty`, `control-openhands browser hover 'testid=agent-message >> nth=0'` and `control-openhands browser click 'testid=agent-message >> nth=0 >> testid=copy-to-clipboard >> nth=0'` (the bubble's button; the code block has a second, hidden `copy-to-clipboard`). `control-openhands browser attr 'testid=agent-message >> nth=0 >> testid=copy-to-clipboard >> nth=0' aria-label` is `Copied to clipboard`, and `Copy to clipboard` again about 2 s later; `control-openhands browser clipboard` starts with `| Step | Result |` and ends with the fenced `print(1)` block. Then `control-openhands browser hover 'testid=agent-message >> pre'`, `control-openhands browser click 'testid=agent-message >> pre >> testid=copy-to-clipboard'` and `control-openhands browser clipboard` is exactly `print(1)`.", "raw": "**Copy a message and a code block (`F06.message-copy`, `F06.code-block-copy`).** On the same page run `control-openhands browser clipboard --write qa-empty`, `control-openhands browser hover 'testid=agent-message >> nth=0'` and `control-openhands browser click 'testid=agent-message >> nth=0 >> testid=copy-to-clipboard >> nth=0'` (the bubble's button; the code block has a second, hidden `copy-to-clipboard`). `control-openhands browser attr 'testid=agent-message >> nth=0 >> testid=copy-to-clipboard >> nth=0' aria-label` is `Copied to clipboard`, and `Copy to clipboard` again about 2 s later; `control-openhands browser clipboard` starts with `| Step | Result |` and ends with the fenced `print(1)` block. Then `control-openhands browser hover 'testid=agent-message >> pre'`, `control-openhands browser click 'testid=agent-message >> pre >> testid=copy-to-clipboard'` and `control-openhands browser clipboard` is exactly `print(1)`.", "source_line": 79, "expected": [ "hovering a bubble shows Copy to clipboard; it copies the message's markdown source and the label flips to Copied to clipboard for 2 s.", "hovering a fenced code block in an agent reply shows its own Copy button, which copies only the code." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--long-table-cell", "title": "Long table cell", "ids": [ "F06.markdown-rendering" ], "body": "On the stream conversation write a prompt file whose first line is `Reply with only the word OK. Do not run any tools.`, then a blank line and the table `| Name | Description | Status |`, `|---|---|---|`, `| Alpha | This is a really long sentence in a single cell that goes on and on to explain a very involved aspect of the system, which nobody could read on one line without scrolling forever to the right. | ok |`, `| Beta | brief | fail |`. Run `control-openhands browser fill 'testid=chat-input' x --value-file <file>`, `control-openhands browser click 'testid=submit-button'` and `control-openhands conversation wait <id> --fresh --timeout 120`; the user bubble renders the table. Cells wrap once they reach 28rem (`max-w-md`): `control-openhands browser eval \"getComputedStyle(document.querySelector('[data-testid=markdown-table-scroll] td')).whiteSpace + ' ' + getComputedStyle(document.querySelector('[data-testid=markdown-table-scroll] td')).maxWidth\"` is `normal 448px`. At desktop with the right panel closed, `control-openhands browser bbox 'testid=user-message >> nth=-1 >> testid=markdown-table-scroll'` has `scrollWidth` `582` equal to `clientWidth` and `Status` is visible (`browser screenshot 'testid=chat-interface' --feature F06.markdown-rendering --name long-cell-table`). A table still wider than the conversation keeps the horizontal scroll with its edge fade: with the panel open (`control-openhands browser click 'testid=right-panel-toggle'`) `clientWidth` is `474`, and at phone width `326`, so `Status` is scrolled off there while the long cell stays wrapped. An agent reply with such a table behaves the same.", "raw": "**Long table cell (`F06.markdown-rendering`).** On the stream conversation write a prompt file whose first line is `Reply with only the word OK. Do not run any tools.`, then a blank line and the table `| Name | Description | Status |`, `|---|---|---|`, `| Alpha | This is a really long sentence in a single cell that goes on and on to explain a very involved aspect of the system, which nobody could read on one line without scrolling forever to the right. | ok |`, `| Beta | brief | fail |`. Run `control-openhands browser fill 'testid=chat-input' x --value-file <file>`, `control-openhands browser click 'testid=submit-button'` and `control-openhands conversation wait <id> --fresh --timeout 120`; the user bubble renders the table. Cells wrap once they reach 28rem (`max-w-md`): `control-openhands browser eval \"getComputedStyle(document.querySelector('[data-testid=markdown-table-scroll] td')).whiteSpace + ' ' + getComputedStyle(document.querySelector('[data-testid=markdown-table-scroll] td')).maxWidth\"` is `normal 448px`. At desktop with the right panel closed, `control-openhands browser bbox 'testid=user-message >> nth=-1 >> testid=markdown-table-scroll'` has `scrollWidth` `582` equal to `clientWidth` and `Status` is visible (`browser screenshot 'testid=chat-interface' --feature F06.markdown-rendering --name long-cell-table`). A table still wider than the conversation keeps the horizontal scroll with its edge fade: with the panel open (`control-openhands browser click 'testid=right-panel-toggle'`) `clientWidth` is `474`, and at phone width `326`, so `Status` is scrolled off there while the long cell stays wrapped. An agent reply with such a table behaves the same.", "source_line": 80, "expected": [ "agent replies render markdown: tables inside a horizontal scroller with edge fades, fenced code, inline code." ], "notes": [ "Known issue [#17161](https://github.com/OpenHands/OpenHands/issues/17161) (`F06.markdown-rendering`): markdown sanitizer lets untrusted content set rel=\"opener\" and any target on links. Live prerequisite: A real reasoning, activated-skill, tool-description or error-detail body rendered without includeStandard. The attempted skills dialog exposed public skills only and did not reach the affected chat renderer." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--non-zero-exit-badge", "title": "Non-zero exit badge", "ids": [ "F06.tool-visualizers" ], "body": "In an open conversation run `control-openhands browser fill 'testid=chat-input' \"Run exactly this terminal command, verbatim, with nothing appended: ls /qa_missing_dir2 . Then reply OK.\"`, `control-openhands browser click 'testid=submit-button'` and `control-openhands conversation wait <id> --fresh --timeout 150` (without `--fresh` it returns the previous run's `finished` at once). If the reply's actions render as a single row, run `control-openhands browser click 'testid=chat-scroll-container >> role=button[name=\"Expand\"][exact] >> nth=-1'`; if they render as a group (`browser count 'testid=event-group'` grew, e.g. `2 actions completed` after a terminal reset and rerun), run `control-openhands browser click 'testid=event-group-toggle >> nth=-1'` and `control-openhands browser click 'testid=event-group-content >> nth=-1 >> role=button[name=\"Expand\"][exact] >> nth=-1'`. Then `control-openhands browser count 'testid=chat-scroll-container >> text=\"exit 2\"'` is `1`; the badge sits above the `No such file or directory` output (`browser screenshot 'testid=event-group >> nth=-1' --feature F06.tool-visualizers --name exit-code`). Repeating the check in the same conversation needs a new directory name and a count that grows, because an earlier expanded badge stays on the page.", "raw": "**Non-zero exit badge (`F06.tool-visualizers`).** In an open conversation run `control-openhands browser fill 'testid=chat-input' \"Run exactly this terminal command, verbatim, with nothing appended: ls /qa_missing_dir2 . Then reply OK.\"`, `control-openhands browser click 'testid=submit-button'` and `control-openhands conversation wait <id> --fresh --timeout 150` (without `--fresh` it returns the previous run's `finished` at once). If the reply's actions render as a single row, run `control-openhands browser click 'testid=chat-scroll-container >> role=button[name=\"Expand\"][exact] >> nth=-1'`; if they render as a group (`browser count 'testid=event-group'` grew, e.g. `2 actions completed` after a terminal reset and rerun), run `control-openhands browser click 'testid=event-group-toggle >> nth=-1'` and `control-openhands browser click 'testid=event-group-content >> nth=-1 >> role=button[name=\"Expand\"][exact] >> nth=-1'`. Then `control-openhands browser count 'testid=chat-scroll-container >> text=\"exit 2\"'` is `1`; the badge sits above the `No such file or directory` output (`browser screenshot 'testid=event-group >> nth=-1' --feature F06.tool-visualizers --name exit-code`). Repeating the check in the same conversation needs a new directory name and a count that grows, because an earlier expanded badge stays on the page.", "source_line": 81, "expected": [ "expanding a tool row shows a rich body: terminal command and output (non-zero exit badge `exit N`), file-editor path chip and content." ], "notes": [ "Known issue [#17492](https://github.com/OpenHands/OpenHands/issues/17492) (`F06.tool-visualizers`): Failed browser, glob, grep, and file editor tool calls render as successful in chat. Live prerequisite: A working model emitting failed browser, glob, grep, planning-editor and file-editor observations with is_error=true.", "Known issue [#17490](https://github.com/OpenHands/OpenHands/issues/17490) (`F06.tool-visualizers`): Terminal commands that fail with an ordinary non-zero exit render as successful in chat. Live prerequisite: A working model producing a completed terminal command with nonzero exit_code and is_error=false.", "Known issue [#16341](https://github.com/OpenHands/OpenHands/issues/16341) (`F06.tool-visualizers`): Code-explorer subagent frequently errors. Live prerequisite: A working model with native code-explorer delegation and the original repository task; no delegated-tool output can be generated without credentials.", "Known issue [OpenHands/software-agent-sdk#5108](https://github.com/OpenHands/software-agent-sdk/issues/5108) (`F06.tool-visualizers`): The native SDK Gemini editor can report success while corrupting unrelated invalid-UTF8 bytes in the file. Live prerequisite: An Agent Server configured with the native Gemini edit tool set and a real tool edit of an invalid-UTF8 fixture; Gemini ACP uses a different external executor.", "Known issue [OpenHands/software-agent-sdk#5096](https://github.com/OpenHands/software-agent-sdk/issues/5096) (`F06.tool-visualizers`): The canonical agent file editor can replace a symlink instead of editing its target; Undo does not restore the link. Live prerequisite: A working model invoking canonical file-editor str_replace/insert through a real symlink, then undo.", "Known issue [OpenHands/software-agent-sdk#5074](https://github.com/OpenHands/software-agent-sdk/issues/5074) (`F06.tool-visualizers`): After server restart a native delegated child can remain Running while the parent loses its task mapping/result. Live prerequisite: A native delegated child still running during an owned server crash and parent recovery.", "Known issue [OpenHands/software-agent-sdk#5038](https://github.com/OpenHands/software-agent-sdk/issues/5038) (`F06.tool-visualizers`): The agent file editor can refuse valid Chinese UTF8 text as binary when its sample splits a character; Canvas Files uses a separate preview detector. Live prerequisite: A live canonical file-editor view/edit on the specific UTF8 sample alignment; Canvas Files preview uses a separate NUL sniff, not this validator.", "Known issue [OpenHands/software-agent-sdk#4983](https://github.com/OpenHands/software-agent-sdk/issues/4983) (`F06.tool-visualizers`): Agent grep can silently miss extended-regex matches when its runtime falls back to system grep. Live prerequisite: A runtime without ripgrep and a live agent grep call using extended regex and escaped literal punctuation.", "Known issue [OpenHands/software-agent-sdk#4668](https://github.com/OpenHands/software-agent-sdk/issues/4668) (`F06.tool-visualizers`): A planning editor card can lose old/new content and fail to show the complete edit diff. Live prerequisite: A live planner create/edit yielding PlanningFileEditorObservation with old/new contents for the chat diff.", "Known issue [OpenHands/software-agent-sdk#4663](https://github.com/OpenHands/software-agent-sdk/issues/4663) (`F06.tool-visualizers`): Parallel Python-fallback glob calls can change process cwd and return files from the wrong workspace. Live prerequisite: Concurrent real glob/tool calls in distinct workspaces with the Python fallback active.", "Known issue [OpenHands/software-agent-sdk#4661](https://github.com/OpenHands/software-agent-sdk/issues/4661) (`F06.tool-visualizers`): Same-name project subagents can resolve to another conversation workspace definition. Live prerequisite: Two native task-enabled conversations with same-name project subagent definitions and a working model for delegation.", "Known issue [OpenHands/software-agent-sdk#4583](https://github.com/OpenHands/software-agent-sdk/issues/4583) (`F06.tool-visualizers`): An agent editor insertion at EOF can join onto the last line when the file has no trailing newline. Live prerequisite: A working model invoking file-editor insert at EOF on a file with no trailing newline.", "Known issue [OpenHands/software-agent-sdk#4540](https://github.com/OpenHands/software-agent-sdk/issues/4540) (`F06.tool-visualizers`): Synthetic GLM/Kimi tool calls can render as raw tool_call syntax instead of executing. Live prerequisite: Synthetic provider credentials and a GLM/Kimi turn emitting the raw tool_call wrapper." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--task-list", "title": "Task list", "ids": [ "F06.task-list", "F06.status-indicator" ], "body": "From `/` run `control-openhands browser goto /`, `control-openhands browser fill 'testid=chat-input' \"Use your task_tracker tool to plan exactly two tasks titled 'qa step one' and 'qa step two'; mark 'qa step one' done and 'qa step two' todo. Do nothing else and reply DONE.\"` and `control-openhands browser click 'testid=submit-button' --expect-url '/conversations/' --observe '[data-testid=chat-status-indicator]' --observe-ms 8000`. `observed` contains `Starting` for a moment (the pill). Then `control-openhands conversation wait <id> --timeout 150` and `control-openhands browser text 'testid=chat-scroll-container'`: it contains `Tasks`, `qa step one`, `qa step two`; the screenshot shows a check icon and muted text for the done task, an empty circle for the todo one.", "raw": "**Task list (`F06.task-list`, `F06.status-indicator`).** From `/` run `control-openhands browser goto /`, `control-openhands browser fill 'testid=chat-input' \"Use your task_tracker tool to plan exactly two tasks titled 'qa step one' and 'qa step two'; mark 'qa step one' done and 'qa step two' todo. Do nothing else and reply DONE.\"` and `control-openhands browser click 'testid=submit-button' --expect-url '/conversations/' --observe '[data-testid=chat-status-indicator]' --observe-ms 8000`. `observed` contains `Starting` for a moment (the pill). Then `control-openhands conversation wait <id> --timeout 150` and `control-openhands browser text 'testid=chat-scroll-container'`: it contains `Tasks`, `qa step one`, `qa step two`; the screenshot shows a check icon and muted text for the done task, an empty circle for the todo one.", "source_line": 82, "expected": [ "task-tracker calls render a \"Tasks\" card with done / todo icons.", "while a new conversation starts, a \"Starting\" pill shows above the composer." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--empty-state", "title": "Empty state", "ids": [ "F06.empty-state-suggestions" ], "body": "Run `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=launch-no-workspace' --expect-url '/conversations/'`, then `control-openhands browser text 'testid=chat-suggestions'`: `Let's start building!` plus `Increase test coverage`, `Auto-merge PRs`, `Fix README`, `Clean dependencies`. Run `control-openhands browser click 'testid=chat-suggestions >> text=Fix README'` and `control-openhands browser text 'testid=chat-input'`: the composer holds the README-improvement prompt.", "raw": "**Empty state (`F06.empty-state-suggestions`).** Run `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=launch-no-workspace' --expect-url '/conversations/'`, then `control-openhands browser text 'testid=chat-suggestions'`: `Let's start building!` plus `Increase test coverage`, `Auto-merge PRs`, `Fix README`, `Clean dependencies`. Run `control-openhands browser click 'testid=chat-suggestions >> text=Fix README'` and `control-openhands browser text 'testid=chat-input'`: the composer holds the README-improvement prompt.", "source_line": 83, "expected": [ "an empty conversation shows \"Let's start building!\" with four suggestion chips; a chip fills the composer with its prompt." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--pending-bubble-and-long-messages", "title": "Pending bubble and long messages", "ids": [ "F06.pending-messages", "F06.user-and-agent-messages" ], "body": "Write a 25-line message to a file (first line `Reply with only the word OK. Ignore the filler lines below.`, then `filler line 1`…`filler line 24`), run `control-openhands browser fill 'testid=chat-input' x --value-file <file>` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending]' --observe-ms 3000`. `observed` shows `Sending...` then `<absent>` once the server echoes it. After `control-openhands conversation wait <id> --timeout 120`, `control-openhands browser testids 'testid=chat-scroll-container'` lists `chat-message-truncation-gradient` and `chat-message-expand` (\"View More\"); `control-openhands browser click 'testid=chat-message-expand'` removes both and the whole text shows. `testid=agent-message` reads `OK`.", "raw": "**Pending bubble and long messages (`F06.pending-messages`, `F06.user-and-agent-messages`).** Write a 25-line message to a file (first line `Reply with only the word OK. Ignore the filler lines below.`, then `filler line 1`…`filler line 24`), run `control-openhands browser fill 'testid=chat-input' x --value-file <file>` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending]' --observe-ms 3000`. `observed` shows `Sending...` then `<absent>` once the server echoes it. After `control-openhands conversation wait <id> --timeout 120`, `control-openhands browser testids 'testid=chat-scroll-container'` lists `chat-message-truncation-gradient` and `chat-message-expand` (\"View More\"); `control-openhands browser click 'testid=chat-message-expand'` removes both and the whole text shows. `testid=agent-message` reads `OK`.", "source_line": 84, "expected": [ "a sent message shows \"Sending...\" until the server echoes it; a failed send shows Failed to send with Retry and Dismiss; Retry delivers it exactly once.", "user and agent bubbles; a long user message is clipped with a gradient and expands on click (\"View More\")." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--browser-clock-ahead", "title": "Browser clock ahead", "ids": [ "F06.clock-skew-send" ], "body": "On the same conversation, skew the page's clock before loading it: `control-openhands browser clock --offset-ms 300000`, `control-openhands browser goto /conversations/<id>`, `control-openhands browser wait 'testid=chat-input'`; `control-openhands browser eval \"new Date().toISOString()\"` is about five minutes ahead of `date -u` (read-only shell check). Run `control-openhands browser fill 'testid=chat-input' 'Reply with only: clock-ok'` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending],[data-testid=chat-message-error]' --observe-ms 6000`: `observed` shows `Sending...` and then `<absent>` (the server echo confirmed it), never `Failed to send`. `control-openhands browser count 'testid=chat-message-sending'` and `control-openhands browser count 'testid=chat-message-retry'` are `0`, and `control-openhands conversation events <id> --kinds MessageEvent` gains one user row `Reply with only: clock-ok` whose `ts` is the server's time (close to `date -u`), not the browser's. Undo the skew before moving on: `control-openhands browser clock --offset-ms 0` and `control-openhands browser reload` (with a model, `control-openhands conversation wait <id> --fresh --timeout 120` first).", "raw": "**Browser clock ahead (`F06.clock-skew-send`).** On the same conversation, skew the page's clock before loading it: `control-openhands browser clock --offset-ms 300000`, `control-openhands browser goto /conversations/<id>`, `control-openhands browser wait 'testid=chat-input'`; `control-openhands browser eval \"new Date().toISOString()\"` is about five minutes ahead of `date -u` (read-only shell check). Run `control-openhands browser fill 'testid=chat-input' 'Reply with only: clock-ok'` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending],[data-testid=chat-message-error]' --observe-ms 6000`: `observed` shows `Sending...` and then `<absent>` (the server echo confirmed it), never `Failed to send`. `control-openhands browser count 'testid=chat-message-sending'` and `control-openhands browser count 'testid=chat-message-retry'` are `0`, and `control-openhands conversation events <id> --kinds MessageEvent` gains one user row `Reply with only: clock-ok` whose `ts` is the server's time (close to `date -u`), not the browser's. Undo the skew before moving on: `control-openhands browser clock --offset-ms 0` and `control-openhands browser reload` (with a model, `control-openhands conversation wait <id> --fresh --timeout 120` first).", "source_line": 85, "expected": [ "a message sent while the browser clock runs ahead of the server is still confirmed by the server echo: no lingering \"Sending...\" and no Retry." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--server-lost-banner-failed-send-reconnect", "title": "Server lost: banner, failed send, reconnect", "ids": [ "F06.error-banner", "F06.pending-messages", "F06.reconnect" ], "body": "With a conversation open run `control-openhands service stop agent-server` and `control-openhands browser wait 'testid=error-message-banner' --timeout 20000`: the banner reads `Unable to connect to server` with `error-message-banner-retry`, `-copy` and `-dismiss`; the composer shows `Disconnected`. Run `control-openhands browser fill 'testid=chat-input' 'Reply with only the word PONG.'` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending],[data-testid=chat-message-error]' --observe-ms 8000`: `Sending...` turns into `Failed to send Retry Dismiss`. `control-openhands browser click 'testid=chat-message-retry'` while down fails again; `control-openhands browser click 'testid=chat-message-dismiss'` removes the bubble (`count 'testid=chat-message-error'` is `0`). Send the same message again so one failed bubble remains, `control-openhands browser click 'testid=error-message-banner-dismiss'` (banner count `0`), then `control-openhands restart`. Poll `control-openhands browser text 'testid=interactive-chat-box'` every 10 s: within about a minute after `restart` returns it no longer contains `Disconnected` (reconnected without reload). `control-openhands browser click 'testid=chat-message-retry'`, `control-openhands conversation wait <id> --timeout 120`, then `control-openhands conversation events <id> --kinds MessageEvent`: `Reply with only the word PONG.` appears once and the last `agent-message` is `PONG`.", "raw": "**Server lost: banner, failed send, reconnect (`F06.error-banner`, `F06.pending-messages`, `F06.reconnect`).** With a conversation open run `control-openhands service stop agent-server` and `control-openhands browser wait 'testid=error-message-banner' --timeout 20000`: the banner reads `Unable to connect to server` with `error-message-banner-retry`, `-copy` and `-dismiss`; the composer shows `Disconnected`. Run `control-openhands browser fill 'testid=chat-input' 'Reply with only the word PONG.'` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending],[data-testid=chat-message-error]' --observe-ms 8000`: `Sending...` turns into `Failed to send Retry Dismiss`. `control-openhands browser click 'testid=chat-message-retry'` while down fails again; `control-openhands browser click 'testid=chat-message-dismiss'` removes the bubble (`count 'testid=chat-message-error'` is `0`). Send the same message again so one failed bubble remains, `control-openhands browser click 'testid=error-message-banner-dismiss'` (banner count `0`), then `control-openhands restart`. Poll `control-openhands browser text 'testid=interactive-chat-box'` every 10 s: within about a minute after `restart` returns it no longer contains `Disconnected` (reconnected without reload). `control-openhands browser click 'testid=chat-message-retry'`, `control-openhands conversation wait <id> --timeout 120`, then `control-openhands conversation events <id> --kinds MessageEvent`: `Reply with only the word PONG.` appears once and the last `agent-message` is `PONG`.", "source_line": 86, "expected": [ "losing the server shows \"Unable to connect to server\" with Retry, Copy and Close; Retry reconnects once the server is back.", "a sent message shows \"Sending...\" until the server echoes it; a failed send shows Failed to send with Retry and Dismiss; Retry delivers it exactly once.", "when the Agent Server goes away the composer shows Disconnected; after it returns the page reconnects without a reload." ], "notes": [ "Known issue [#16842](https://github.com/OpenHands/OpenHands/issues/16842) (`F06.reconnect`): A late close from a replaced WebSocket marks the active connection disconnected. Live prerequisite: A transport fixture delaying an old socket close until its replacement is open; ordinary server restart cannot select that ordering.", "Known issue [#15527](https://github.com/OpenHands/OpenHands/issues/15527) (`F06.reconnect`): Canvas briefly shows \"disconnected\" when coming back to an idle tab. Live prerequisite: The original custom-domain deployment, idle duration and tab-suspension/network transition; those details are missing.", "Known failure [OpenHands/software-agent-sdk#5635](https://github.com/OpenHands/software-agent-sdk/issues/5635) (`F06.reconnect`), reproduced 2026-10-12 at `fa1f724ab573` with SDK 1.54.0 and Chromium 153.0.8010.12: sandbox pause returned 204 but an already communicating session socket remained open for more than 10 seconds. A new Canvas tab received a subsequent `run:false` marker while the original tab did not. DELETE returned 200 and GET returned 404, yet both held sockets remained open for more than 10 seconds; Canvas later redirected by metadata polling. Recipe: open an idle dummy-profile conversation and confirm its marker/socket; POST `/api/conversations/prepare-for-sandbox-pause`, retain the original tab for 10 seconds, open a second tab, POST a new unique user event with `run:false`, then compare the two transcripts. DELETE the owned conversation and read lifecycle metadata for another 10 seconds. Use `--run <run>` and `browser events --kinds websocket-created,websocket-open,websocket-close --last 100`; no frame bodies or auth data are recorded. Evidence: `backfill-18309-surface-rechecks/F06.reconnect/pause-delete-sockets-5635.json`, `pause-new-tab-5635.png`, `pause-stale-tab-5635.png`. The expected socket shutdown/reconnect remains unchanged; polling recovery must not be mistaken for socket closure.", "Known issue [#18445](https://github.com/OpenHands/OpenHands/issues/18445) (`F06.load-older-history`, `F06.reconnect`): a failed initial Cloud history search can be swallowed into an empty page while the runtime count succeeds, advancing `after_seq` past history that was never displayed. Live prerequisite: a Cloud account and owned conversation with a genuinely failing App API `events/search` and a successful runtime `events/count`. The Local failure path throws instead and is not an equivalent check." ], "annotations": [], "states": [ "failure", "issue", "blocked" ] }, { "name": "F06--banner-retry", "title": "Banner Retry", "ids": [ "F06.error-banner" ], "body": "Run `control-openhands service stop agent-server`, wait for `testid=error-message-banner` as above, `control-openhands restart`, then `control-openhands browser click 'testid=error-message-banner-retry'`. `control-openhands browser count 'testid=error-message-banner'` is `0` and the composer is no longer Disconnected. Take `browser screenshot 'testid=chat-interface' --feature F06.error-banner --name disconnected` while the banner is up.", "raw": "**Banner Retry (`F06.error-banner`).** Run `control-openhands service stop agent-server`, wait for `testid=error-message-banner` as above, `control-openhands restart`, then `control-openhands browser click 'testid=error-message-banner-retry'`. `control-openhands browser count 'testid=error-message-banner'` is `0` and the composer is no longer Disconnected. Take `browser screenshot 'testid=chat-interface' --feature F06.error-banner --name disconnected` while the banner is up.", "source_line": 87, "expected": [ "losing the server shows \"Unable to connect to server\" with Retry, Copy and Close; Retry reconnects once the server is back." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--failed-send-survives-leaving", "title": "Failed send survives leaving", "ids": [ "F06.failed-send-persists" ], "body": "With `/conversations/<id>` open and healthy, run `control-openhands service stop agent-server`, `control-openhands browser wait 'testid=error-message-banner' --timeout 20000`, `control-openhands browser fill 'testid=chat-input' 'Reply with only: retry-ok'` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending],[data-testid=chat-message-error]' --observe-ms 8000`: `Sending...` turns into `Failed to send Retry Dismiss` and `control-openhands browser count 'testid=chat-message-retry'` is `1` (`control-openhands browser screenshot 'testid=chat-interface' --feature F06.failed-send-persists --name failed-before-leaving`). Run `control-openhands restart` and `control-openhands browser wait 'testid=error-message-banner' --state hidden --timeout 90000` (the page reconnects by itself; `testid=error-message-banner-retry` hurries it while the banner is still up): the Retry row is still there (count `1`). Leave through the sidebar and come back: `control-openhands browser click 'role=link[name=\"New Chat\"]'`, `control-openhands browser wait 'testid=home-chat-launcher'`, `control-openhands browser click 'a[href*=\"/conversations/<id>\"] >> nth=0' --expect-url '/conversations/<id>'`, `control-openhands browser wait 'testid=user-message'`; `control-openhands browser count 'testid=chat-message-retry'` is still `1` after the history reloaded (`control-openhands browser screenshot 'testid=chat-interface' --feature F06.failed-send-persists --name retry-after-reopen`). Send the same text again (fill `Reply with only: retry-ok`, click `testid=submit-button` with the same `--observe`): the new bubble goes `Sending...` and settles while `Failed to send Retry Dismiss` stays; `control-openhands browser count 'testid=chat-message-sending'` is `0`, `control-openhands browser count 'testid=chat-message-retry'` is `1`, and the rows of `control-openhands conversation events <id> --kinds MessageEvent --grep 'retry-ok'` whose `source` is `user` number exactly one, `Reply with only: retry-ok` (in a model-free run `count` is `1`; with a model the agent's `retry-ok` reply matches the grep too, so count the `user` rows, not `count`). Then `control-openhands browser click 'testid=chat-message-retry'` and `control-openhands browser wait 'testid=chat-message-retry' --state detached --timeout 15000`: `control-openhands browser count 'testid=chat-message-sending'` is `0` and the same `--grep` now lists exactly two `user` rows `Reply with only: retry-ok` (one per attempt, none duplicated). With a model, `control-openhands conversation wait <id> --fresh --timeout 120` before the next bullet.", "raw": "**Failed send survives leaving (`F06.failed-send-persists`).** With `/conversations/<id>` open and healthy, run `control-openhands service stop agent-server`, `control-openhands browser wait 'testid=error-message-banner' --timeout 20000`, `control-openhands browser fill 'testid=chat-input' 'Reply with only: retry-ok'` and `control-openhands browser click 'testid=submit-button' --observe '[data-testid=chat-message-sending],[data-testid=chat-message-error]' --observe-ms 8000`: `Sending...` turns into `Failed to send Retry Dismiss` and `control-openhands browser count 'testid=chat-message-retry'` is `1` (`control-openhands browser screenshot 'testid=chat-interface' --feature F06.failed-send-persists --name failed-before-leaving`). Run `control-openhands restart` and `control-openhands browser wait 'testid=error-message-banner' --state hidden --timeout 90000` (the page reconnects by itself; `testid=error-message-banner-retry` hurries it while the banner is still up): the Retry row is still there (count `1`). Leave through the sidebar and come back: `control-openhands browser click 'role=link[name=\"New Chat\"]'`, `control-openhands browser wait 'testid=home-chat-launcher'`, `control-openhands browser click 'a[href*=\"/conversations/<id>\"] >> nth=0' --expect-url '/conversations/<id>'`, `control-openhands browser wait 'testid=user-message'`; `control-openhands browser count 'testid=chat-message-retry'` is still `1` after the history reloaded (`control-openhands browser screenshot 'testid=chat-interface' --feature F06.failed-send-persists --name retry-after-reopen`). Send the same text again (fill `Reply with only: retry-ok`, click `testid=submit-button` with the same `--observe`): the new bubble goes `Sending...` and settles while `Failed to send Retry Dismiss` stays; `control-openhands browser count 'testid=chat-message-sending'` is `0`, `control-openhands browser count 'testid=chat-message-retry'` is `1`, and the rows of `control-openhands conversation events <id> --kinds MessageEvent --grep 'retry-ok'` whose `source` is `user` number exactly one, `Reply with only: retry-ok` (in a model-free run `count` is `1`; with a model the agent's `retry-ok` reply matches the grep too, so count the `user` rows, not `count`). Then `control-openhands browser click 'testid=chat-message-retry'` and `control-openhands browser wait 'testid=chat-message-retry' --state detached --timeout 15000`: `control-openhands browser count 'testid=chat-message-sending'` is `0` and the same `--grep` now lists exactly two `user` rows `Reply with only: retry-ok` (one per attempt, none duplicated). With a model, `control-openhands conversation wait <id> --fresh --timeout 120` before the next bullet.", "source_line": 88, "expected": [ "a failed send keeps its Retry row after the backend returns and after leaving and reopening the conversation from the sidebar; sending the same text again succeeds without clearing it, and Retry then delivers that attempt exactly once." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--live-chip-stop-and-resume", "title": "Live chip, stop and resume", "ids": [ "F06.live-activity", "F06.stop-resume" ], "body": "Run `control-openhands conversation start --prompt \"Run exactly one terminal command: sleep 25 && echo qa-slept. Then reply with only its output.\"` (no `--wait`), `control-openhands browser wait 'testid=live-activity-chip' --timeout 30000` and `control-openhands browser text 'testid=live-activity-chip'`: first `Thinking`, then the action title (e.g. `Run sleep 25 then echo qa-slept`); `browser text 'testid=interactive-chat-box'` contains `Running`. Run `control-openhands browser click 'testid=stop-button'`: `control-openhands conversation status <id>` is `paused`, the composer reads `Stopped`, `count 'testid=play-button'` is `1` and the chip is gone. Stop interrupts the running `sleep` too: `control-openhands conversation events <id>` shows an `AgentErrorEvent` (`Tool call interrupted before completion. The conversation was paused.`) followed by an `InterruptEvent`, and `browser text 'testid=chat-scroll-container'` shows an `Agent error` row under the action title. Run `control-openhands browser click 'testid=play-button'`: status `running`, then `control-openhands conversation wait <id> --timeout 120` ends `finished` with a new agent `MessageEvent` (the model reports that the command was interrupted; `qa-slept` never prints).", "raw": "**Live chip, stop and resume (`F06.live-activity`, `F06.stop-resume`).** Run `control-openhands conversation start --prompt \"Run exactly one terminal command: sleep 25 && echo qa-slept. Then reply with only its output.\"` (no `--wait`), `control-openhands browser wait 'testid=live-activity-chip' --timeout 30000` and `control-openhands browser text 'testid=live-activity-chip'`: first `Thinking`, then the action title (e.g. `Run sleep 25 then echo qa-slept`); `browser text 'testid=interactive-chat-box'` contains `Running`. Run `control-openhands browser click 'testid=stop-button'`: `control-openhands conversation status <id>` is `paused`, the composer reads `Stopped`, `count 'testid=play-button'` is `1` and the chip is gone. Stop interrupts the running `sleep` too: `control-openhands conversation events <id>` shows an `AgentErrorEvent` (`Tool call interrupted before completion. The conversation was paused.`) followed by an `InterruptEvent`, and `browser text 'testid=chat-scroll-container'` shows an `Agent error` row under the action title. Run `control-openhands browser click 'testid=play-button'`: status `running`, then `control-openhands conversation wait <id> --timeout 120` ends `finished` with a new agent `MessageEvent` (the model reports that the command was interrupted; `qa-slept` never prints).", "source_line": 89, "expected": [ "while the agent runs, a chip above the composer names the current action (\"Thinking\", then the action title) and the composer status reads Running.", "Stop in the composer interrupts the agent, including a running tool call (status Stopped, play button, an Agent error row for the interrupted call); Play resumes it to completion." ], "notes": [ "Known issue [#16719](https://github.com/OpenHands/OpenHands/issues/16719) (`F06.live-activity`): Agent status always says \"thinking\" even when nothing is happening. Live prerequisite: The event/status trace and deterministic stalled or errored state from the original report; its trigger is unspecified.", "Known issue [#16582](https://github.com/OpenHands/OpenHands/issues/16582) (`F06.stop-resume`): After reconnect, a maximum-iteration error can fail to pause the agent through the legacy event handler. Live prerequisite: A maximum-iteration legacy error delivered after socket replacement, with a real model run reaching that limit.", "Known issue [OpenHands/software-agent-sdk#4990](https://github.com/OpenHands/software-agent-sdk/issues/4990) (`F06.stop-resume`): An ACP turn can keep executing tool calls while the UI already reports Paused. Live prerequisite: An authenticated Claude Code ACP turn executing separate marker-writing tools while Pause is pressed.", "Known issue [OpenHands/software-agent-sdk#4777](https://github.com/OpenHands/software-agent-sdk/issues/4777) (`F06.stop-resume`): Stop can still start a queued tool after it finishes waiting on a resource lock. Live prerequisite: A live batch with a second real tool waiting on a declared-resource lock when Stop is pressed.", "Known issue [OpenHands/software-agent-sdk#5969](https://github.com/OpenHands/software-agent-sdk/issues/5969) (`F06.stop-resume`): a conversation can report finished but reject Play with already-running, or delay a follow-up, while a slow event webhook is still pending. Live prerequisite: a genuinely successful model run on an owned backend with a bounded slow webhook receiver, then Play or send during that completion window." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--reload-mid-run", "title": "Reload mid-run", "ids": [ "F06.reload-mid-run" ], "body": "Run `control-openhands conversation start --prompt \"Run exactly one terminal command: sleep 15 && echo qa-reloaded. Then reply with only its output.\"`, wait for `testid=live-activity-chip`, then `control-openhands browser reload`. The composer still reads `Running` and the chip is back. After `control-openhands conversation wait <id> --timeout 120`: `testid=agent-message` is `qa-reloaded`, `count 'testid=user-message'` is `1`, `control-openhands browser eval \"document.querySelectorAll('[data-testid=chat-scroll-container] [data-testid=generic-event-message-title]').length\"` is `1`.", "raw": "**Reload mid-run (`F06.reload-mid-run`).** Run `control-openhands conversation start --prompt \"Run exactly one terminal command: sleep 15 && echo qa-reloaded. Then reply with only its output.\"`, wait for `testid=live-activity-chip`, then `control-openhands browser reload`. The composer still reads `Running` and the chip is back. After `control-openhands conversation wait <id> --timeout 120`: `testid=agent-message` is `qa-reloaded`, `count 'testid=user-message'` is `1`, `control-openhands browser eval \"document.querySelectorAll('[data-testid=chat-scroll-container] [data-testid=generic-event-message-title]').length\"` is `1`.", "source_line": 90, "expected": [ "reloading while the agent runs reattaches to the live run; it finishes without duplicated events." ], "notes": [ "Known issue [#15606](https://github.com/OpenHands/OpenHands/issues/15606) (`F06.reload-mid-run`): When re-loading ACP conversations, messages disappear or move. Live prerequisite: An authenticated Codex ACP session emitting intermediate messages while another window reloads history.", "Known issue [OpenHands/software-agent-sdk#4077](https://github.com/OpenHands/software-agent-sdk/issues/4077) (`F06.reload-mid-run`): Retried or late streaming deltas can duplicate live text; a stalled subscriber can delay Stop and finalization. Live prerequisite: A working stream interrupted/retried or delivered to a controlled stalled subscriber, with delta/state-order capture." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--older-history", "title": "Older history", "ids": [ "F06.load-older-history" ], "body": "Run `control-openhands conversation start --prompt \"Run these 16 terminal commands strictly one per tool call, sequentially (never combine them): echo qa1, echo qa2, echo qa3, echo qa4, echo qa5, echo qa6, echo qa7, echo qa8, echo qa9, echo qa10, echo qa11, echo qa12, echo qa13, echo qa14, echo qa15, echo qa16. Then reply DONE.\" --wait --timeout 400` (note `<history-id>`); `control-openhands conversation events <history-id> --last 500` reports a count above 50. Run `control-openhands browser network --clear`, `control-openhands browser reload`, then `control-openhands browser network --last 300`: two `GET /api/conversations/<history-id>/events/search` requests (newest page, then the older page). `control-openhands browser text 'testid=chat-scroll-container'` starts with the prompt and shows `16 actions completed` and `DONE`. The other trigger is the user scrolling to the top, driven without a model on the dummy profile from Preconditions (`qa-zeta` active; in a keyed run create and activate it with that same `llm set` command for this bullet, then re-run `control-openhands llm preset deepseek` and remove it with `control-openhands api DELETE /api/profiles/qa-zeta --write`, so later families still count two profiles; arrange, not proof). Run `control-openhands conversation start --prompt \"qa older 1\"` (no `--wait`; note `<older-id>`), then `for N in $(seq 2 30); do control-openhands conversation send <older-id> --prompt \"qa older $N\"; done`: every send is accepted while the agent sits in error, about 1.5 s apiece (42 s for the 29 here), and each refused message adds five events (a `MessageEvent`, three `ConversationStateUpdateEvent`s, a `ConversationErrorEvent`), so the conversation ends with about 150 events (152 here). `control-openhands conversation events <older-id> --last 500` reports `\"total\": 152`, `\"pages\": 2`, `\"more\": false` (both server pages read, see Gotchas); its first row is the `SystemPromptEvent` and the first `MessageEvent` is `qa older 1`, while `--last 25` reports `\"pages\": 1`, `\"more\": true` and starts at `qa older 26`. Run `control-openhands browser network --clear`, `control-openhands browser reload`, `control-openhands browser wait 'testid=chat-input'`, then `control-openhands browser network --filter 'events/search' --last 10`: two requests, `?limit=50&sort_order=TIMESTAMP_DESC` and one with `timestamp__lt=<ts of qa older 21>`, because the newest 50 events hold only ten bubbles, shorter than the viewport, so the automatic fetch ran once and then stopped. `control-openhands browser count 'testid=user-message'` is `20`, `control-openhands browser text 'testid=chat-scroll-container'` starts `qa older 11` and holds none of `qa older 1`…`qa older 10`, and `control-openhands browser bbox 'testid=chat-scroll-container'` has `scrollHeight` `1576` against `clientHeight` `814`: taller than the viewport, so the third page waits for the user. Run `control-openhands browser scroll 'testid=chat-scroll-container' --by -100000`: `control-openhands browser network --filter 'events/search' --last 5` gains a request with `timestamp__lt=<ts of qa older 11>`, `control-openhands browser count 'testid=user-message'` is `29` and the text starts `qa older 2` (that page held messages 2–10 plus the tail of the first message's events; not seen in two runs: if the first refusal ever logs six events instead of seven, the arithmetic gives count `30`, text `qa older 1` and a last page of only the `SystemPromptEvent`), and the view stays on `qa older 11`: `control-openhands browser eval \"document.querySelector('[data-testid=chat-scroll-container]').scrollTop\"` is `684`, the height of the prepended rows. Scroll to the top again the same way: a fourth request (`timestamp__lt=` the oldest loaded event), the count is `30`, the text starts `qa older 1` and `scrollTop` is `76` (`control-openhands browser screenshot 'testid=chat-interface' --feature F06.load-older-history --name scroll-top-loaded-qa-older-1`). A third scroll adds no request: the short page ended the paging. `control-openhands browser count 'testid=loading-older-events'` is `0` after each scroll because the `Fetching older messages…` row lasts a few tens of milliseconds on a local stack; to see it, scroll with `control-openhands browser scroll 'testid=chat-scroll-container' --by -100000 --observe 'testid=loading-older-events' --observe-ms 1500` instead (both scrolls were driven this way here, with the same counts), whose `observed` goes `<absent>` at `0` ms, `Fetching older messages…` at about 20 ms and `<absent>` again within about 40 to 120 ms (`116` ms on the first scroll, `39` on the second; `observedBy` `mutation`); the `timestamp__lt=` network rows remain the durable proof. Return to the 16-command conversation for the next bullet: `control-openhands browser goto /conversations/<history-id>` and `control-openhands browser wait 'testid=chat-input'`.", "raw": "**Older history (`F06.load-older-history`).** Run `control-openhands conversation start --prompt \"Run these 16 terminal commands strictly one per tool call, sequentially (never combine them): echo qa1, echo qa2, echo qa3, echo qa4, echo qa5, echo qa6, echo qa7, echo qa8, echo qa9, echo qa10, echo qa11, echo qa12, echo qa13, echo qa14, echo qa15, echo qa16. Then reply DONE.\" --wait --timeout 400` (note `<history-id>`); `control-openhands conversation events <history-id> --last 500` reports a count above 50. Run `control-openhands browser network --clear`, `control-openhands browser reload`, then `control-openhands browser network --last 300`: two `GET /api/conversations/<history-id>/events/search` requests (newest page, then the older page). `control-openhands browser text 'testid=chat-scroll-container'` starts with the prompt and shows `16 actions completed` and `DONE`. The other trigger is the user scrolling to the top, driven without a model on the dummy profile from Preconditions (`qa-zeta` active; in a keyed run create and activate it with that same `llm set` command for this bullet, then re-run `control-openhands llm preset deepseek` and remove it with `control-openhands api DELETE /api/profiles/qa-zeta --write`, so later families still count two profiles; arrange, not proof). Run `control-openhands conversation start --prompt \"qa older 1\"` (no `--wait`; note `<older-id>`), then `for N in $(seq 2 30); do control-openhands conversation send <older-id> --prompt \"qa older $N\"; done`: every send is accepted while the agent sits in error, about 1.5 s apiece (42 s for the 29 here), and each refused message adds five events (a `MessageEvent`, three `ConversationStateUpdateEvent`s, a `ConversationErrorEvent`), so the conversation ends with about 150 events (152 here). `control-openhands conversation events <older-id> --last 500` reports `\"total\": 152`, `\"pages\": 2`, `\"more\": false` (both server pages read, see Gotchas); its first row is the `SystemPromptEvent` and the first `MessageEvent` is `qa older 1`, while `--last 25` reports `\"pages\": 1`, `\"more\": true` and starts at `qa older 26`. Run `control-openhands browser network --clear`, `control-openhands browser reload`, `control-openhands browser wait 'testid=chat-input'`, then `control-openhands browser network --filter 'events/search' --last 10`: two requests, `?limit=50&sort_order=TIMESTAMP_DESC` and one with `timestamp__lt=<ts of qa older 21>`, because the newest 50 events hold only ten bubbles, shorter than the viewport, so the automatic fetch ran once and then stopped. `control-openhands browser count 'testid=user-message'` is `20`, `control-openhands browser text 'testid=chat-scroll-container'` starts `qa older 11` and holds none of `qa older 1`…`qa older 10`, and `control-openhands browser bbox 'testid=chat-scroll-container'` has `scrollHeight` `1576` against `clientHeight` `814`: taller than the viewport, so the third page waits for the user. Run `control-openhands browser scroll 'testid=chat-scroll-container' --by -100000`: `control-openhands browser network --filter 'events/search' --last 5` gains a request with `timestamp__lt=<ts of qa older 11>`, `control-openhands browser count 'testid=user-message'` is `29` and the text starts `qa older 2` (that page held messages 2–10 plus the tail of the first message's events; not seen in two runs: if the first refusal ever logs six events instead of seven, the arithmetic gives count `30`, text `qa older 1` and a last page of only the `SystemPromptEvent`), and the view stays on `qa older 11`: `control-openhands browser eval \"document.querySelector('[data-testid=chat-scroll-container]').scrollTop\"` is `684`, the height of the prepended rows. Scroll to the top again the same way: a fourth request (`timestamp__lt=` the oldest loaded event), the count is `30`, the text starts `qa older 1` and `scrollTop` is `76` (`control-openhands browser screenshot 'testid=chat-interface' --feature F06.load-older-history --name scroll-top-loaded-qa-older-1`). A third scroll adds no request: the short page ended the paging. `control-openhands browser count 'testid=loading-older-events'` is `0` after each scroll because the `Fetching older messages…` row lasts a few tens of milliseconds on a local stack; to see it, scroll with `control-openhands browser scroll 'testid=chat-scroll-container' --by -100000 --observe 'testid=loading-older-events' --observe-ms 1500` instead (both scrolls were driven this way here, with the same counts), whose `observed` goes `<absent>` at `0` ms, `Fetching older messages…` at about 20 ms and `<absent>` again within about 40 to 120 ms (`116` ms on the first scroll, `39` on the second; `observedBy` `mutation`); the `timestamp__lt=` network rows remain the durable proof. Return to the 16-command conversation for the next bullet: `control-openhands browser goto /conversations/<history-id>` and `control-openhands browser wait 'testid=chat-input'`.", "source_line": 91, "expected": [ "only the newest 50 events load first; the previous 50 are fetched when the user scrolls the transcript to its top (a \"Fetching older messages…\" row shows until they arrive) and automatically when the content is too short to scroll, so the first prompt and all actions appear." ], "notes": [ "Known issue [#17559](https://github.com/OpenHands/OpenHands/issues/17559) (`F06.load-older-history`): History pagination can show both an action and its observation (ready-for-dev companion to #16947). Live prerequisite: A real persisted history page boundary splitting an action/observation or ACP started/completed pair; dummy-key errors do not produce these tool pairs.", "Known issue [#16947](https://github.com/OpenHands/OpenHands/issues/16947) (`F06.load-older-history`): Loading older history can leave both the action and its newer observation visible because UI projection depends on arrival order. Live prerequisite: A real long conversation whose REST boundary splits an action/observation pair; keep companion #17559 linked separately.", "Known issue [#18445](https://github.com/OpenHands/OpenHands/issues/18445) (`F06.load-older-history`, `F06.reconnect`): a failed initial Cloud history search can be swallowed into an empty page while the runtime count succeeds, advancing `after_seq` past history that was never displayed. Live prerequisite: a Cloud account and owned conversation with a genuinely failing App API `events/search` and a successful runtime `events/count`. The Local failure path throws instead and is not an equivalent check." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--scroll-to-bottom", "title": "Scroll to bottom", "ids": [ "F06.scroll-to-bottom" ], "body": "On the 16-command conversation (`<history-id>`; in a model-free run this bullet is blocked with the automatic-fetch half, prerequisite `DEEPSEEK_API_KEY`), `control-openhands browser click 'testid=event-group-toggle'` makes the transcript taller than the viewport while the view stays where it was. Run `control-openhands browser scroll 'testid=chat-scroll-container' --by 2000` first (the button only reacts to scroll events, so a page that is already at the top never shows it): `count 'testid=scroll-to-bottom'` is `0`. Run `control-openhands browser scroll 'testid=chat-scroll-container' --by -2000`; the count is `1`. `control-openhands browser click 'testid=scroll-to-bottom'`, then the count is `0` and `control-openhands browser eval \"(()=>{const e=document.querySelector('[data-testid=chat-scroll-container]');return Math.round(e.scrollHeight-e.scrollTop-e.clientHeight)})()\"` is `0`.", "raw": "**Scroll to bottom (`F06.scroll-to-bottom`).** On the 16-command conversation (`<history-id>`; in a model-free run this bullet is blocked with the automatic-fetch half, prerequisite `DEEPSEEK_API_KEY`), `control-openhands browser click 'testid=event-group-toggle'` makes the transcript taller than the viewport while the view stays where it was. Run `control-openhands browser scroll 'testid=chat-scroll-container' --by 2000` first (the button only reacts to scroll events, so a page that is already at the top never shows it): `count 'testid=scroll-to-bottom'` is `0`. Run `control-openhands browser scroll 'testid=chat-scroll-container' --by -2000`; the count is `1`. `control-openhands browser click 'testid=scroll-to-bottom'`, then the count is `0` and `control-openhands browser eval \"(()=>{const e=document.querySelector('[data-testid=chat-scroll-container]');return Math.round(e.scrollHeight-e.scrollTop-e.clientHeight)})()\"` is `0`.", "source_line": 92, "expected": [ "scrolling up shows a scroll-to-bottom button; clicking it returns to the bottom and hides it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--confirmation-mode-on", "title": "Confirmation mode on", "ids": [ "F06.confirmation-mode" ], "body": "Arrange through the settings UI, in this order (the Security Analyzer field appears only once Confirmation Mode is on): `control-openhands browser goto /settings/verification`, `control-openhands browser click 'testid=verification-settings-screen >> text=Confirmation Mode'`, `control-openhands browser click 'testid=sdk-section-all-toggle'`, `control-openhands browser click 'testid=verification-settings-screen >> role=button[name=\"Show suggestions\"]'`, `control-openhands browser click 'role=listbox >> role=option[name=\"None\"]'`, `control-openhands browser value 'testid=verification-settings-screen >> role=combobox[name=\"Security Analyzer\"]'` (`None`), `control-openhands browser click 'testid=verification-settings-screen >> testid=save-button'`; `control-openhands api GET /api/settings` shows `\"confirmation_mode\": true` and `\"security_analyzer\": \"none\"` (None means every action is confirmed; the default LLM analyzer only stops HIGH-risk actions, so `echo` would run unprompted). Run `control-openhands conversation start --prompt \"Run the terminal command 'echo qa-confirm-ok' and reply with its output.\" --wait --timeout 180`: it returns `\"status\": \"waiting_for_confirmation\"`. `control-openhands browser text 'testid=chat-scroll-container'` ends `Do you want to continue with this action? Cancel Ctrl+⇧+⌫ Continue Ctrl+↩` on Linux (`Cancel ⇧⌘⌫ Continue ⌘↩` on macOS, #18127), `control-openhands browser tooltip 'testid=action-confirm-button'` is `Confirm the requested action`, and a lock icon sits above the composer (screenshot). The tooltip stays open over the Cancel button, so move the pointer first: `control-openhands browser hover 'testid=chat-input'`. Run `control-openhands browser click 'testid=action-reject-button'`: status `idle`, the buttons are gone, `conversation events <id> --last 6` ends with `UserRejectObservation`, and the action's row reads `Rejected` (`control-openhands browser count 'testid=rejected-indicator'` is `1`) with no spinner. Send `Run the terminal command 'echo qa-confirm-two' and reply with its output.` through `testid=chat-input` / `testid=submit-button`, `control-openhands browser wait 'testid=action-confirm-button' --timeout 20000`, `control-openhands browser click 'testid=action-confirm-button'`; about 15 s later the status is `finished` and the events contain the observation `qa-confirm-two`. A rejection inside a group resolves it too: send `Run the terminal command 'echo qa-g1'. After it finishes, run the terminal command 'echo qa-g2' as a separate tool call. Then reply DONE.`, continue the first action (`browser wait 'testid=action-confirm-button'`, then `control-openhands browser press Control+Enter`, which is also the next bullet's shortcut) and cancel the second (`browser wait 'testid=action-confirm-button' --timeout 60000`, `control-openhands browser press Control+Shift+Backspace`): status `idle`, `control-openhands browser text 'testid=event-group >> nth=-1'` ends `2 actions completed`, `control-openhands browser count 'testid=event-group >> nth=-1 >> .animate-spin'` is `0`, the same after `browser reload`, and the expanded group (`control-openhands browser click 'testid=event-group >> nth=-1 >> button >> nth=0'`) shows `Rejected` on the `qa-g2` row (`browser screenshot 'testid=chat-interface' --feature F06.confirmation-mode --name group-after-reload`). Each button keeps its label and hint on one line: while the prompt is waiting, `control-openhands browser eval \"[...document.querySelectorAll('[data-testid=action-reject-button],[data-testid=action-confirm-button]')].map(b=>b.offsetHeight+'/'+b.scrollHeight).join(' ')\"` is `26/26 26/26` at desktop, also with the side panel open (`testid=right-panel-toggle`). At phone width (`browser viewport phone`) the shortcut hints are hidden (no hardware keyboard), the buttons read `Cancel` and `Continue`, and `browser text 'testid=chat-scroll-container'` ends `Do you want to continue with this action? Cancel Continue` (screenshot the awaiting prompt at phone width).", "raw": "**Confirmation mode on (`F06.confirmation-mode`).** Arrange through the settings UI, in this order (the Security Analyzer field appears only once Confirmation Mode is on): `control-openhands browser goto /settings/verification`, `control-openhands browser click 'testid=verification-settings-screen >> text=Confirmation Mode'`, `control-openhands browser click 'testid=sdk-section-all-toggle'`, `control-openhands browser click 'testid=verification-settings-screen >> role=button[name=\"Show suggestions\"]'`, `control-openhands browser click 'role=listbox >> role=option[name=\"None\"]'`, `control-openhands browser value 'testid=verification-settings-screen >> role=combobox[name=\"Security Analyzer\"]'` (`None`), `control-openhands browser click 'testid=verification-settings-screen >> testid=save-button'`; `control-openhands api GET /api/settings` shows `\"confirmation_mode\": true` and `\"security_analyzer\": \"none\"` (None means every action is confirmed; the default LLM analyzer only stops HIGH-risk actions, so `echo` would run unprompted). Run `control-openhands conversation start --prompt \"Run the terminal command 'echo qa-confirm-ok' and reply with its output.\" --wait --timeout 180`: it returns `\"status\": \"waiting_for_confirmation\"`. `control-openhands browser text 'testid=chat-scroll-container'` ends `Do you want to continue with this action? Cancel Ctrl+⇧+⌫ Continue Ctrl+↩` on Linux (`Cancel ⇧⌘⌫ Continue ⌘↩` on macOS, #18127), `control-openhands browser tooltip 'testid=action-confirm-button'` is `Confirm the requested action`, and a lock icon sits above the composer (screenshot). The tooltip stays open over the Cancel button, so move the pointer first: `control-openhands browser hover 'testid=chat-input'`. Run `control-openhands browser click 'testid=action-reject-button'`: status `idle`, the buttons are gone, `conversation events <id> --last 6` ends with `UserRejectObservation`, and the action's row reads `Rejected` (`control-openhands browser count 'testid=rejected-indicator'` is `1`) with no spinner. Send `Run the terminal command 'echo qa-confirm-two' and reply with its output.` through `testid=chat-input` / `testid=submit-button`, `control-openhands browser wait 'testid=action-confirm-button' --timeout 20000`, `control-openhands browser click 'testid=action-confirm-button'`; about 15 s later the status is `finished` and the events contain the observation `qa-confirm-two`. A rejection inside a group resolves it too: send `Run the terminal command 'echo qa-g1'. After it finishes, run the terminal command 'echo qa-g2' as a separate tool call. Then reply DONE.`, continue the first action (`browser wait 'testid=action-confirm-button'`, then `control-openhands browser press Control+Enter`, which is also the next bullet's shortcut) and cancel the second (`browser wait 'testid=action-confirm-button' --timeout 60000`, `control-openhands browser press Control+Shift+Backspace`): status `idle`, `control-openhands browser text 'testid=event-group >> nth=-1'` ends `2 actions completed`, `control-openhands browser count 'testid=event-group >> nth=-1 >> .animate-spin'` is `0`, the same after `browser reload`, and the expanded group (`control-openhands browser click 'testid=event-group >> nth=-1 >> button >> nth=0'`) shows `Rejected` on the `qa-g2` row (`browser screenshot 'testid=chat-interface' --feature F06.confirmation-mode --name group-after-reload`). Each button keeps its label and hint on one line: while the prompt is waiting, `control-openhands browser eval \"[...document.querySelectorAll('[data-testid=action-reject-button],[data-testid=action-confirm-button]')].map(b=>b.offsetHeight+'/'+b.scrollHeight).join(' ')\"` is `26/26 26/26` at desktop, also with the side panel open (`testid=right-panel-toggle`). At phone width (`browser viewport phone`) the shortcut hints are hidden (no hardware keyboard), the buttons read `Cancel` and `Continue`, and `browser text 'testid=chat-scroll-container'` ends `Do you want to continue with this action? Cancel Continue` (screenshot the awaiting prompt at phone width).", "source_line": 93, "expected": [ "with confirmation mode on, a lock chip sits above the composer and each action waits for \"Do you want to continue with this action?\" with Cancel and Continue (hints `⇧⌘⌫` / `⌘↩` on macOS, `Ctrl+⇧+⌫` / `Ctrl+↩` elsewhere, hidden below 640 px); a cancelled action is marked Rejected and counts as resolved in its group." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5092](https://github.com/OpenHands/software-agent-sdk/issues/5092) (`F06.confirmation-mode`): A revocation message sent while waiting for confirmation can implicitly approve the pending action. Live prerequisite: A working model proposing a real confirmation-gated marker action, then a revocation user message.", "Known issue [OpenHands/software-agent-sdk#4532](https://github.com/OpenHands/software-agent-sdk/issues/4532) (`F06.confirmation-mode`): After restarting while awaiting confirmation, approval can execute the tool but the next model request omits its assistant call. Live prerequisite: A real confirmation-paused action persisted across backend restart, then approval and a working provider call." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--confirmation-shortcuts", "title": "Confirmation shortcuts", "ids": [ "F06.confirmation-shortcuts" ], "body": "The previous bullet resolved its action, so arrange a new one in the same conversation: send `Run the terminal command 'echo qa-confirm-three' and reply with its output.` through `testid=chat-input` / `testid=submit-button` and `control-openhands browser wait 'testid=action-confirm-button' --timeout 20000`. `control-openhands browser text 'testid=action-confirm-button'` is `Continue Ctrl+↩` off macOS. Ctrl works off macOS (#17928, fixed by #18127): the previous bullet's `Control+Enter` continued and `Control+Shift+Backspace` cancelled. `control-openhands browser press Meta+Enter` continues too: `control-openhands conversation wait <id> --fresh --timeout 120` ends `finished` and `control-openhands conversation events <id> --kinds ObservationEvent` contains `qa-confirm-three`. Restore afterwards: `browser goto /settings/verification`, click `testid=sdk-section-all-toggle`, `Show suggestions`, `role=listbox >> role=option[name=\"LLM\"]`, then `testid=verification-settings-screen >> text=Confirmation Mode`, then `save-button`; `api GET /api/settings` shows `\"confirmation_mode\": false`, `\"security_analyzer\": \"llm\"`.", "raw": "**Confirmation shortcuts (`F06.confirmation-shortcuts`).** The previous bullet resolved its action, so arrange a new one in the same conversation: send `Run the terminal command 'echo qa-confirm-three' and reply with its output.` through `testid=chat-input` / `testid=submit-button` and `control-openhands browser wait 'testid=action-confirm-button' --timeout 20000`. `control-openhands browser text 'testid=action-confirm-button'` is `Continue Ctrl+↩` off macOS. Ctrl works off macOS (#17928, fixed by #18127): the previous bullet's `Control+Enter` continued and `Control+Shift+Backspace` cancelled. `control-openhands browser press Meta+Enter` continues too: `control-openhands conversation wait <id> --fresh --timeout 120` ends `finished` and `control-openhands conversation events <id> --kinds ObservationEvent` contains `qa-confirm-three`. Restore afterwards: `browser goto /settings/verification`, click `testid=sdk-section-all-toggle`, `Show suggestions`, `role=listbox >> role=option[name=\"LLM\"]`, then `testid=verification-settings-screen >> text=Confirmation Mode`, then `save-button`; `api GET /api/settings` shows `\"confirmation_mode\": false`, `\"security_analyzer\": \"llm\"`.", "source_line": 94, "expected": [ "Cmd+Enter or Ctrl+Enter continues and Shift+Cmd+Backspace or Shift+Ctrl+Backspace cancels a pending action." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--branch-from-here", "title": "Branch from here", "ids": [ "F06.branch-from-here" ], "body": "Open a conversation with at least two exchanges (the OK/PONG one above): `control-openhands browser goto /conversations/<id>` with its id (the previous bullet left `/settings/verification`). Agent message: `control-openhands browser hover 'testid=agent-message >> nth=0'`, `control-openhands browser click 'testid=agent-message >> nth=0 >> role=button[name=\"Branch from here\"]' --expect-url '/conversations/(?!<id>)'`; after `control-openhands browser wait 'testid=user-message'` (the history mounts a moment after the URL changes, so an immediate count reads `0`) the new conversation has `count 'testid=user-message'` `1`, the agent message `OK`, and `browser eval \"document.title\"` contains `(branch)`. User message (edit): back on the OK/PONG conversation (`control-openhands browser goto /conversations/<id>`; the branch above has one user message), `control-openhands browser hover 'testid=user-message >> nth=1'` and `control-openhands browser click 'testid=user-message >> nth=1 >> role=button[name=\"Branch from here\"]' --observe '[data-testid=chat-input]' --observe-ms 4000`: `observed` shows the composer change from empty to `Reply with only the word PONG.` and stay. The new conversation (`<branch-id>` from `browser url`) omits that message, `control-openhands browser text 'testid=chat-input'` is `Reply with only the word PONG.`, and nothing was sent: `control-openhands conversation events <branch-id> --kinds MessageEvent` lists only the copied first exchange (count `2`: the long message that starts `Reply with only the word OK.`, then `OK`; no `PONG`). `control-openhands browser enabled 'testid=submit-button'` should be `true` with the prefilled text; today it is `false` until the text is edited or the page is reloaded (see Gotchas). Send it with `control-openhands browser press Enter --selector 'testid=chat-input'` (Enter sends whatever the Send button's state is) and `control-openhands conversation wait <branch-id> --fresh --timeout 120`: expected `finished` with a reply; today it ends `error` (Agent Server bug, see Gotchas).", "raw": "**Branch from here (`F06.branch-from-here`).** Open a conversation with at least two exchanges (the OK/PONG one above): `control-openhands browser goto /conversations/<id>` with its id (the previous bullet left `/settings/verification`). Agent message: `control-openhands browser hover 'testid=agent-message >> nth=0'`, `control-openhands browser click 'testid=agent-message >> nth=0 >> role=button[name=\"Branch from here\"]' --expect-url '/conversations/(?!<id>)'`; after `control-openhands browser wait 'testid=user-message'` (the history mounts a moment after the URL changes, so an immediate count reads `0`) the new conversation has `count 'testid=user-message'` `1`, the agent message `OK`, and `browser eval \"document.title\"` contains `(branch)`. User message (edit): back on the OK/PONG conversation (`control-openhands browser goto /conversations/<id>`; the branch above has one user message), `control-openhands browser hover 'testid=user-message >> nth=1'` and `control-openhands browser click 'testid=user-message >> nth=1 >> role=button[name=\"Branch from here\"]' --observe '[data-testid=chat-input]' --observe-ms 4000`: `observed` shows the composer change from empty to `Reply with only the word PONG.` and stay. The new conversation (`<branch-id>` from `browser url`) omits that message, `control-openhands browser text 'testid=chat-input'` is `Reply with only the word PONG.`, and nothing was sent: `control-openhands conversation events <branch-id> --kinds MessageEvent` lists only the copied first exchange (count `2`: the long message that starts `Reply with only the word OK.`, then `OK`; no `PONG`). `control-openhands browser enabled 'testid=submit-button'` should be `true` with the prefilled text; today it is `false` until the text is edited or the page is reloaded (see Gotchas). Send it with `control-openhands browser press Enter --selector 'testid=chat-input'` (Enter sends whatever the Send button's state is) and `control-openhands conversation wait <branch-id> --fresh --timeout 120`: expected `finished` with a reply; today it ends `error` (Agent Server bug, see Gotchas).", "source_line": 95, "expected": [ "hovering a message shows Branch from here; on an agent message it forks inclusively; on a user message it forks without that message and pre-fills its text in the new composer, ready to send. The branch is titled \"<title> (branch)\" and keeps working." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5465](https://github.com/OpenHands/software-agent-sdk/issues/5465) (`F06.branch-from-here`): Branch from here can lose encrypted LLM credentials, so the fork fails its first turn with LLMAuthenticationError. Live prerequisite: A working cipher-backed model conversation, then fork and a successful turn on the fork.", "Known issue [OpenHands/software-agent-sdk#5806](https://github.com/OpenHands/software-agent-sdk/issues/5806) (`F06.branch-from-here`): editing a displayed user message first reads its parent event; if that event has become unreadable/missing on the server, the action can fail with 500 instead of a missing-event response. Live prerequisite: an owned persisted conversation with a still-displayed user message whose event storage is missing, followed by the actual Edit action. A random unknown-ID API call proves neither this UI path nor the batch-get contract, which Canvas does not call." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--images", "title": "Images", "ids": [ "F06.image-attachments" ], "body": "Run `control-openhands fixture image --name qa-f06-img` (prints the PNG path), `control-openhands browser upload 'testid=upload-image-input' <png path>` (a thumbnail appears in the composer), fill `testid=chat-input` with `Reply with only the word SEEN.` and click `testid=submit-button`. The user bubble contains `image-carousel` / `image-preview`. Run `control-openhands browser hover 'testid=image-preview'`, `control-openhands browser click 'testid=expand-image-button'` (`count 'testid=image-lightbox'` is `1`), `control-openhands browser press Escape` (`0`), click `expand-image-button` again and `control-openhands browser click 'testid=image-lightbox-close'` (`0`).", "raw": "**Images (`F06.image-attachments`).** Run `control-openhands fixture image --name qa-f06-img` (prints the PNG path), `control-openhands browser upload 'testid=upload-image-input' <png path>` (a thumbnail appears in the composer), fill `testid=chat-input` with `Reply with only the word SEEN.` and click `testid=submit-button`. The user bubble contains `image-carousel` / `image-preview`. Run `control-openhands browser hover 'testid=image-preview'`, `control-openhands browser click 'testid=expand-image-button'` (`count 'testid=image-lightbox'` is `1`), `control-openhands browser press Escape` (`0`), click `expand-image-button` again and `control-openhands browser click 'testid=image-lightbox-close'` (`0`).", "source_line": 96, "expected": [ "images sent with a message show as thumbnails in the bubble; expand opens a lightbox that closes with X or Escape." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--error-events", "title": "Error events", "ids": [ "F06.error-events" ], "body": "Any run that ends in a `ConversationErrorEvent` (today: any message sent in a branched conversation) leaves `conversation wait <id>` at `\"status\": \"error\"`; `control-openhands browser testids 'testid=chat-interface'` lists `error-message-banner` with `warning-message-banner-icon` and the error's text, and the composer status reads `Error`. Read the code with `control-openhands api GET \"/api/conversations/<id>/events/search?limit=3&sort_order=TIMESTAMP_DESC\"`.", "raw": "**Error events (`F06.error-events`).** Any run that ends in a `ConversationErrorEvent` (today: any message sent in a branched conversation) leaves `conversation wait <id>` at `\"status\": \"error\"`; `control-openhands browser testids 'testid=chat-interface'` lists `error-message-banner` with `warning-message-banner-icon` and the error's text, and the composer status reads `Error`. Read the code with `control-openhands api GET \"/api/conversations/<id>/events/search?limit=3&sort_order=TIMESTAMP_DESC\"`.", "source_line": 97, "expected": [ "a conversation error (ConversationErrorEvent) surfaces as a warning banner and the composer status Error." ], "notes": [ "Known issue [#16686](https://github.com/OpenHands/OpenHands/issues/16686) (`F06.error-events`): LLM errors not propagated to the frontend. Live prerequisite: A Nebius credential and its rejected DeepSeek-V3 request, or the original complete emitted error fixture.", "Known issue [#15626](https://github.com/OpenHands/OpenHands/issues/15626) (`F06.error-events`): Conversation can fail with LiteLLM auth error when database server at 127.0.0.1:5432 is unreachable. Live prerequisite: The original LiteLLM proxy/database-failure event or a disposable proxy with its PostgreSQL dependency unavailable.", "Known issue [OpenHands/software-agent-sdk#5447](https://github.com/OpenHands/software-agent-sdk/issues/5447) (`F06.error-events`): An Anthropic Claude Opus5.5 conversation can fail with400 when its next request ends in an assistant prefill. Live prerequisite: Anthropic Claude Opus5.5 credentials and a real trailing-assistant-prefill request.", "Known issue [OpenHands/software-agent-sdk#5225](https://github.com/OpenHands/software-agent-sdk/issues/5225) (`F06.error-events`): A provider-rejected image tool result can leave the conversation permanently failing on every resume. Live prerequisite: A real provider-rejected poisoned image tool_result in a disposable persisted history, followed by retry.", "Known issue [OpenHands/software-agent-sdk#4965](https://github.com/OpenHands/software-agent-sdk/issues/4965) (`F06.error-events`): A strict OpenAI-compatible provider can reject blank content in the saved history, preventing later turns. Live prerequisite: A strict OpenAI-compatible provider and a real history containing blank text or empty content arrays.", "Known issue [OpenHands/software-agent-sdk#4558](https://github.com/OpenHands/software-agent-sdk/issues/4558) (`F06.error-events`): A saved encrypted fallback profile can send ciphertext as its key and fail authentication, leaving only the primary error visible. Live prerequisite: A working encrypted saved fallback profile and real primary-model failure that triggers fallback resolution.", "Known issue [OpenHands/software-agent-sdk#5740](https://github.com/OpenHands/software-agent-sdk/issues/5740) (`F06.error-events`): a Responses display delta without `item_id` can terminate the turn with an AttributeError. Live prerequisite: the genuine provider/transformer streaming path producing that incomplete event; the report did not retain its original raw SSE. A credential-free parser diagnostic proves the SDK exception, not a Canvas model turn." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F06--skill-installed-banner", "title": "Skill installed banner", "ids": [ "F06.skill-install-banner" ], "body": "The banner keys on the installer's success line in terminal output. Put this one line in a prompt file: `Run exactly this one terminal command, verbatim, then reply DONE: mkdir -p .agents/skills/qa-skill && printf '# qa-skill\\n' > .agents/skills/qa-skill/SKILL.md && echo \"✅ Successfully installed 'qa-skill' to $PWD/.agents/skills/qa-skill\"`, then run `control-openhands conversation start --prompt \"$(cat <prompt file>)\" --wait --timeout 150`. `control-openhands browser text 'testid=skill-install-restart-banner'` reads `Installed to this workspace: qa-skill. Skills load when a conversation starts, so this conversation can't use them yet.` Run `control-openhands browser click 'testid=skill-install-restart-action' --expect-url '/conversations/(?!<id>)'`; `control-openhands api GET /api/conversations/<new id>` has the same `workspace.working_dir` as the source. Back on the source (`control-openhands browser goto /conversations/<id>`), `control-openhands browser click 'testid=skill-install-restart-dismiss'` hides it (count `0`); after `browser reload` it is back (dismissal is session-only by design).", "raw": "**Skill installed banner (`F06.skill-install-banner`).** The banner keys on the installer's success line in terminal output. Put this one line in a prompt file: `Run exactly this one terminal command, verbatim, then reply DONE: mkdir -p .agents/skills/qa-skill && printf '# qa-skill\\n' > .agents/skills/qa-skill/SKILL.md && echo \"✅ Successfully installed 'qa-skill' to $PWD/.agents/skills/qa-skill\"`, then run `control-openhands conversation start --prompt \"$(cat <prompt file>)\" --wait --timeout 150`. `control-openhands browser text 'testid=skill-install-restart-banner'` reads `Installed to this workspace: qa-skill. Skills load when a conversation starts, so this conversation can't use them yet.` Run `control-openhands browser click 'testid=skill-install-restart-action' --expect-url '/conversations/(?!<id>)'`; `control-openhands api GET /api/conversations/<new id>` has the same `workspace.working_dir` as the source. Back on the source (`control-openhands browser goto /conversations/<id>`), `control-openhands browser click 'testid=skill-install-restart-dismiss'` hides it (count `0`); after `browser reload` it is back (dismissal is session-only by design).", "source_line": 98, "expected": [ "after the agent installs a skill into the workspace, a banner offers \"Start new conversation with this skill\" (same workspace) and Close." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--no-llm", "title": "No LLM", "ids": [ "F06.llm-not-configured-banner" ], "body": "On the fresh no-LLM run: `control-openhands onboard --skip`, `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=launch-no-workspace' --expect-url '/conversations/'`. `control-openhands browser text 'testid=home-llm-not-configured-banner'` reads `Your LLM isn't set up yet, so conversations won't run. Finish setup to get started. Set up LLM`; `count 'testid=chat-suggestions'` is `0`; the composer cannot take text (`control-openhands browser attr 'testid=chat-input' contenteditable` is `false`, so `browser fill` fails) and `control-openhands browser enabled 'testid=submit-button'` is `false`. `control-openhands browser click 'testid=home-llm-not-configured-action'` then `browser url` ends in `/settings/llm`.", "raw": "**No LLM (`F06.llm-not-configured-banner`).** On the fresh no-LLM run: `control-openhands onboard --skip`, `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=launch-no-workspace' --expect-url '/conversations/'`. `control-openhands browser text 'testid=home-llm-not-configured-banner'` reads `Your LLM isn't set up yet, so conversations won't run. Finish setup to get started. Set up LLM`; `count 'testid=chat-suggestions'` is `0`; the composer cannot take text (`control-openhands browser attr 'testid=chat-input' contenteditable` is `false`, so `browser fill` fails) and `control-openhands browser enabled 'testid=submit-button'` is `false`. `control-openhands browser click 'testid=home-llm-not-configured-action'` then `browser url` ends in `/settings/llm`.", "source_line": 99, "expected": [ "with no usable LLM, a conversation shows \"Your LLM isn't set up yet...\" with Set up LLM; suggestions are hidden and sending is disabled." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--critic-f06-critic-result-blocked", "title": "Critic (`F06.critic-result`, blocked)", "ids": [ "F06.critic-result" ], "body": "Toggle the hidden switch through its label on `/settings/verification`: `control-openhands browser goto /settings/verification`, `control-openhands browser click 'label:has([data-testid=\"sdk-settings-verification.critic_enabled\"])'`, save, `api GET /api/settings` shows `\"critic_enabled\": true`. Without a Critic API key a `Reply with only the word OK.` conversation renders no critic block. With a key and a model, run `control-openhands conversation start --prompt \"Create qa_critic.txt containing hi, then finish.\" --wait --timeout 180`: every scored event carries a block, so `control-openhands browser count 'text=Critic: agent success likelihood'` is at least `1` and a finishing action (`finish` tool row) shows its own block next to the agent's message; the stars carry the score as their label, which an ARIA snapshot does not list: `control-openhands browser count '[aria-label^=\"Score: \"]'` is at least `2` and `control-openhands browser attr '[aria-label^=\"Score: \"] >> nth=0' aria-label` reads `Score: <N.N>%` with one decimal (`Score: 82.0%`), the same number as the `(82.0%)` text beside the stars. `control-openhands browser click 'role=button[name=\"Expand details\"] >> nth=0'` opens the categories `Potential Issues:`, `Infrastructure:` and `Likely Follow-up:` with their named issues in `control-openhands browser text 'testid=chat-scroll-container'`, and the button's label flips to `Collapse details`. Blocked here: DEEPSEEK_API_KEY and a Cloud critic key. Turn it off again the same way.", "raw": "**Critic (`F06.critic-result`, blocked).** Toggle the hidden switch through its label on `/settings/verification`: `control-openhands browser goto /settings/verification`, `control-openhands browser click 'label:has([data-testid=\"sdk-settings-verification.critic_enabled\"])'`, save, `api GET /api/settings` shows `\"critic_enabled\": true`. Without a Critic API key a `Reply with only the word OK.` conversation renders no critic block. With a key and a model, run `control-openhands conversation start --prompt \"Create qa_critic.txt containing hi, then finish.\" --wait --timeout 180`: every scored event carries a block, so `control-openhands browser count 'text=Critic: agent success likelihood'` is at least `1` and a finishing action (`finish` tool row) shows its own block next to the agent's message; the stars carry the score as their label, which an ARIA snapshot does not list: `control-openhands browser count '[aria-label^=\"Score: \"]'` is at least `2` and `control-openhands browser attr '[aria-label^=\"Score: \"] >> nth=0' aria-label` reads `Score: <N.N>%` with one decimal (`Score: 82.0%`), the same number as the `(82.0%)` text beside the stars. `control-openhands browser click 'role=button[name=\"Expand details\"] >> nth=0'` opens the categories `Potential Issues:`, `Infrastructure:` and `Likely Follow-up:` with their named issues in `control-openhands browser text 'testid=chat-scroll-container'`, and the button's label flips to `Collapse details`. Blocked here: DEEPSEEK_API_KEY and a Cloud critic key. Turn it off again the same way.", "source_line": 100, "expected": [ "with the critic enabled, agent messages and the finishing action each carry a \"Critic: agent success likelihood\" block (stars labelled `Score: <N.N>%`, one decimal, with the percentage beside them); Expand details lists the issue categories (Potential Issues, Infrastructure, Likely Follow-up)." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F06--phone", "title": "Phone", "ids": [ "F06.phone" ], "body": "Run `control-openhands browser viewport phone`, open the table conversation (`control-openhands browser goto /conversations/<stream id>`), `control-openhands browser bbox 'testid=chat-interface'` (`insideViewport` `true`, `pageHorizontalOverflow` `false`) and `control-openhands browser bbox 'testid=markdown-table-scroll'` (`scrollWidth` larger than `clientWidth`: the table scrolls inside itself). `control-openhands browser screenshot --feature F06.phone --name conversation`, then `control-openhands browser viewport desktop`.", "raw": "**Phone (`F06.phone`).** Run `control-openhands browser viewport phone`, open the table conversation (`control-openhands browser goto /conversations/<stream id>`), `control-openhands browser bbox 'testid=chat-interface'` (`insideViewport` `true`, `pageHorizontalOverflow` `false`) and `control-openhands browser bbox 'testid=markdown-table-scroll'` (`scrollWidth` larger than `clientWidth`: the table scrolls inside itself). `control-openhands browser screenshot --feature F06.phone --name conversation`, then `control-openhands browser viewport desktop`.", "source_line": 101, "expected": [ "at 390 px the transcript fits the viewport; wide tables scroll inside their own container." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F06--after-each-group", "title": "After each group", "ids": [], "body": "After each group run `control-openhands browser errors --app-only`. Outages driven with `service stop` log connection-refused console errors by design; `pageErrors` must stay `0`.", "raw": "After each group run `control-openhands browser errors --app-only`. Outages driven with `service stop` log connection-refused console errors by design; `pageErrors` must stay `0`.", "source_line": 102, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F06--planner-discovery-with-many-children", "title": "Planner discovery with many children", "ids": [ "F06.tool-visualizers" ], "body": "Expected: reloading a parent still discovers its `plannerparent`-tagged planner when it has100 child conversations. Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5790](https://github.com/OpenHands/software-agent-sdk/issues/5790)): the unchunked100-ID request returns500, shows `Internal Server Error` and omits the planner session connection. A99-child control returns200 and opens both planner and parent sockets. These are genuine idle API-created children, without a model run or browser-storage changes; a new parent ID has no UI-created planner hint.\n ```sh known-bug F06.tool-visualizers planner-batch-100\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n workspace = fixture / \"workspace\"\n workspace.mkdir()\n parent = create(workspace)\n for index in range(100):\n create(workspace, parent, {\"plannerparent\":parent} if index == 0 else None)\n print(parent)\n PY\n )\"\n control-openhands assert --field value --equals 100 -- api GET \"/api/conversations/$QA_ID\" --pick sub_conversation_ids.length\n QA_PLANNER=\"$(control-openhands api GET \"/api/conversations/$QA_ID\" --pick sub_conversation_ids.0 | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"value\"])')\"\n control-openhands browser network --clear\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser wait 'testid=chat-input'\n control-openhands browser screenshot --feature F06.tool-visualizers --name planner-batch-100\n control-openhands assert --field recent.0.status --equals 200 --bug-shows 500 -- browser network --filter \"ids=$QA_PLANNER\" # bug\n ```", "raw": "**Planner discovery with many children (`F06.tool-visualizers`).** Expected: reloading a parent still discovers its `plannerparent`-tagged planner when it has100 child conversations. Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5790](https://github.com/OpenHands/software-agent-sdk/issues/5790)): the unchunked100-ID request returns500, shows `Internal Server Error` and omits the planner session connection. A99-child control returns200 and opens both planner and parent sockets. These are genuine idle API-created children, without a model run or browser-storage changes; a new parent ID has no UI-created planner hint.\n ```sh known-bug F06.tool-visualizers planner-batch-100\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n workspace = fixture / \"workspace\"\n workspace.mkdir()\n parent = create(workspace)\n for index in range(100):\n create(workspace, parent, {\"plannerparent\":parent} if index == 0 else None)\n print(parent)\n PY\n )\"\n control-openhands assert --field value --equals 100 -- api GET \"/api/conversations/$QA_ID\" --pick sub_conversation_ids.length\n QA_PLANNER=\"$(control-openhands api GET \"/api/conversations/$QA_ID\" --pick sub_conversation_ids.0 | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"value\"])')\"\n control-openhands browser network --clear\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser wait 'testid=chat-input'\n control-openhands browser screenshot --feature F06.tool-visualizers --name planner-batch-100\n control-openhands assert --field recent.0.status --equals 200 --bug-shows 500 -- browser network --filter \"ids=$QA_PLANNER\" # bug\n ```", "source_line": 104, "expected": [ "expanding a tool row shows a rich body: terminal command and output (non-zero exit badge `exit N`), file-editor path chip and content." ], "notes": [ "Known issue [#17492](https://github.com/OpenHands/OpenHands/issues/17492) (`F06.tool-visualizers`): Failed browser, glob, grep, and file editor tool calls render as successful in chat. Live prerequisite: A working model emitting failed browser, glob, grep, planning-editor and file-editor observations with is_error=true.", "Known issue [#17490](https://github.com/OpenHands/OpenHands/issues/17490) (`F06.tool-visualizers`): Terminal commands that fail with an ordinary non-zero exit render as successful in chat. Live prerequisite: A working model producing a completed terminal command with nonzero exit_code and is_error=false.", "Known issue [#16341](https://github.com/OpenHands/OpenHands/issues/16341) (`F06.tool-visualizers`): Code-explorer subagent frequently errors. Live prerequisite: A working model with native code-explorer delegation and the original repository task; no delegated-tool output can be generated without credentials.", "Known issue [OpenHands/software-agent-sdk#5108](https://github.com/OpenHands/software-agent-sdk/issues/5108) (`F06.tool-visualizers`): The native SDK Gemini editor can report success while corrupting unrelated invalid-UTF8 bytes in the file. Live prerequisite: An Agent Server configured with the native Gemini edit tool set and a real tool edit of an invalid-UTF8 fixture; Gemini ACP uses a different external executor.", "Known issue [OpenHands/software-agent-sdk#5096](https://github.com/OpenHands/software-agent-sdk/issues/5096) (`F06.tool-visualizers`): The canonical agent file editor can replace a symlink instead of editing its target; Undo does not restore the link. Live prerequisite: A working model invoking canonical file-editor str_replace/insert through a real symlink, then undo.", "Known issue [OpenHands/software-agent-sdk#5074](https://github.com/OpenHands/software-agent-sdk/issues/5074) (`F06.tool-visualizers`): After server restart a native delegated child can remain Running while the parent loses its task mapping/result. Live prerequisite: A native delegated child still running during an owned server crash and parent recovery.", "Known issue [OpenHands/software-agent-sdk#5038](https://github.com/OpenHands/software-agent-sdk/issues/5038) (`F06.tool-visualizers`): The agent file editor can refuse valid Chinese UTF8 text as binary when its sample splits a character; Canvas Files uses a separate preview detector. Live prerequisite: A live canonical file-editor view/edit on the specific UTF8 sample alignment; Canvas Files preview uses a separate NUL sniff, not this validator.", "Known issue [OpenHands/software-agent-sdk#4983](https://github.com/OpenHands/software-agent-sdk/issues/4983) (`F06.tool-visualizers`): Agent grep can silently miss extended-regex matches when its runtime falls back to system grep. Live prerequisite: A runtime without ripgrep and a live agent grep call using extended regex and escaped literal punctuation.", "Known issue [OpenHands/software-agent-sdk#4668](https://github.com/OpenHands/software-agent-sdk/issues/4668) (`F06.tool-visualizers`): A planning editor card can lose old/new content and fail to show the complete edit diff. Live prerequisite: A live planner create/edit yielding PlanningFileEditorObservation with old/new contents for the chat diff.", "Known issue [OpenHands/software-agent-sdk#4663](https://github.com/OpenHands/software-agent-sdk/issues/4663) (`F06.tool-visualizers`): Parallel Python-fallback glob calls can change process cwd and return files from the wrong workspace. Live prerequisite: Concurrent real glob/tool calls in distinct workspaces with the Python fallback active.", "Known issue [OpenHands/software-agent-sdk#4661](https://github.com/OpenHands/software-agent-sdk/issues/4661) (`F06.tool-visualizers`): Same-name project subagents can resolve to another conversation workspace definition. Live prerequisite: Two native task-enabled conversations with same-name project subagent definitions and a working model for delegation.", "Known issue [OpenHands/software-agent-sdk#4583](https://github.com/OpenHands/software-agent-sdk/issues/4583) (`F06.tool-visualizers`): An agent editor insertion at EOF can join onto the last line when the file has no trailing newline. Live prerequisite: A working model invoking file-editor insert at EOF on a file with no trailing newline.", "Known issue [OpenHands/software-agent-sdk#4540](https://github.com/OpenHands/software-agent-sdk/issues/4540) (`F06.tool-visualizers`): Synthetic GLM/Kimi tool calls can render as raw tool_call syntax instead of executing. Live prerequisite: Synthetic provider credentials and a GLM/Kimi turn emitting the raw tool_call wrapper." ], "annotations": [ "Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5790](https://github.com/OpenHands/software-agent-sdk/issues/5790)): the unchunked100-ID request returns500, shows `Internal Server Error` and omits the planner session connection. A99-child control returns200 and opens both planner and parent sockets. These are genuine idle API-created children, without a model run or browser-storage changes; a new parent ID has no UI-created planner hint." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F06--contract-f06-hook-events", "title": "Hook events", "ids": [ "F06.hook-events" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "hook executions render as their own rows in the stream." ], "notes": [], "annotations": [], "states": [ "contract" ] }, { "name": "F06--contract-f06-corrective-nudge", "title": "Corrective nudge", "ids": [ "F06.corrective-nudge" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "when the model answers with neither a message nor a tool call, the SDK's nudge (`Your last response did not include a function call or a message. …`) shows as a muted, italic `role=note` line with an info icon (`corrective-nudge-message`), not as a user bubble (#17864)." ], "notes": [], "annotations": [], "states": [ "contract" ] } ] }, { "id": "F07", "title": "Conversation page, header and management menu", "file": "F07-conversation-page.md", "sha256": "06f2f175260fbdfb29137ebc6881868279ceef74b6cd63f39a72ae50618849a8", "overview": "The conversation page (`/conversations/<id>`) shows the chat pane: a header with\nthe runtime status dot, the editable title, the \"...\" management menu and the\nGit actions, Overview and panel toggles, then the message history and the\ncomposer. From the header a user renames, stops or resumes the runtime, opens\nthe skills, hooks and agent-tools dialogs, exports or downloads the\nconversation, jumps to usage and deletes it.", "entry": "- Sidebar conversation row (`conversation-card`), or the home composer submit, which navigates to the new conversation.\n- Direct URL `/conversations/<id>`.\n- Direct URL `/conversations/<id>/panel`, or the header panel button, at phone width (up to 1024 px); at desktop width that URL redirects to `/conversations/<id>` with the drawer open.\n- Hovering a message in the history: **Branch from here** (local backends only).\n- Chips under the composer (git control bar) for conversations in a git workspace (home **Open Workspace**).\n- Header controls inside `testid=chat-pane-header`: the status dot (`server-status-menu-trigger`), the title (`conversation-name-title`), the \"...\" button (`ellipsis-button`, aria \"More options\"), `conversation-git-actions-toggle`, `conversation-overview-toggle`, `right-panel-toggle`.", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done) and an active LLM profile (`control-openhands llm preset deepseek`).\n- A finished conversation: `control-openhands conversation start --prompt \"Reply with the single word: pong\" --wait --timeout 240` prints its `<id>`.\n- A running conversation for the active-only items: `control-openhands conversation start --prompt \"Run the shell command: sleep 90 ; then reply with the word done\" --wait --until running --timeout 120` prints `<running-id>`. It finishes after about 90 s, so start it right before `F07.stop-confirm` (after a Stop it stays `paused` until Start Runtime).\n- A conversation in a git workspace for the git control bar: `control-openhands fixture git-repo --name qa-f07-repo`, then `control-openhands browser goto /` (`conversation start` left the browser on `<id>`), `control-openhands browser click 'testid=open-workspace-button'`, `control-openhands browser click 'role=dialog >> role=combobox[name=\"Select a workspace\"]'`, `control-openhands browser click 'role=dialog >> role=button[name=\"+ Add Workspace\"]'`, `control-openhands browser click 'testid=folder-browser-sidebar-workspace'`, `control-openhands browser click 'testid=folder-browser-up'` three times (to the run directory), `control-openhands browser click 'testid=folder-browser-entry-workspace'`, `control-openhands browser click 'testid=folder-browser-entry-qa-f07-repo'`, `control-openhands browser click 'testid=folder-browser-use'`, `control-openhands browser click 'testid=workspace-launch-button'`, then `control-openhands conversation start --prompt \"Reply with the single word: ok\" --stay --wait --timeout 240` prints `<git-id>`.\n- An automation setup conversation for the setup-leaving bullet, made right after the finished conversation: `control-openhands browser goto /automations`, `control-openhands browser click 'testid=automations-add-automation'`, `control-openhands browser click 'testid=automations-add-automation-create' --expect-url '/automations/setup'`, `control-openhands conversation start --stay --prompt 'Reply with only the word ok. Do not run any tools.' --wait --timeout 180` (prints `<setup-id>`), then `control-openhands browser goto /`.\n- Cloud-only rows (`F07.cloud-only`) are blocked without an OpenHands Cloud account.\n- Desktop viewport unless a bullet says otherwise.", "gotchas": "- The stop and delete confirmation buttons carry `cancel-button` and `confirm-button`; role names inside `role=dialog` work too.\n- `stop-button`, `show-skills-button`, `show-hooks-button` and `show-agent-tools-button` also exist in the composer, and `ellipsis-button` in sidebar rows: always scope to `testid=chat-pane-header` or `testid=conversation-name-context-menu`.\n- Stop Runtime (status menu) and Stop Conversation (menu) both navigate to `/conversations`. On a finished local conversation the agent server keeps `finished`; use a running conversation to see `paused` and Start Runtime.\n- Rename has no Escape-to-cancel: Escape leaves the input open and the next blur saves whatever was typed.\n- Escape closes the status-dot menu, as it closes the \"...\" and Git actions menus (`browser count 'testid=server-status-context-menu'` is `0`). A menu opened by click or Enter returns focus to `server-status-menu-trigger`; one opened by hover closes without moving focus (focus left on a menu item falls to `body`). Touch taps cannot be driven with the CLI, so the tap path is unverified.\n- The status menu reads `Running` / `Stop Runtime` for a finished local conversation too; only a paused one shows `Server Stopped` / `Start Runtime`.\n- Panel and Overview columns animate: read their state with `browser wait <sel> --state hidden|visible`, not an immediate `browser visible`.\n- The Skills dialog shows the list read with the page or at the last Refresh: a project skill added later appears after Refresh (or `browser reload`), not on reopening the dialog within 10 minutes. Its own text says to click Refresh.\n- `tabs-pane-header` and `conversation-overview-column` stay in the DOM when hidden: assert with `browser visible` or `aria-pressed`, not `count`.\n- `browser visible 'testid=usage-panel'` right after Display Usage and Cost can read `false`; `browser wait` first.\n- Exports and the zip land in `<run>/private/downloads`; the transcript holds the prompts and the zip holds the full event store. Review before sharing.\n- `--observe` accepts the same selector syntax as other verbs (`'testid=chat-messages-skeleton'`) as well as CSS.\n- A sidebar row click lands on `/conversations/<id>?backend=default-local`: end `--expect-url` regexes without `$` there.\n- `role=button[name=\"Pull\"]` also matches `Pull Request`; add `[exact]`.\n- The Open Workspace folder browser has no path field: walk it with `folder-browser-up` and `folder-browser-entry-<name>`. `fixture git-repo` puts the repo in `<run>/workspace`, three levels above the `workspace` favorite.\n- The repo, branch and Pull/Push chips come from `git remote get-url origin` in the workspace, read when the page loads: reload after changing the remote.\n- The Usage tab logs `GET /api/llm/balance` 404 on a local backend (F27); count it in `browser errors --app-only` but it is not an F07 failure.\n- The Overview drawer has no Automations, Skills, MCP or Secrets sections. The Overview menu offers Workspace and Git rows only, and its drawer opens only Pull Requests (`F27.overview-drawer`).\n- Dialogs move focus inside when they open and keep Tab inside: the Skills and Hooks dialogs start on their close button, Stop and Delete on `cancel-button`, Export on `close-transcript-export-modal`, Agent Tools on `close-system-message-modal`. In the Hooks dialog, focusing Refresh with Tab opens its tooltip, which takes the first Escape; a second Escape closes the dialog.\n- The Hooks dialog shows the workspace's `hooks.json` as of its last read (its first open for the conversation after a page load, or Refresh), not the hooks the conversation started with: after the hooks steps above, `browser goto /conversations/<git-id>` (started before the file existed; its `hook_config` is `null`) and opening the dialog lists both events too. A reopen within 5 minutes of that read shows the same list and sends no request: click Refresh after editing `hooks.json`.\n- A prompt- or agent-type hook shows its text under Commands (the `prompt`, or an agent hook's `system_prompt`). With the Hooks dialog closed (the hooks steps above end that way; otherwise `control-openhands browser click 'testid=close-hooks-modal'`), write `<run>/workspace/qa-f07-repo/.openhands/hooks.json` as `{\"stop\":[{\"hooks\":[{\"type\":\"prompt\",\"prompt\":\"QA_F07 prompt hook\"}]}],\"session_start\":[{\"hooks\":[{\"type\":\"agent\",\"system_prompt\":\"QA_F07 agent hook\"}]}]}`, run `control-openhands browser goto /conversations/<hooks-id>`, open the menu and `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-hooks-button'`, click `testid=hooks-modal >> testid=refresh-hooks`, then `role=button[name=\"Session Start 1 hook\"]` and `role=button[name=\"Stop 1 hook\"]` inside `testid=hooks-modal` (an event expanded before a Refresh stays expanded, and a second click collapses it, so start from a reopened dialog). In `control-openhands browser snapshot 'testid=hooks-modal'` they expand, in that order, to `Matcher * Commands QA_F07 agent hook Type: agent Timeout: 60s` and `Matcher * Commands QA_F07 prompt hook Type: prompt Timeout: 60s`, and `browser text 'testid=hooks-modal >> pre'` returns `QA_F07 agent hook` and `QA_F07 prompt hook` (`control-openhands browser screenshot --feature F07.hooks-modal --name prompt-agent-hooks`). An agent hook without `system_prompt` shows no box: with `{\"session_start\":[{\"hooks\":[{\"type\":\"agent\"}]}]}`, close, reopen and Refresh, `Session Start 1 hook` expands to `Matcher * Commands Type: agent Timeout: 60s` and `browser count 'testid=hooks-modal >> pre'` is `0`.\n- An agent hook shows its `system_prompt`, the text it runs with, even when it also carries a `prompt`, and an agent hook with only a `prompt` shows no box. With `{\"session_start\":[{\"hooks\":[{\"type\":\"agent\",\"prompt\":\"QA_F07 stray prompt\",\"system_prompt\":\"QA_F07 agent system prompt\"}]}],\"stop\":[{\"hooks\":[{\"type\":\"agent\",\"prompt\":\"QA_F07 prompt only\"}]}]}` in `hooks.json`, close, reopen and Refresh the dialog, then expand both events: `Session Start 1 hook` reads `Matcher * Commands QA_F07 agent system prompt Type: agent Timeout: 60s`, `Stop 1 hook` reads `Matcher * Commands Type: agent Timeout: 60s`, and `control-openhands browser count 'text=QA_F07 stray prompt'` and `browser count 'text=QA_F07 prompt only'` are `0` (`browser screenshot --feature F07.hooks-modal --name agent-hook-system-prompt`). Agent hooks in `hooks.json` really run on the next turn in that workspace (the session-start one shows a `Hook: SessionStart failed` row), so write the recipe's two-event `hooks.json` (`pre_tool_use` and `stop`) back and Refresh afterwards.\n- With the Agent Server down, Refresh in the Hooks dialog shows its own fetch error (#18177): after `control-openhands service stop agent-server` on an open dialog, `refresh-hooks` and `browser wait 'testid=hooks-modal >> text=Failed to fetch hooks' --timeout 30000`, the dialog reads `Failed to fetch hooks. Please try again later.` under its explanation (`browser count 'testid=hooks-modal >> text=Failed to fetch skills'` is `0`). The 502s and the closed WebSocket that `browser errors --app-only` then lists are the outage itself. Run it last, or `restart` afterwards.\n- After the user-message branch in `F07.branch-from-message`, Send stays disabled next to the prefilled text (`browser enabled 'testid=submit-button'` is `false`; Known failure #18202, see F05 Gotchas). Enter in the field sends it.\n- Known issue [#18136](https://github.com/OpenHands/OpenHands/issues/18136) (`F07.cloud-only`): Connect Repo on a cloud conversation does not save the repository on the server. Live prerequisite: An authenticated Cloud sandbox/repository provider and a second fresh browser to compare server repository metadata.\n- Known issue [#17494](https://github.com/OpenHands/OpenHands/issues/17494) (`F07.export-transcript`): Transcript export leaks raw i18n template for ACP tool calls without a title. Live prerequisite: An authenticated ACP session that emits a titleless tool call, then Markdown and HTML export.\n- Known issue [#16876](https://github.com/OpenHands/OpenHands/issues/16876) (`F07.history-skeleton`): Planning-agent history flag stays true forever when sub-conversations fetch fails — infinite chat loading skeleton. Live prerequisite: A real planning sub-conversation whose batch lookup fails or yields null, with the main history available.\n- Known issue [#16727](https://github.com/OpenHands/OpenHands/issues/16727) (`F07.open-by-url`): Conversation corrupted after VM restart: \"Missing event file\". Live prerequisite: A disposable VM interrupted at the event-write boundary or the original missing-event persisted fixture.\n- Known issue [#16356](https://github.com/OpenHands/OpenHands/issues/16356) (`F07.open-by-url`): Conversation LLM config is frozen at creation time — resuming after API key rotation fails permanently with LLMAuthenticationError. Live prerequisite: A working provider key that can be rotated/revoked, an existing conversation using it, and a replacement profile.\n- Known issue [#15543](https://github.com/OpenHands/OpenHands/issues/15543) (`F07.git-control-bar`): Bad parsing for branch retrieval. Live prerequisite: The original Git probe stdout/interleaving that temporarily puts a filename in the branch badge; dirty state alone is not a deterministic trigger.\n- Known issue [OpenHands/software-agent-sdk#5640](https://github.com/OpenHands/software-agent-sdk/issues/5640) (`F07.download-zip`): Download conversation data can return404 after a warm-pool server initializes a custom conversation-storage path. Although closed upstream, the pinned SDK 1.55.0 still reads the default configuration path here; this remains a supported-version distinction, not a freshly reproduced failure. Live prerequisite: A deferred-init warm-pool Agent Server initialized with a non-default conversations_path; this launcher starts an already-initialized server.\n- Known failure [OpenHands/software-agent-sdk#5629](https://github.com/OpenHands/software-agent-sdk/issues/5629) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A second server's native first load rewrote a newer saved title with its startup copy; the connected Canvas header showed that older title. Recipe: create an idle conversation on A and PATCH its title to `QA_SHARED_BEFORE_B_5629_REPEAT`; run pinned SDK Python `python .agents/skills/verify-openhands/scripts/fixtures/shared-storage-server.py --run <run> --seconds 300` in the background. It starts only a bounded no-editor SDK B sharing this run's conversation directory/cipher/session key; its private `server.json` supplies the port and owned operator/PGID. After B listens, PATCH A to `QA_SHARED_AFTER_B_5629_REPEAT` and confirm disk. Run `control-openhands service stop agent-server --run <run>`, then `python .../fixtures/shared-storage-server.py --run <run> --load-id <id>`: native B `/events/count` returned 200 and disk reverted to BEFORE_B. Restart A only to serve Canvas, choose default-local and doctor; use the Add Backend recipe from [F25](F25-backends-and-cloud.md) to connect B at its recorded port with the private key file, then open its card: the header reads BEFORE_B. SIGTERM only the recorded helper operator and verify its port/PGID gone; the helper also bounds runtime and terminates remaining group descendants. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-repeat-before-b-load-5629.json`, `metadata-repeat-after-b-load-5629.json`, `shared-first-load-reverted-5629.png`. The native first-load request is explicit; stale catalog display alone is insufficient proof of this overwrite.\n- Known failure [OpenHands/software-agent-sdk#5539](https://github.com/OpenHands/software-agent-sdk/issues/5539) (`F07.download-zip`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0: three real overlapping receivers of one idle conversation made 929 requests in 20 seconds, including 31 completed corrupt ZIPs with status 200, 106 server 500s, 269 ingress 502s and 133 RemoteProtocolErrors. A sequential 2,000-request control returned valid ZIPs. Recipe (use the pinned SDK Python with httpx): `python .agents/skills/verify-openhands/scripts/fixtures/download-trajectory-overlap.py --base-url <own-base-url> --conversation-id <id> --api-key-file <run>/private/session-key --output-dir <run>/private/trajectory-overlap --seconds 20 --clients 3 --rounds 5000 --slow-ms 2`. It enforces loopback, request/duration/network/uncompressed/member budgets and bounded CRC reads; aborts and unsupported archives remain unverified. While it runs, use `browser goto /conversations/<id>`, open `chat-pane-header >> ellipsis-button`, click `download-trajectory-button`, then `browser downloads --last 3 --inspect` (all CLI commands with `--run <run>`). That Canvas click downloaded an inspectable ZIP; its particular result is not claimed corrupt. Private raw samples stay private. Evidence: `backfill-18309-surface-rechecks/F07.download-zip/trajectory-overlap-5539-summary.json` and `overlapping-download-5539.png`. The summary correction excludes two ambiguous Content-Length differences that could be intentional deadline aborts; only completed corrupt ZIPs count as corrupt downloads.\n- Known issue [OpenHands/software-agent-sdk#5094](https://github.com/OpenHands/software-agent-sdk/issues/5094) (`F07.open-by-url`): After ACP session/load fails, the backend can create a fresh model session while Canvas still shows the old context. Live prerequisite: An authenticated Codex ACP session whose original session/load fails while persisted history remains.\n- Known issue [OpenHands/software-agent-sdk#4893](https://github.com/OpenHands/software-agent-sdk/issues/4893) (`F07.status-menu`): A sandbox frozen mid-run can resume with a stale Running state that rejects normal resume requests. Live prerequisite: A disposable sandbox/VM frozen mid-run with a live model and persisted ownership state.\n- Known failure [OpenHands/software-agent-sdk#4800](https://github.com/OpenHands/software-agent-sdk/issues/4800) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A refused rename changed the server's GET title while `meta.json` kept the previous title; graceful shutdown subsequently persisted the rejected title. Recipe: create an idle dummy-profile conversation, then double-click `conversation-name-title`, fill `conversation-name-input` with `QA_META_BEFORE_4800` and press Enter; verify GET/disk. Record the original mode and temporarily `chmod 0500 <run>/private/state/dev_conversations/<id-hex>` to deny an atomic metadata replacement. Rename through that UI to `QA_META_REFUSED_4800`: the permission-denied toast/HTTP 500 appears, GET shows REFUSED, disk still shows BEFORE. Restore the exact original directory mode in `finally`, then `restart`, `doctor`, GET and reload (CLI with `--run <run>`): REFUSED was persisted during graceful shutdown. Only the owned temporary directory is changed; evidence contains title/timestamp fields, not stored secrets. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-before-4800.json`, `metadata-denied-4800.json`, `refused-title-4800.png`, `metadata-after-graceful-restart-4800.json`. Tags/autotitle and ownership-conflict branches remain untested.\n\n- Known issue [#16877](https://github.com/OpenHands/OpenHands/issues/16877) (`F07.open-by-url`): Untitled terminal-status conversations are reported to keep the 3-second title polling interval. On frozen target `fa1f724ab5737a17453ad6081129eb91c5a423c6`, a real Local `autotitle:false`/idle conversation retained `title:null`, while eight browser batch GETs had 30.008–30.031-second intervals over 210 seconds. The direct adapter supplies a default display title. This does not exercise the Cloud null-title IDLE/FINISHED branch, which needs an approved Cloud conversation with a failed/unsupported title job. Run `backfill-18309-coordinator-checks`, Agent Server/SDK 1.54.0, automation 1.19.3; retained `evidence/F07.open-by-url/16877-local-polling.json` and `16877-local-idle.png`. Arrange the empty conversation with `api POST /api/conversations --write --data @<idle-autotitle-disabled-payload>`, open `/conversations/<id>`, clear `browser network`, leave the tab foreground, then read `browser network --filter ids= --last 100` and `conversation status <id>`. No model call or response was used.", "features": [ { "id": "F07.open-by-url", "description": "`/conversations/<id>` shows header, history and composer, and remembers the id as the backend's last conversation." }, { "id": "F07.missing-redirect", "description": "an unknown id toasts \"This conversation does not exist…\" and redirects to `/conversations`." }, { "id": "F07.history-skeleton", "description": "opening a conversation shows a message skeleton until its history has loaded." }, { "id": "F07.panel-route", "description": "at phone width (up to 1024 px) the header panel button opens `/conversations/<id>/panel`, the tabs panel full width with a Back button; wider, that URL redirects to `/conversations/<id>` with the drawer open beside the chat." }, { "id": "F07.rename-inline", "description": "double-click the title (or menu Rename), Enter or blur saves with a toast; an empty value reverts." }, { "id": "F07.status-menu", "description": "hovering the status dot (or tapping it on touch) opens a menu with the state and Stop Runtime (active) or Start Runtime (paused)." }, { "id": "F07.menu-open", "description": "the \"...\" menu opens under the title; Escape closes it and returns focus to \"...\", and so does an outside click (without the focus move)." }, { "id": "F07.menu-items-by-state", "description": "Show Available Hooks and Stop Conversation appear only while the conversation is active." }, { "id": "F07.skills-modal", "description": "Show Available Skills lists the loaded skills with expandable rows and Refresh." }, { "id": "F07.hooks-modal", "description": "Show Available Hooks lists the workspace's `.openhands/hooks.json` hooks by event (read when it first opens for a conversation after a page load, and on Refresh; a reopen within 5 minutes of the last read shows the cached list), or \"No hooks configured for this conversation.\"" }, { "id": "F07.skills-modal-project", "description": "a workspace with `.agents/skills/<name>/SKILL.md` adds a \"Project skills\" section above the public ones; each row expands to its triggers and body. The list is read with the page and on Refresh; opening the dialog within 10 minutes of the last read shows the cached list." }, { "id": "F07.agent-tools-modal", "description": "Show Agent Tools & Metadata shows the system message and the tool list." }, { "id": "F07.export-transcript", "description": "Export… downloads a Markdown or HTML transcript with optional tool details and timestamps." }, { "id": "F07.download-zip", "description": "Download conversation data downloads `conversation_<id>.zip`." }, { "id": "F07.display-cost", "description": "Display Usage and Cost opens the right panel on the Usage tab." }, { "id": "F07.stop-confirm", "description": "Stop Conversation asks for confirmation; Confirm Close pauses the conversation and returns to `/conversations`." }, { "id": "F07.delete-confirm", "description": "Delete Conversation names the title in a confirmation; Cancel keeps it, Confirm Delete removes it and returns to `/conversations`." }, { "id": "F07.branch-from-message", "description": "Branch from here on a message forks the conversation as `<title> (branch)`; on a user message the message is left out and its text goes back into the composer." }, { "id": "F07.right-panel-toggle", "description": "the header panel button shows and hides the right tabs panel (closed again after a reload)." }, { "id": "F07.overview-toggle", "description": "the header (i) button shows and hides the Overview column, which is exclusive with the right panel." }, { "id": "F07.git-actions-menu", "description": "the header Git actions menu offers Commits, Pull, Push, Create PR and Create New Branch; each pre-fills the composer without sending, and Send becomes enabled so the user can submit or edit the prompt." }, { "id": "F07.git-control-bar", "description": "under the composer, a folder workspace shows an inert workspace chip and the branch; with a GitHub `origin` the repo and branch chips link to GitHub and Pull, Push and Pull Request appear." }, { "id": "F07.git-control-bar-send", "description": "the Pull, Push and Pull Request chips send their git prompt to the agent at once." }, { "id": "F07.cloud-only", "description": "Share publicly, Connect Repo, `task-<uuid>` start URLs, paused-sandbox resume, archived read-only state and the shared-conversation redirect exist only on a Cloud backend." }, { "id": "F07.phone", "description": "at 390 px the header keeps its controls and the info dialogs fit the viewport." } ], "scenarios": [ { "name": "F07--open-by-url", "title": "Open by URL", "ids": [ "F07.open-by-url" ], "body": "Run `control-openhands browser goto /conversations/<id>`, `control-openhands browser wait 'testid=chat-pane-header'`, `control-openhands browser text 'testid=conversation-name-title'` and `control-openhands browser count 'testid=chat-interface >> testid=agent-message'`. The title is the generated one and the count is `1`. `control-openhands browser eval \"localStorage.getItem('openhands-last-conversation-by-backend')\"` reads `{\"default-local\":\"<id>\"}`.", "raw": "**Open by URL (`F07.open-by-url`).** Run `control-openhands browser goto /conversations/<id>`, `control-openhands browser wait 'testid=chat-pane-header'`, `control-openhands browser text 'testid=conversation-name-title'` and `control-openhands browser count 'testid=chat-interface >> testid=agent-message'`. The title is the generated one and the count is `1`. `control-openhands browser eval \"localStorage.getItem('openhands-last-conversation-by-backend')\"` reads `{\"default-local\":\"<id>\"}`.", "source_line": 61, "expected": [ "`/conversations/<id>` shows header, history and composer, and remembers the id as the backend's last conversation." ], "notes": [ "Known issue [#16727](https://github.com/OpenHands/OpenHands/issues/16727) (`F07.open-by-url`): Conversation corrupted after VM restart: \"Missing event file\". Live prerequisite: A disposable VM interrupted at the event-write boundary or the original missing-event persisted fixture.", "Known issue [#16356](https://github.com/OpenHands/OpenHands/issues/16356) (`F07.open-by-url`): Conversation LLM config is frozen at creation time — resuming after API key rotation fails permanently with LLMAuthenticationError. Live prerequisite: A working provider key that can be rotated/revoked, an existing conversation using it, and a replacement profile.", "Known issue [OpenHands/software-agent-sdk#5094](https://github.com/OpenHands/software-agent-sdk/issues/5094) (`F07.open-by-url`): After ACP session/load fails, the backend can create a fresh model session while Canvas still shows the old context. Live prerequisite: An authenticated Codex ACP session whose original session/load fails while persisted history remains.", "Known issue [#16877](https://github.com/OpenHands/OpenHands/issues/16877) (`F07.open-by-url`): Untitled terminal-status conversations are reported to keep the 3-second title polling interval. On frozen target `fa1f724ab5737a17453ad6081129eb91c5a423c6`, a real Local `autotitle:false`/idle conversation retained `title:null`, while eight browser batch GETs had 30.008–30.031-second intervals over 210 seconds. The direct adapter supplies a default display title. This does not exercise the Cloud null-title IDLE/FINISHED branch, which needs an approved Cloud conversation with a failed/unsupported title job. Run `backfill-18309-coordinator-checks`, Agent Server/SDK 1.54.0, automation 1.19.3; retained `evidence/F07.open-by-url/16877-local-polling.json` and `16877-local-idle.png`. Arrange the empty conversation with `api POST /api/conversations --write --data @<idle-autotitle-disabled-payload>`, open `/conversations/<id>`, clear `browser network`, leave the tab foreground, then read `browser network --filter ids= --last 100` and `conversation status <id>`. No model call or response was used." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--leaving-an-automation-setup-conversation", "title": "Leaving an automation setup conversation", "ids": [ "F07.open-by-url" ], "body": "A conversation tagged `automationsetup: draft` (started from `/automations/setup`, see `F04.automation-setup-tag`) opens as a setup page (`testid=automation-setup-topbar`, no `chat-pane-header`). Make an empty conversation: `control-openhands browser goto /`, `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=launch-no-workspace' --expect-url '/conversations/'` (note `<plain-id>`; `control-openhands api GET /api/conversations/<plain-id> --pick tags` is only `clientsource`). Open the setup conversation from the sidebar (`control-openhands browser click 'a[href*=\"/conversations/<setup-id>\"] >> nth=0' --expect-url '/conversations/<setup-id>'`), then go back the same way (`'a[href*=\"/conversations/<plain-id>\"] >> nth=0'`). Expected: `<plain-id>` opens as a normal conversation: `control-openhands browser wait 'testid=chat-pane-header'`, `browser count 'testid=chat-pane-header'` is `1`, `browser count 'testid=automation-setup-topbar'` is `0`, and `api GET /api/conversations/<plain-id> --pick tags` is still only `clientsource`, also after `browser reload` and after repeating the round trip; the sidebar row of `<plain-id>` has no `Automation setup: Draft` chip (`browser screenshot --feature F07.open-by-url --name plain-stays-plain`). The setup conversation is in this tab's `openhands-automation-setup-handoffs` sessionStorage entry during the check (`control-openhands browser storage --session --values`). The same holds at phone width through the sidebar drawer (`sidebar-mobile-menu-toggle`, then the rows inside `testid=sidebar-mobile-drawer`).", "raw": "**Leaving an automation setup conversation (`F07.open-by-url`).** A conversation tagged `automationsetup: draft` (started from `/automations/setup`, see `F04.automation-setup-tag`) opens as a setup page (`testid=automation-setup-topbar`, no `chat-pane-header`). Make an empty conversation: `control-openhands browser goto /`, `control-openhands browser click 'testid=conversation-panel-new-thread-picker'`, `control-openhands browser click 'testid=launch-no-workspace' --expect-url '/conversations/'` (note `<plain-id>`; `control-openhands api GET /api/conversations/<plain-id> --pick tags` is only `clientsource`). Open the setup conversation from the sidebar (`control-openhands browser click 'a[href*=\"/conversations/<setup-id>\"] >> nth=0' --expect-url '/conversations/<setup-id>'`), then go back the same way (`'a[href*=\"/conversations/<plain-id>\"] >> nth=0'`). Expected: `<plain-id>` opens as a normal conversation: `control-openhands browser wait 'testid=chat-pane-header'`, `browser count 'testid=chat-pane-header'` is `1`, `browser count 'testid=automation-setup-topbar'` is `0`, and `api GET /api/conversations/<plain-id> --pick tags` is still only `clientsource`, also after `browser reload` and after repeating the round trip; the sidebar row of `<plain-id>` has no `Automation setup: Draft` chip (`browser screenshot --feature F07.open-by-url --name plain-stays-plain`). The setup conversation is in this tab's `openhands-automation-setup-handoffs` sessionStorage entry during the check (`control-openhands browser storage --session --values`). The same holds at phone width through the sidebar drawer (`sidebar-mobile-menu-toggle`, then the rows inside `testid=sidebar-mobile-drawer`).", "source_line": 62, "expected": [ "`/conversations/<id>` shows header, history and composer, and remembers the id as the backend's last conversation." ], "notes": [ "Known issue [#16727](https://github.com/OpenHands/OpenHands/issues/16727) (`F07.open-by-url`): Conversation corrupted after VM restart: \"Missing event file\". Live prerequisite: A disposable VM interrupted at the event-write boundary or the original missing-event persisted fixture.", "Known issue [#16356](https://github.com/OpenHands/OpenHands/issues/16356) (`F07.open-by-url`): Conversation LLM config is frozen at creation time — resuming after API key rotation fails permanently with LLMAuthenticationError. Live prerequisite: A working provider key that can be rotated/revoked, an existing conversation using it, and a replacement profile.", "Known issue [OpenHands/software-agent-sdk#5094](https://github.com/OpenHands/software-agent-sdk/issues/5094) (`F07.open-by-url`): After ACP session/load fails, the backend can create a fresh model session while Canvas still shows the old context. Live prerequisite: An authenticated Codex ACP session whose original session/load fails while persisted history remains.", "Known issue [#16877](https://github.com/OpenHands/OpenHands/issues/16877) (`F07.open-by-url`): Untitled terminal-status conversations are reported to keep the 3-second title polling interval. On frozen target `fa1f724ab5737a17453ad6081129eb91c5a423c6`, a real Local `autotitle:false`/idle conversation retained `title:null`, while eight browser batch GETs had 30.008–30.031-second intervals over 210 seconds. The direct adapter supplies a default display title. This does not exercise the Cloud null-title IDLE/FINISHED branch, which needs an approved Cloud conversation with a failed/unsupported title job. Run `backfill-18309-coordinator-checks`, Agent Server/SDK 1.54.0, automation 1.19.3; retained `evidence/F07.open-by-url/16877-local-polling.json` and `16877-local-idle.png`. Arrange the empty conversation with `api POST /api/conversations --write --data @<idle-autotitle-disabled-payload>`, open `/conversations/<id>`, clear `browser network`, leave the tab foreground, then read `browser network --filter ids= --last 100` and `conversation status <id>`. No model call or response was used." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--unknown-id", "title": "Unknown id", "ids": [ "F07.missing-redirect" ], "body": "Run `control-openhands browser goto /conversations/00000000-0000-0000-0000-000000000000`, `control-openhands browser wait-url '/conversations$'` and `control-openhands browser toasts`. A toast reads `This conversation does not exist, or you do not have permission to access it. If this is your conversation, try switching to the workspace where it was created.` and the last-conversation slot is `{}`. It is the only toast: `control-openhands browser toasts --history` has no entry with `does not understand`.", "raw": "**Unknown id (`F07.missing-redirect`).** Run `control-openhands browser goto /conversations/00000000-0000-0000-0000-000000000000`, `control-openhands browser wait-url '/conversations$'` and `control-openhands browser toasts`. A toast reads `This conversation does not exist, or you do not have permission to access it. If this is your conversation, try switching to the workspace where it was created.` and the last-conversation slot is `{}`. It is the only toast: `control-openhands browser toasts --history` has no entry with `does not understand`.", "source_line": 63, "expected": [ "an unknown id toasts \"This conversation does not exist…\" and redirects to `/conversations`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--rename-inline", "title": "Rename inline", "ids": [ "F07.rename-inline" ], "body": "On `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`; the unknown id redirected to `/conversations`) run `control-openhands browser dblclick 'testid=conversation-name-title'`, `control-openhands browser fill 'testid=conversation-name-input' 'QA_F07 renamed'`, `control-openhands browser press Enter`, `control-openhands browser toasts` (`Conversation title updated successfully`), then `control-openhands browser reload` and `control-openhands browser text 'testid=conversation-name-title'`. It reads `QA_F07 renamed`, and so does `control-openhands browser text 'testid=conversation-card >> testid=conversation-card-title'`.", "raw": "**Rename inline (`F07.rename-inline`).** On `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`; the unknown id redirected to `/conversations`) run `control-openhands browser dblclick 'testid=conversation-name-title'`, `control-openhands browser fill 'testid=conversation-name-input' 'QA_F07 renamed'`, `control-openhands browser press Enter`, `control-openhands browser toasts` (`Conversation title updated successfully`), then `control-openhands browser reload` and `control-openhands browser text 'testid=conversation-name-title'`. It reads `QA_F07 renamed`, and so does `control-openhands browser text 'testid=conversation-card >> testid=conversation-card-title'`.", "source_line": 64, "expected": [ "double-click the title (or menu Rename), Enter or blur saves with a toast; an empty value reverts." ], "notes": [ "Known failure [OpenHands/software-agent-sdk#5629](https://github.com/OpenHands/software-agent-sdk/issues/5629) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A second server's native first load rewrote a newer saved title with its startup copy; the connected Canvas header showed that older title. Recipe: create an idle conversation on A and PATCH its title to `QA_SHARED_BEFORE_B_5629_REPEAT`; run pinned SDK Python `python .agents/skills/verify-openhands/scripts/fixtures/shared-storage-server.py --run <run> --seconds 300` in the background. It starts only a bounded no-editor SDK B sharing this run's conversation directory/cipher/session key; its private `server.json` supplies the port and owned operator/PGID. After B listens, PATCH A to `QA_SHARED_AFTER_B_5629_REPEAT` and confirm disk. Run `control-openhands service stop agent-server --run <run>`, then `python .../fixtures/shared-storage-server.py --run <run> --load-id <id>`: native B `/events/count` returned 200 and disk reverted to BEFORE_B. Restart A only to serve Canvas, choose default-local and doctor; use the Add Backend recipe from [F25](F25-backends-and-cloud.md) to connect B at its recorded port with the private key file, then open its card: the header reads BEFORE_B. SIGTERM only the recorded helper operator and verify its port/PGID gone; the helper also bounds runtime and terminates remaining group descendants. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-repeat-before-b-load-5629.json`, `metadata-repeat-after-b-load-5629.json`, `shared-first-load-reverted-5629.png`. The native first-load request is explicit; stale catalog display alone is insufficient proof of this overwrite.", "Known failure [OpenHands/software-agent-sdk#4800](https://github.com/OpenHands/software-agent-sdk/issues/4800) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A refused rename changed the server's GET title while `meta.json` kept the previous title; graceful shutdown subsequently persisted the rejected title. Recipe: create an idle dummy-profile conversation, then double-click `conversation-name-title`, fill `conversation-name-input` with `QA_META_BEFORE_4800` and press Enter; verify GET/disk. Record the original mode and temporarily `chmod 0500 <run>/private/state/dev_conversations/<id-hex>` to deny an atomic metadata replacement. Rename through that UI to `QA_META_REFUSED_4800`: the permission-denied toast/HTTP 500 appears, GET shows REFUSED, disk still shows BEFORE. Restore the exact original directory mode in `finally`, then `restart`, `doctor`, GET and reload (CLI with `--run <run>`): REFUSED was persisted during graceful shutdown. Only the owned temporary directory is changed; evidence contains title/timestamp fields, not stored secrets. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-before-4800.json`, `metadata-denied-4800.json`, `refused-title-4800.png`, `metadata-after-graceful-restart-4800.json`. Tags/autotitle and ownership-conflict branches remain untested." ], "annotations": [], "states": [ "failure" ] }, { "name": "F07--rename-from-the-menu-empty-value", "title": "Rename from the menu, empty value", "ids": [ "F07.rename-inline" ], "body": "Run `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=rename-button'`, `control-openhands browser fill 'testid=conversation-name-input' ''` and `control-openhands browser press Enter`. The title stays `QA_F07 renamed` and no toast appears.", "raw": "**Rename from the menu, empty value (`F07.rename-inline`).** Run `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=rename-button'`, `control-openhands browser fill 'testid=conversation-name-input' ''` and `control-openhands browser press Enter`. The title stays `QA_F07 renamed` and no toast appears.", "source_line": 65, "expected": [ "double-click the title (or menu Rename), Enter or blur saves with a toast; an empty value reverts." ], "notes": [ "Known failure [OpenHands/software-agent-sdk#5629](https://github.com/OpenHands/software-agent-sdk/issues/5629) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A second server's native first load rewrote a newer saved title with its startup copy; the connected Canvas header showed that older title. Recipe: create an idle conversation on A and PATCH its title to `QA_SHARED_BEFORE_B_5629_REPEAT`; run pinned SDK Python `python .agents/skills/verify-openhands/scripts/fixtures/shared-storage-server.py --run <run> --seconds 300` in the background. It starts only a bounded no-editor SDK B sharing this run's conversation directory/cipher/session key; its private `server.json` supplies the port and owned operator/PGID. After B listens, PATCH A to `QA_SHARED_AFTER_B_5629_REPEAT` and confirm disk. Run `control-openhands service stop agent-server --run <run>`, then `python .../fixtures/shared-storage-server.py --run <run> --load-id <id>`: native B `/events/count` returned 200 and disk reverted to BEFORE_B. Restart A only to serve Canvas, choose default-local and doctor; use the Add Backend recipe from [F25](F25-backends-and-cloud.md) to connect B at its recorded port with the private key file, then open its card: the header reads BEFORE_B. SIGTERM only the recorded helper operator and verify its port/PGID gone; the helper also bounds runtime and terminates remaining group descendants. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-repeat-before-b-load-5629.json`, `metadata-repeat-after-b-load-5629.json`, `shared-first-load-reverted-5629.png`. The native first-load request is explicit; stale catalog display alone is insufficient proof of this overwrite.", "Known failure [OpenHands/software-agent-sdk#4800](https://github.com/OpenHands/software-agent-sdk/issues/4800) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A refused rename changed the server's GET title while `meta.json` kept the previous title; graceful shutdown subsequently persisted the rejected title. Recipe: create an idle dummy-profile conversation, then double-click `conversation-name-title`, fill `conversation-name-input` with `QA_META_BEFORE_4800` and press Enter; verify GET/disk. Record the original mode and temporarily `chmod 0500 <run>/private/state/dev_conversations/<id-hex>` to deny an atomic metadata replacement. Rename through that UI to `QA_META_REFUSED_4800`: the permission-denied toast/HTTP 500 appears, GET shows REFUSED, disk still shows BEFORE. Restore the exact original directory mode in `finally`, then `restart`, `doctor`, GET and reload (CLI with `--run <run>`): REFUSED was persisted during graceful shutdown. Only the owned temporary directory is changed; evidence contains title/timestamp fields, not stored secrets. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-before-4800.json`, `metadata-denied-4800.json`, `refused-title-4800.png`, `metadata-after-graceful-restart-4800.json`. Tags/autotitle and ownership-conflict branches remain untested." ], "annotations": [], "states": [ "failure" ] }, { "name": "F07--history-skeleton-and-sidebar-entry", "title": "History skeleton and sidebar entry", "ids": [ "F07.history-skeleton", "F07.open-by-url" ], "body": "From `/` (`control-openhands browser goto /`) run `control-openhands browser click 'testid=conversation-card >> has-text=QA_F07 renamed' --expect-url '/conversations/[0-9a-f-]+' --observe 'testid=chat-messages-skeleton' --observe-ms 3000` (run it after the rename above; before it, match the generated title instead). The `observed` list shows the skeleton present for a few hundred ms, then `<absent>`; the URL is `/conversations/<id>?backend=default-local`.", "raw": "**History skeleton and sidebar entry (`F07.history-skeleton`, `F07.open-by-url`).** From `/` (`control-openhands browser goto /`) run `control-openhands browser click 'testid=conversation-card >> has-text=QA_F07 renamed' --expect-url '/conversations/[0-9a-f-]+' --observe 'testid=chat-messages-skeleton' --observe-ms 3000` (run it after the rename above; before it, match the generated title instead). The `observed` list shows the skeleton present for a few hundred ms, then `<absent>`; the URL is `/conversations/<id>?backend=default-local`.", "source_line": 66, "expected": [ "opening a conversation shows a message skeleton until its history has loaded.", "`/conversations/<id>` shows header, history and composer, and remembers the id as the backend's last conversation." ], "notes": [ "Known issue [#16876](https://github.com/OpenHands/OpenHands/issues/16876) (`F07.history-skeleton`): Planning-agent history flag stays true forever when sub-conversations fetch fails — infinite chat loading skeleton. Live prerequisite: A real planning sub-conversation whose batch lookup fails or yields null, with the main history available.", "Known issue [#16727](https://github.com/OpenHands/OpenHands/issues/16727) (`F07.open-by-url`): Conversation corrupted after VM restart: \"Missing event file\". Live prerequisite: A disposable VM interrupted at the event-write boundary or the original missing-event persisted fixture.", "Known issue [#16356](https://github.com/OpenHands/OpenHands/issues/16356) (`F07.open-by-url`): Conversation LLM config is frozen at creation time — resuming after API key rotation fails permanently with LLMAuthenticationError. Live prerequisite: A working provider key that can be rotated/revoked, an existing conversation using it, and a replacement profile.", "Known issue [OpenHands/software-agent-sdk#5094](https://github.com/OpenHands/software-agent-sdk/issues/5094) (`F07.open-by-url`): After ACP session/load fails, the backend can create a fresh model session while Canvas still shows the old context. Live prerequisite: An authenticated Codex ACP session whose original session/load fails while persisted history remains.", "Known issue [#16877](https://github.com/OpenHands/OpenHands/issues/16877) (`F07.open-by-url`): Untitled terminal-status conversations are reported to keep the 3-second title polling interval. On frozen target `fa1f724ab5737a17453ad6081129eb91c5a423c6`, a real Local `autotitle:false`/idle conversation retained `title:null`, while eight browser batch GETs had 30.008–30.031-second intervals over 210 seconds. The direct adapter supplies a default display title. This does not exercise the Cloud null-title IDLE/FINISHED branch, which needs an approved Cloud conversation with a failed/unsupported title job. Run `backfill-18309-coordinator-checks`, Agent Server/SDK 1.54.0, automation 1.19.3; retained `evidence/F07.open-by-url/16877-local-polling.json` and `16877-local-idle.png`. Arrange the empty conversation with `api POST /api/conversations --write --data @<idle-autotitle-disabled-payload>`, open `/conversations/<id>`, clear `browser network`, leave the tab foreground, then read `browser network --filter ids= --last 100` and `conversation status <id>`. No model call or response was used." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--status-menu-stop", "title": "Status menu, stop", "ids": [ "F07.status-menu" ], "body": "On `/conversations/<id>` run `control-openhands browser hover 'testid=server-status-menu-trigger'` (for a mouse the hover opens the menu and a click on the open menu's dot closes it; a CLI `click` from elsewhere usually opens it only because the click lands before the hover renders, a race that sometimes leaves it closed, so do not click), `control-openhands browser text 'testid=server-status-context-menu'` (`Running` / `Stop Runtime`), then `control-openhands browser click 'testid=server-status-context-menu >> testid=stop-server-button' --expect-url '/conversations$'` and `control-openhands browser toasts`. The toasts read `Stopping conversation...` and `Conversation stopped` and the URL is `/conversations`.", "raw": "**Status menu, stop (`F07.status-menu`).** On `/conversations/<id>` run `control-openhands browser hover 'testid=server-status-menu-trigger'` (for a mouse the hover opens the menu and a click on the open menu's dot closes it; a CLI `click` from elsewhere usually opens it only because the click lands before the hover renders, a race that sometimes leaves it closed, so do not click), `control-openhands browser text 'testid=server-status-context-menu'` (`Running` / `Stop Runtime`), then `control-openhands browser click 'testid=server-status-context-menu >> testid=stop-server-button' --expect-url '/conversations$'` and `control-openhands browser toasts`. The toasts read `Stopping conversation...` and `Conversation stopped` and the URL is `/conversations`.", "source_line": 67, "expected": [ "hovering the status dot (or tapping it on touch) opens a menu with the state and Stop Runtime (active) or Start Runtime (paused)." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#4893](https://github.com/OpenHands/software-agent-sdk/issues/4893) (`F07.status-menu`): A sandbox frozen mid-run can resume with a stale Running state that rejects normal resume requests. Live prerequisite: A disposable sandbox/VM frozen mid-run with a live model and persisted ownership state." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--menu", "title": "Menu", "ids": [ "F07.menu-open" ], "body": "Stop Runtime left the browser on `/conversations`: run `control-openhands browser goto /conversations/<git-id>` (a finished conversation whose runtime was not stopped), then `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'` and `control-openhands browser testids 'testid=conversation-name-context-menu'`. On a finished or running conversation the items are `rename-button`, `show-skills-button`, `show-hooks-button`, `show-agent-tools-button`, `export-transcript-button` (`Export…`), `download-trajectory-button`, `display-cost-button` (`Display Usage and Cost`), `stop-button`, `delete-button`. `control-openhands browser press Escape` closes it (`control-openhands browser count 'testid=conversation-name-context-menu'` is `0`) and focus is back on the trigger (`control-openhands browser eval \"document.activeElement.dataset.testid\"` is `ellipsis-button`). Open it again: `control-openhands browser mouse-click 900 500` closes it too (count `0`).", "raw": "**Menu (`F07.menu-open`).** Stop Runtime left the browser on `/conversations`: run `control-openhands browser goto /conversations/<git-id>` (a finished conversation whose runtime was not stopped), then `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'` and `control-openhands browser testids 'testid=conversation-name-context-menu'`. On a finished or running conversation the items are `rename-button`, `show-skills-button`, `show-hooks-button`, `show-agent-tools-button`, `export-transcript-button` (`Export…`), `download-trajectory-button`, `display-cost-button` (`Display Usage and Cost`), `stop-button`, `delete-button`. `control-openhands browser press Escape` closes it (`control-openhands browser count 'testid=conversation-name-context-menu'` is `0`) and focus is back on the trigger (`control-openhands browser eval \"document.activeElement.dataset.testid\"` is `ellipsis-button`). Open it again: `control-openhands browser mouse-click 900 500` closes it too (count `0`).", "source_line": 68, "expected": [ "the \"...\" menu opens under the title; Escape closes it and returns focus to \"...\", and so does an outside click (without the focus move)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--stop-with-confirmation", "title": "Stop with confirmation", "ids": [ "F07.stop-confirm" ], "body": "On `/conversations/<running-id>` run `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=stop-button'` and `control-openhands browser snapshot 'role=dialog'` (`Confirm Stop Conversation`, buttons `Cancel` and `Confirm Close`, test ids `cancel-button` and `confirm-button`). `control-openhands browser click 'role=dialog >> testid=cancel-button'` closes it and `control-openhands conversation status <running-id>` stays `running`. Repeat the two clicks, then `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations$'`; the toasts read `Conversation stopped` and `conversation status <running-id>` is `paused`.", "raw": "**Stop with confirmation (`F07.stop-confirm`).** On `/conversations/<running-id>` run `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=stop-button'` and `control-openhands browser snapshot 'role=dialog'` (`Confirm Stop Conversation`, buttons `Cancel` and `Confirm Close`, test ids `cancel-button` and `confirm-button`). `control-openhands browser click 'role=dialog >> testid=cancel-button'` closes it and `control-openhands conversation status <running-id>` stays `running`. Repeat the two clicks, then `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations$'`; the toasts read `Conversation stopped` and `conversation status <running-id>` is `paused`.", "source_line": 69, "expected": [ "Stop Conversation asks for confirmation; Confirm Close pauses the conversation and returns to `/conversations`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--branch-from-a-message", "title": "Branch from a message", "ids": [ "F07.branch-from-message" ], "body": "On `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`; Stop with confirmation ended on `/conversations`) run `control-openhands browser hover 'testid=agent-message'`, then `control-openhands browser click 'testid=agent-message >> role=button[name=\"Branch from here\"]' --expect-url '/conversations/(?!<id>)[0-9a-f-]+'`, `control-openhands browser text 'testid=conversation-name-title'` (`QA_F07 renamed (branch)`) and `control-openhands browser count 'testid=chat-interface >> testid=user-message'` (`1`, the history is copied up to that reply). Back on `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`), `control-openhands browser hover 'testid=user-message'` and `control-openhands browser click 'testid=user-message >> role=button[name=\"Branch from here\"]' --expect-url '/conversations/(?!<id>)[0-9a-f-]+'` (it prints `/conversations/<branch-id>`): the new branch has `0` user messages and `control-openhands browser text 'testid=chat-input'` reads the original prompt `Reply with the single word: pong`, still there a few seconds later and not sent (`control-openhands conversation events <branch-id> --kinds MessageEvent` has count `0`).", "raw": "**Branch from a message (`F07.branch-from-message`).** On `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`; Stop with confirmation ended on `/conversations`) run `control-openhands browser hover 'testid=agent-message'`, then `control-openhands browser click 'testid=agent-message >> role=button[name=\"Branch from here\"]' --expect-url '/conversations/(?!<id>)[0-9a-f-]+'`, `control-openhands browser text 'testid=conversation-name-title'` (`QA_F07 renamed (branch)`) and `control-openhands browser count 'testid=chat-interface >> testid=user-message'` (`1`, the history is copied up to that reply). Back on `/conversations/<id>` (`control-openhands browser goto /conversations/<id>`), `control-openhands browser hover 'testid=user-message'` and `control-openhands browser click 'testid=user-message >> role=button[name=\"Branch from here\"]' --expect-url '/conversations/(?!<id>)[0-9a-f-]+'` (it prints `/conversations/<branch-id>`): the new branch has `0` user messages and `control-openhands browser text 'testid=chat-input'` reads the original prompt `Reply with the single word: pong`, still there a few seconds later and not sent (`control-openhands conversation events <branch-id> --kinds MessageEvent` has count `0`).", "source_line": 70, "expected": [ "Branch from here on a message forks the conversation as `<title> (branch)`; on a user message the message is left out and its text goes back into the composer." ], "notes": [ "After the user-message branch in `F07.branch-from-message`, Send stays disabled next to the prefilled text (`browser enabled 'testid=submit-button'` is `false`; Known failure #18202, see F05 Gotchas). Enter in the field sends it." ], "annotations": [], "states": [ "failure" ] }, { "name": "F07--delete-with-confirmation", "title": "Delete with confirmation", "ids": [ "F07.delete-confirm" ], "body": "On a disposable conversation (for example the branch from `F07.branch-from-message`) run `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'` and `control-openhands browser snapshot 'role=dialog'`: `Confirm Delete Are you sure you want to delete the \"<title>\" conversation? This action cannot be undone.` with `Cancel` (`cancel-button`) and `Confirm Delete` (`confirm-button`). `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps the URL. Repeat the two clicks, then `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations$'`; `control-openhands api GET '/api/conversations?ids=<id>'` returns `[null]` and `browser goto /conversations/<id>` redirects as in `F07.missing-redirect`.", "raw": "**Delete with confirmation (`F07.delete-confirm`).** On a disposable conversation (for example the branch from `F07.branch-from-message`) run `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'`, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'` and `control-openhands browser snapshot 'role=dialog'`: `Confirm Delete Are you sure you want to delete the \"<title>\" conversation? This action cannot be undone.` with `Cancel` (`cancel-button`) and `Confirm Delete` (`confirm-button`). `control-openhands browser click 'role=dialog >> testid=cancel-button'` keeps the URL. Repeat the two clicks, then `control-openhands browser click 'role=dialog >> testid=confirm-button' --expect-url '/conversations$'`; `control-openhands api GET '/api/conversations?ids=<id>'` returns `[null]` and `browser goto /conversations/<id>` redirects as in `F07.missing-redirect`.", "source_line": 71, "expected": [ "Delete Conversation names the title in a confirmation; Cancel keeps it, Confirm Delete removes it and returns to `/conversations`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--paused-conversation", "title": "Paused conversation", "ids": [ "F07.menu-items-by-state", "F07.status-menu" ], "body": "Run `control-openhands browser goto /conversations/<running-id>`, open the \"...\" menu and `control-openhands browser testids 'testid=conversation-name-context-menu'`: `show-hooks-button` and `stop-button` are absent. Close it with `control-openhands browser mouse-click 900 500`. Run `control-openhands browser click 'testid=server-status-menu-trigger'` and `control-openhands browser text 'testid=server-status-context-menu'` (`Server Stopped` / `Start Runtime`), then `control-openhands browser click 'testid=server-status-context-menu >> testid=start-server-button'` and `control-openhands conversation wait <running-id> --until running,finished,idle --timeout 60`. The status is `running` again and the menu reads `Running` / `Stop Runtime`.", "raw": "**Paused conversation (`F07.menu-items-by-state`, `F07.status-menu`).** Run `control-openhands browser goto /conversations/<running-id>`, open the \"...\" menu and `control-openhands browser testids 'testid=conversation-name-context-menu'`: `show-hooks-button` and `stop-button` are absent. Close it with `control-openhands browser mouse-click 900 500`. Run `control-openhands browser click 'testid=server-status-menu-trigger'` and `control-openhands browser text 'testid=server-status-context-menu'` (`Server Stopped` / `Start Runtime`), then `control-openhands browser click 'testid=server-status-context-menu >> testid=start-server-button'` and `control-openhands conversation wait <running-id> --until running,finished,idle --timeout 60`. The status is `running` again and the menu reads `Running` / `Stop Runtime`.", "source_line": 72, "expected": [ "Show Available Hooks and Stop Conversation appear only while the conversation is active.", "hovering the status dot (or tapping it on touch) opens a menu with the state and Stop Runtime (active) or Start Runtime (paused)." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#4893](https://github.com/OpenHands/software-agent-sdk/issues/4893) (`F07.status-menu`): A sandbox frozen mid-run can resume with a stale Running state that rejects normal resume requests. Live prerequisite: A disposable sandbox/VM frozen mid-run with a live model and persisted ownership state." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--hooks", "title": "Hooks", "ids": [ "F07.hooks-modal" ], "body": "On `/conversations/<running-id>` (no workspace attached; Paused conversation left the browser there) open the \"...\" menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-hooks-button'` and `control-openhands browser text 'testid=hooks-modal'`. It reads `Available Hooks`, the workspace explanation and `No hooks configured for this conversation.` (the backend workspace root has no `hooks.json`). Run `control-openhands browser network --clear` and `control-openhands browser click 'testid=hooks-modal >> testid=refresh-hooks'`: the text stays and `control-openhands browser network` shows `POST /api/hooks`. `control-openhands browser click 'testid=close-hooks-modal'` closes it.", "raw": "**Hooks (`F07.hooks-modal`).** On `/conversations/<running-id>` (no workspace attached; Paused conversation left the browser there) open the \"...\" menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-hooks-button'` and `control-openhands browser text 'testid=hooks-modal'`. It reads `Available Hooks`, the workspace explanation and `No hooks configured for this conversation.` (the backend workspace root has no `hooks.json`). Run `control-openhands browser network --clear` and `control-openhands browser click 'testid=hooks-modal >> testid=refresh-hooks'`: the text stays and `control-openhands browser network` shows `POST /api/hooks`. `control-openhands browser click 'testid=close-hooks-modal'` closes it.", "source_line": 73, "expected": [ "Show Available Hooks lists the workspace's `.openhands/hooks.json` hooks by event (read when it first opens for a conversation after a page load, and on Refresh; a reopen within 5 minutes of the last read shows the cached list), or \"No hooks configured for this conversation.\"" ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--skills", "title": "Skills", "ids": [ "F07.skills-modal" ], "body": "Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-skills-button'`, `control-openhands browser snapshot 'role=dialog'` (`Public skills 12` and one button per skill), `control-openhands browser click 'testid=skills-modal >> role=button[name=\"github Knowledge\"]'`; the row expands to `Triggers` and `Content`. `control-openhands browser press Escape` closes it (`browser count 'testid=skills-modal'` is `0`).", "raw": "**Skills (`F07.skills-modal`).** Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-skills-button'`, `control-openhands browser snapshot 'role=dialog'` (`Public skills 12` and one button per skill), `control-openhands browser click 'testid=skills-modal >> role=button[name=\"github Knowledge\"]'`; the row expands to `Triggers` and `Content`. `control-openhands browser press Escape` closes it (`browser count 'testid=skills-modal'` is `0`).", "source_line": 74, "expected": [ "Show Available Skills lists the loaded skills with expandable rows and Refresh." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--project-skills-and-configured-hooks", "title": "Project skills and configured hooks", "ids": [ "F07.skills-modal-project", "F07.hooks-modal" ], "body": "Arrange (outside the UI) a project skill and a hook in the git fixture: `mkdir -p <run>/workspace/qa-f07-repo/.agents/skills/qa-f07-skill <run>/workspace/qa-f07-repo/.openhands`, write `<run>/workspace/qa-f07-repo/.agents/skills/qa-f07-skill/SKILL.md` with front matter `name: qa-f07-skill`, `description: QA fixture skill for F07.`, `triggers: [qaf07trigger]` and the body `QA_F07 skill body.`, and write `<run>/workspace/qa-f07-repo/.openhands/hooks.json` as `{\"pre_tool_use\":[{\"matcher\":\"terminal\",\"hooks\":[{\"type\":\"command\",\"command\":\"true\",\"timeout\":10}]}]}`. Then `control-openhands conversation start --workspace qa-f07-repo --prompt \"Reply with the single word: hi\" --wait --timeout 240` prints `<hooks-id>`; `control-openhands api GET /api/conversations/<hooks-id>` shows that `hook_config.pre_tool_use` holds the `terminal` matcher. On `/conversations/<hooks-id>` open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-skills-button'` and `control-openhands browser snapshot 'role=dialog'`: `Project skills 1` with `button \"qa-f07-skill AgentSkills\"` above `Public skills 12`. `control-openhands browser click 'testid=skills-modal >> role=button[name=\"qa-f07-skill AgentSkills\"]'` expands it to `Triggers qaf07trigger Content QA_F07 skill body.`; `control-openhands browser click 'testid=skills-modal >> testid=refresh-skills'` re-reads the workspace (`browser network` shows `POST /api/skills`). Close with `control-openhands browser press Escape`. Hooks: open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-hooks-button'` and `control-openhands browser snapshot 'testid=hooks-modal'`: one collapsed event, `button \"Pre Tool Use 1 hook\"`. `control-openhands browser click 'testid=hooks-modal >> role=button[name=\"Pre Tool Use 1 hook\"]'` expands it to `Matcher terminal Commands true Type: command Timeout: 10s` (`control-openhands browser screenshot --feature F07.hooks-modal --name workspace-hooks`). Close it with `control-openhands browser click 'testid=close-hooks-modal'` and write `<run>/workspace/qa-f07-repo/.openhands/hooks.json` as `{\"pre_tool_use\":[{\"matcher\":\"terminal\",\"hooks\":[{\"type\":\"command\",\"command\":\"true\",\"timeout\":10}]}],\"stop\":[{\"hooks\":[{\"type\":\"command\",\"command\":\"echo qa-f07-stop\"}]}]}`. A reopen keeps the list it read: run `control-openhands browser network --clear`, open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-hooks-button'` again, then `control-openhands browser network --filter /api/hooks` (no request) and `control-openhands browser snapshot 'testid=hooks-modal'` (still only `button \"Pre Tool Use 1 hook\"`). Refresh re-reads the file: `control-openhands browser click 'testid=hooks-modal >> testid=refresh-hooks'`, then `control-openhands browser network --filter /api/hooks` shows `POST /api/hooks` and the snapshot adds `button \"Stop 1 hook\"`; `control-openhands browser click 'testid=hooks-modal >> role=button[name=\"Stop 1 hook\"]'` expands it to `Matcher * Commands echo qa-f07-stop Type: command Timeout: 60s` (the SDK defaults). Close with `control-openhands browser click 'testid=close-hooks-modal'`.", "raw": "**Project skills and configured hooks (`F07.skills-modal-project`, `F07.hooks-modal`).** Arrange (outside the UI) a project skill and a hook in the git fixture: `mkdir -p <run>/workspace/qa-f07-repo/.agents/skills/qa-f07-skill <run>/workspace/qa-f07-repo/.openhands`, write `<run>/workspace/qa-f07-repo/.agents/skills/qa-f07-skill/SKILL.md` with front matter `name: qa-f07-skill`, `description: QA fixture skill for F07.`, `triggers: [qaf07trigger]` and the body `QA_F07 skill body.`, and write `<run>/workspace/qa-f07-repo/.openhands/hooks.json` as `{\"pre_tool_use\":[{\"matcher\":\"terminal\",\"hooks\":[{\"type\":\"command\",\"command\":\"true\",\"timeout\":10}]}]}`. Then `control-openhands conversation start --workspace qa-f07-repo --prompt \"Reply with the single word: hi\" --wait --timeout 240` prints `<hooks-id>`; `control-openhands api GET /api/conversations/<hooks-id>` shows that `hook_config.pre_tool_use` holds the `terminal` matcher. On `/conversations/<hooks-id>` open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-skills-button'` and `control-openhands browser snapshot 'role=dialog'`: `Project skills 1` with `button \"qa-f07-skill AgentSkills\"` above `Public skills 12`. `control-openhands browser click 'testid=skills-modal >> role=button[name=\"qa-f07-skill AgentSkills\"]'` expands it to `Triggers qaf07trigger Content QA_F07 skill body.`; `control-openhands browser click 'testid=skills-modal >> testid=refresh-skills'` re-reads the workspace (`browser network` shows `POST /api/skills`). Close with `control-openhands browser press Escape`. Hooks: open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-hooks-button'` and `control-openhands browser snapshot 'testid=hooks-modal'`: one collapsed event, `button \"Pre Tool Use 1 hook\"`. `control-openhands browser click 'testid=hooks-modal >> role=button[name=\"Pre Tool Use 1 hook\"]'` expands it to `Matcher terminal Commands true Type: command Timeout: 10s` (`control-openhands browser screenshot --feature F07.hooks-modal --name workspace-hooks`). Close it with `control-openhands browser click 'testid=close-hooks-modal'` and write `<run>/workspace/qa-f07-repo/.openhands/hooks.json` as `{\"pre_tool_use\":[{\"matcher\":\"terminal\",\"hooks\":[{\"type\":\"command\",\"command\":\"true\",\"timeout\":10}]}],\"stop\":[{\"hooks\":[{\"type\":\"command\",\"command\":\"echo qa-f07-stop\"}]}]}`. A reopen keeps the list it read: run `control-openhands browser network --clear`, open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-hooks-button'` again, then `control-openhands browser network --filter /api/hooks` (no request) and `control-openhands browser snapshot 'testid=hooks-modal'` (still only `button \"Pre Tool Use 1 hook\"`). Refresh re-reads the file: `control-openhands browser click 'testid=hooks-modal >> testid=refresh-hooks'`, then `control-openhands browser network --filter /api/hooks` shows `POST /api/hooks` and the snapshot adds `button \"Stop 1 hook\"`; `control-openhands browser click 'testid=hooks-modal >> role=button[name=\"Stop 1 hook\"]'` expands it to `Matcher * Commands echo qa-f07-stop Type: command Timeout: 60s` (the SDK defaults). Close with `control-openhands browser click 'testid=close-hooks-modal'`.", "source_line": 75, "expected": [ "a workspace with `.agents/skills/<name>/SKILL.md` adds a \"Project skills\" section above the public ones; each row expands to its triggers and body. The list is read with the page and on Refresh; opening the dialog within 10 minutes of the last read shows the cached list.", "Show Available Hooks lists the workspace's `.openhands/hooks.json` hooks by event (read when it first opens for a conversation after a page load, and on Refresh; a reopen within 5 minutes of the last read shows the cached list), or \"No hooks configured for this conversation.\"" ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--agent-tools", "title": "Agent tools", "ids": [ "F07.agent-tools-modal" ], "body": "Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-agent-tools-button'`; the dialog `testid=system-message-modal` has tabs `System Message` (selected, the system prompt) and `Available Tools`. `control-openhands browser click 'role=dialog >> role=tab[name=\"Available Tools\"]'`, `control-openhands browser count 'testid=system-message-modal >> testid=toggle-button'` (backend-provided count; `10` on Agent Server 1.54.0: terminal, file_editor, task_tracker, canvas_ui_control, launch_child_conversation, request_secret (local backends, see `F14.agent-request`), finish, think, switch_llm, invoke_skill), `control-openhands browser click 'testid=system-message-modal >> testid=toggle-button >> nth=0'` expands `terminal` with its description. Close with `control-openhands browser click 'testid=close-system-message-modal'`.", "raw": "**Agent tools (`F07.agent-tools-modal`).** Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=show-agent-tools-button'`; the dialog `testid=system-message-modal` has tabs `System Message` (selected, the system prompt) and `Available Tools`. `control-openhands browser click 'role=dialog >> role=tab[name=\"Available Tools\"]'`, `control-openhands browser count 'testid=system-message-modal >> testid=toggle-button'` (backend-provided count; `10` on Agent Server 1.54.0: terminal, file_editor, task_tracker, canvas_ui_control, launch_child_conversation, request_secret (local backends, see `F14.agent-request`), finish, think, switch_llm, invoke_skill), `control-openhands browser click 'testid=system-message-modal >> testid=toggle-button >> nth=0'` expands `terminal` with its description. Close with `control-openhands browser click 'testid=close-system-message-modal'`.", "source_line": 76, "expected": [ "Show Agent Tools & Metadata shows the system message and the tool list." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--export-transcript", "title": "Export transcript", "ids": [ "F07.export-transcript" ], "body": "Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=export-transcript-button'`; `control-openhands browser snapshot 'role=dialog'` shows `Export conversation`, radio `Markdown document (.md)` checked, `Standalone web page (.html)`, and checkboxes `Include tool call details` and `Include timestamps` checked. `control-openhands browser click 'testid=confirm-transcript-export'` then `control-openhands browser downloads --last 1 --inspect`: a `<ms>-conversation-<id>.md` (the harness prefixes a timestamp to the suggested `conversation-<id>.md`) lands in `<run>/private/downloads` starting `# QA_F07 renamed`, `**Model:** deepseek/deepseek-flash`, `## User` with `<sub>` timestamps. Reopen, `control-openhands browser click 'role=dialog >> role=radio[name=\"Standalone web page (.html)\"]'`, `control-openhands browser click 'role=dialog >> role=checkbox[name=\"Include timestamps\"]'`, export: the `.html` has `<title>QA_F07 renamed` and no timestamps. Reopen once more: `control-openhands browser click 'testid=cancel-transcript-export'` closes it without a download (the file count in `/private/downloads` is unchanged); `browser press Escape` and `testid=close-transcript-export-modal` close it too.", "raw": "**Export transcript (`F07.export-transcript`).** Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=export-transcript-button'`; `control-openhands browser snapshot 'role=dialog'` shows `Export conversation`, radio `Markdown document (.md)` checked, `Standalone web page (.html)`, and checkboxes `Include tool call details` and `Include timestamps` checked. `control-openhands browser click 'testid=confirm-transcript-export'` then `control-openhands browser downloads --last 1 --inspect`: a `-conversation-.md` (the harness prefixes a timestamp to the suggested `conversation-.md`) lands in `/private/downloads` starting `# QA_F07 renamed`, `**Model:** deepseek/deepseek-flash`, `## User` with `` timestamps. Reopen, `control-openhands browser click 'role=dialog >> role=radio[name=\"Standalone web page (.html)\"]'`, `control-openhands browser click 'role=dialog >> role=checkbox[name=\"Include timestamps\"]'`, export: the `.html` has `QA_F07 renamed` and no timestamps. Reopen once more: `control-openhands browser click 'testid=cancel-transcript-export'` closes it without a download (the file count in `/private/downloads` is unchanged); `browser press Escape` and `testid=close-transcript-export-modal` close it too.", "source_line": 77, "expected": [ "Export… downloads a Markdown or HTML transcript with optional tool details and timestamps." ], "notes": [ "Known issue [#17494](https://github.com/OpenHands/OpenHands/issues/17494) (`F07.export-transcript`): Transcript export leaks raw i18n template for ACP tool calls without a title. Live prerequisite: An authenticated ACP session that emits a titleless tool call, then Markdown and HTML export." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--download-data", "title": "Download data", "ids": [ "F07.download-zip" ], "body": "Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=download-trajectory-button'`, then `control-openhands browser downloads`. A `-conversation_.zip` lands in `/private/downloads`; `browser downloads --last 1 --inspect` lists `/base_state.json` and `bash_events/`, and every LLM `api_key` in `base_state.json` is masked as asterisks (`**********`).", "raw": "**Download data (`F07.download-zip`).** Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=download-trajectory-button'`, then `control-openhands browser downloads`. A `-conversation_.zip` lands in `/private/downloads`; `browser downloads --last 1 --inspect` lists `/base_state.json` and `bash_events/`, and every LLM `api_key` in `base_state.json` is masked as asterisks (`**********`).", "source_line": 78, "expected": [ "Download conversation data downloads `conversation_.zip`." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5640](https://github.com/OpenHands/software-agent-sdk/issues/5640) (`F07.download-zip`): Download conversation data can return404 after a warm-pool server initializes a custom conversation-storage path. Although closed upstream, the pinned SDK 1.55.0 still reads the default configuration path here; this remains a supported-version distinction, not a freshly reproduced failure. Live prerequisite: A deferred-init warm-pool Agent Server initialized with a non-default conversations_path; this launcher starts an already-initialized server.", "Known failure [OpenHands/software-agent-sdk#5539](https://github.com/OpenHands/software-agent-sdk/issues/5539) (`F07.download-zip`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0: three real overlapping receivers of one idle conversation made 929 requests in 20 seconds, including 31 completed corrupt ZIPs with status 200, 106 server 500s, 269 ingress 502s and 133 RemoteProtocolErrors. A sequential 2,000-request control returned valid ZIPs. Recipe (use the pinned SDK Python with httpx): `python .agents/skills/verify-openhands/scripts/fixtures/download-trajectory-overlap.py --base-url --conversation-id --api-key-file /private/session-key --output-dir /private/trajectory-overlap --seconds 20 --clients 3 --rounds 5000 --slow-ms 2`. It enforces loopback, request/duration/network/uncompressed/member budgets and bounded CRC reads; aborts and unsupported archives remain unverified. While it runs, use `browser goto /conversations/`, open `chat-pane-header >> ellipsis-button`, click `download-trajectory-button`, then `browser downloads --last 3 --inspect` (all CLI commands with `--run `). That Canvas click downloaded an inspectable ZIP; its particular result is not claimed corrupt. Private raw samples stay private. Evidence: `backfill-18309-surface-rechecks/F07.download-zip/trajectory-overlap-5539-summary.json` and `overlapping-download-5539.png`. The summary correction excludes two ambiguous Content-Length differences that could be intentional deadline aborts; only completed corrupt ZIPs count as corrupt downloads." ], "annotations": [], "states": [ "failure", "issue", "blocked" ] }, { "name": "F07--usage", "title": "Usage", "ids": [ "F07.display-cost" ], "body": "Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=display-cost-button'`, `control-openhands browser wait 'testid=usage-panel'` and `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`true`). The panel shows Context Window, Token Usage and Total Cost (contents are F27).", "raw": "**Usage (`F07.display-cost`).** Open the menu, `control-openhands browser click 'testid=conversation-name-context-menu >> testid=display-cost-button'`, `control-openhands browser wait 'testid=usage-panel'` and `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`true`). The panel shows Context Window, Token Usage and Total Cost (contents are F27).", "source_line": 79, "expected": [ "Display Usage and Cost opens the right panel on the Usage tab." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--right-panel", "title": "Right panel", "ids": [ "F07.right-panel-toggle" ], "body": "Run `control-openhands browser click 'testid=right-panel-toggle'` and `control-openhands browser attr 'testid=right-panel-toggle' aria-label`: it alternates `Show panel` / `Hide panel`, and `control-openhands browser wait 'testid=tabs-pane-header' --state hidden --timeout 5000` (after `Show panel`) or `--state visible` (after `Hide panel`) follows; a plain `browser visible` right after the click still reads the old value while the panel animates. With the panel open, `control-openhands browser reload` and `browser attr 'testid=right-panel-toggle' aria-pressed` reads `false` (closed by design; the selected tab is remembered).", "raw": "**Right panel (`F07.right-panel-toggle`).** Run `control-openhands browser click 'testid=right-panel-toggle'` and `control-openhands browser attr 'testid=right-panel-toggle' aria-label`: it alternates `Show panel` / `Hide panel`, and `control-openhands browser wait 'testid=tabs-pane-header' --state hidden --timeout 5000` (after `Show panel`) or `--state visible` (after `Hide panel`) follows; a plain `browser visible` right after the click still reads the old value while the panel animates. With the panel open, `control-openhands browser reload` and `browser attr 'testid=right-panel-toggle' aria-pressed` reads `false` (closed by design; the selected tab is remembered).", "source_line": 80, "expected": [ "the header panel button shows and hides the right tabs panel (closed again after a reload)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--overview", "title": "Overview", "ids": [ "F07.overview-toggle" ], "body": "Run `control-openhands browser click 'testid=conversation-overview-toggle'`, `control-openhands browser visible 'testid=conversation-overview-panel'` (`true`) and `browser attr 'testid=conversation-overview-toggle' aria-label` (`Hide overview`). Then `control-openhands browser click 'testid=right-panel-toggle'`: `control-openhands browser wait 'testid=conversation-overview-column' --state hidden --timeout 5000` succeeds (an immediate `browser visible` can still read `true`). With the panel open, `control-openhands browser hover 'testid=conversation-overview-toggle'` and `control-openhands browser wait 'testid=conversation-overview-peek' --timeout 5000` shows the peek. Clicking the overview toggle now closes the panel (`aria-pressed` `false`) and shows the overview (`browser wait 'testid=conversation-overview-column' --state visible`). Click the overview toggle once more to restore the closed state.", "raw": "**Overview (`F07.overview-toggle`).** Run `control-openhands browser click 'testid=conversation-overview-toggle'`, `control-openhands browser visible 'testid=conversation-overview-panel'` (`true`) and `browser attr 'testid=conversation-overview-toggle' aria-label` (`Hide overview`). Then `control-openhands browser click 'testid=right-panel-toggle'`: `control-openhands browser wait 'testid=conversation-overview-column' --state hidden --timeout 5000` succeeds (an immediate `browser visible` can still read `true`). With the panel open, `control-openhands browser hover 'testid=conversation-overview-toggle'` and `control-openhands browser wait 'testid=conversation-overview-peek' --timeout 5000` shows the peek. Clicking the overview toggle now closes the panel (`aria-pressed` `false`) and shows the overview (`browser wait 'testid=conversation-overview-column' --state visible`). Click the overview toggle once more to restore the closed state.", "source_line": 81, "expected": [ "the header (i) button shows and hides the Overview column, which is exclusive with the right panel." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--git-actions", "title": "Git actions", "ids": [ "F07.git-actions-menu" ], "body": "Run `control-openhands browser click 'testid=conversation-git-actions-toggle'` and `control-openhands browser testids 'testid=conversation-git-actions-menu'` (`-commit`, `-pull`, `-push`, `-create-pr`, `-create-new-branch`, labels `Commits`, `Pull`, `Push`, `Create PR`, `Create New Branch`). `control-openhands browser click 'testid=conversation-git-actions-menu >> testid=conversation-git-actions-commit'` then `control-openhands browser text 'testid=chat-input'`: `Please review the current changes and create a git commit with a concise, descriptive message.`; `conversation events --kinds MessageEvent` shows no new message. `control-openhands browser enabled 'testid=submit-button'` should be `true`, without typing. Each item replaces the composer text: `-pull` gives `Please pull the latest code from the repository.`, `-create-new-branch` gives `Please create a new branch with a descriptive name related to the work you plan to do.`, and the Push and Create PR prompts name `GitHub` by default. `control-openhands browser press Escape` closes the open menu. Clear the composer with `control-openhands browser fill 'testid=chat-input' ''`. Known failure ([#18202](https://github.com/OpenHands/OpenHands/issues/18202), reproduced 2026-10-09 on `main@3500d9e5`): after Pull, Send stays disabled until the user types.", "raw": "**Git actions (`F07.git-actions-menu`).** Run `control-openhands browser click 'testid=conversation-git-actions-toggle'` and `control-openhands browser testids 'testid=conversation-git-actions-menu'` (`-commit`, `-pull`, `-push`, `-create-pr`, `-create-new-branch`, labels `Commits`, `Pull`, `Push`, `Create PR`, `Create New Branch`). `control-openhands browser click 'testid=conversation-git-actions-menu >> testid=conversation-git-actions-commit'` then `control-openhands browser text 'testid=chat-input'`: `Please review the current changes and create a git commit with a concise, descriptive message.`; `conversation events --kinds MessageEvent` shows no new message. `control-openhands browser enabled 'testid=submit-button'` should be `true`, without typing. Each item replaces the composer text: `-pull` gives `Please pull the latest code from the repository.`, `-create-new-branch` gives `Please create a new branch with a descriptive name related to the work you plan to do.`, and the Push and Create PR prompts name `GitHub` by default. `control-openhands browser press Escape` closes the open menu. Clear the composer with `control-openhands browser fill 'testid=chat-input' ''`. Known failure ([#18202](https://github.com/OpenHands/OpenHands/issues/18202), reproduced 2026-10-09 on `main@3500d9e5`): after Pull, Send stays disabled until the user types.", "source_line": 82, "expected": [ "the header Git actions menu offers Commits, Pull, Push, Create PR and Create New Branch; each pre-fills the composer without sending, and Send becomes enabled so the user can submit or edit the prompt." ], "notes": [], "annotations": [ "Known failure ([#18202](https://github.com/OpenHands/OpenHands/issues/18202), reproduced 2026-10-09 on `main@3500d9e5`): after Pull, Send stays disabled until the user types." ], "states": [ "failure" ] }, { "name": "F07--panel-route", "title": "Panel route", "ids": [ "F07.panel-route" ], "body": "Run `control-openhands browser viewport phone`, open `/conversations/` (`control-openhands browser goto /conversations/`), `control-openhands browser click 'testid=right-panel-toggle' --expect-url '/panel$'` and `control-openhands browser testids --filter mobile` (`conversation-mobile-panel-top`, `conversation-mobile-panel-back`), then `control-openhands browser click 'testid=conversation-mobile-panel-back' --expect-url '/conversations/[0-9a-f-]+$'`. Direct entry at phone width: `control-openhands browser goto /conversations//panel` and `control-openhands browser wait 'testid=conversation-mobile-panel-back'` succeed (`control-openhands browser screenshot --feature F07.panel-route --name phone-direct`). The panel page serves every width up to 1024 px: after `control-openhands browser viewport tablet` the URL still ends `/panel` and `control-openhands browser count 'testid=conversation-mobile-panel-back'` is `1`. Wider, there is no panel page: run `control-openhands browser viewport desktop`, then `control-openhands browser wait-url '/conversations/[0-9a-f-]+$'` (the app replaces `/panel` with `/conversations/`), `control-openhands browser count 'testid=conversation-mobile-panel-back'` (`0`) and `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`true`: the drawer is open beside the chat). A desktop-width visit lands the same way (#18045): `control-openhands browser goto /conversations//panel`, `control-openhands browser wait-url '/conversations/[0-9a-f-]+$'`, the same count (`0`) and `aria-pressed` (`true`); `control-openhands browser screenshot --feature F07.panel-route --name desktop-redirect` shows the chat with the drawer open beside it.", "raw": "**Panel route (`F07.panel-route`).** Run `control-openhands browser viewport phone`, open `/conversations/` (`control-openhands browser goto /conversations/`), `control-openhands browser click 'testid=right-panel-toggle' --expect-url '/panel$'` and `control-openhands browser testids --filter mobile` (`conversation-mobile-panel-top`, `conversation-mobile-panel-back`), then `control-openhands browser click 'testid=conversation-mobile-panel-back' --expect-url '/conversations/[0-9a-f-]+$'`. Direct entry at phone width: `control-openhands browser goto /conversations//panel` and `control-openhands browser wait 'testid=conversation-mobile-panel-back'` succeed (`control-openhands browser screenshot --feature F07.panel-route --name phone-direct`). The panel page serves every width up to 1024 px: after `control-openhands browser viewport tablet` the URL still ends `/panel` and `control-openhands browser count 'testid=conversation-mobile-panel-back'` is `1`. Wider, there is no panel page: run `control-openhands browser viewport desktop`, then `control-openhands browser wait-url '/conversations/[0-9a-f-]+$'` (the app replaces `/panel` with `/conversations/`), `control-openhands browser count 'testid=conversation-mobile-panel-back'` (`0`) and `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`true`: the drawer is open beside the chat). A desktop-width visit lands the same way (#18045): `control-openhands browser goto /conversations//panel`, `control-openhands browser wait-url '/conversations/[0-9a-f-]+$'`, the same count (`0`) and `aria-pressed` (`true`); `control-openhands browser screenshot --feature F07.panel-route --name desktop-redirect` shows the chat with the drawer open beside it.", "source_line": 83, "expected": [ "at phone width (up to 1024 px) the header panel button opens `/conversations//panel`, the tabs panel full width with a Back button; wider, that URL redirects to `/conversations/` with the drawer open beside the chat." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--git-control-bar", "title": "Git control bar", "ids": [ "F07.git-control-bar" ], "body": "On `/conversations/` (`control-openhands browser goto /conversations/`; Panel route ends on `/conversations/`) run `control-openhands browser snapshot 'testid=interactive-chat-box'`: `button \"qa-f07-repo\" [disabled]` and the branch text `main`, no Pull/Push. Add a GitHub origin to the fixture (arrange step, outside the UI): `git -C /workspace/qa-f07-repo remote add origin https://github.com/qa-example/qa-f07-repo.git`. Then `control-openhands browser reload` and the snapshot again: link `qa-example/qa-f07-repo` (`https://github.com/qa-example/qa-f07-repo`), link `main` (`…/tree/main`) and buttons `Pull`, `Push`, `Pull Request`.", "raw": "**Git control bar (`F07.git-control-bar`).** On `/conversations/` (`control-openhands browser goto /conversations/`; Panel route ends on `/conversations/`) run `control-openhands browser snapshot 'testid=interactive-chat-box'`: `button \"qa-f07-repo\" [disabled]` and the branch text `main`, no Pull/Push. Add a GitHub origin to the fixture (arrange step, outside the UI): `git -C /workspace/qa-f07-repo remote add origin https://github.com/qa-example/qa-f07-repo.git`. Then `control-openhands browser reload` and the snapshot again: link `qa-example/qa-f07-repo` (`https://github.com/qa-example/qa-f07-repo`), link `main` (`…/tree/main`) and buttons `Pull`, `Push`, `Pull Request`.", "source_line": 84, "expected": [ "under the composer, a folder workspace shows an inert workspace chip and the branch; with a GitHub `origin` the repo and branch chips link to GitHub and Pull, Push and Pull Request appear." ], "notes": [ "Known issue [#15543](https://github.com/OpenHands/OpenHands/issues/15543) (`F07.git-control-bar`): Bad parsing for branch retrieval. Live prerequisite: The original Git probe stdout/interleaving that temporarily puts a filename in the branch badge; dirty state alone is not a deterministic trigger." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--pull-sends-at-once", "title": "Pull sends at once", "ids": [ "F07.git-control-bar-send" ], "body": "Run `control-openhands browser click 'testid=interactive-chat-box >> role=button[name=\"Pull\"][exact]'` and `control-openhands browser wait 'testid=user-message >> has-text=Please pull the latest code' --timeout 15000`; `conversation status ` is `running`. Stop it right away with Stop Conversation as in `F07.stop-confirm` (the fake remote cannot be pulled). Push and Pull Request are blocked: they need a real remote and GitHub credentials.", "raw": "**Pull sends at once (`F07.git-control-bar-send`).** Run `control-openhands browser click 'testid=interactive-chat-box >> role=button[name=\"Pull\"][exact]'` and `control-openhands browser wait 'testid=user-message >> has-text=Please pull the latest code' --timeout 15000`; `conversation status ` is `running`. Stop it right away with Stop Conversation as in `F07.stop-confirm` (the fake remote cannot be pulled). Push and Pull Request are blocked: they need a real remote and GitHub credentials.", "source_line": 85, "expected": [ "the Pull, Push and Pull Request chips send their git prompt to the agent at once." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F07--cloud-only", "title": "Cloud-only", "ids": [ "F07.cloud-only" ], "body": "Blocked without a Cloud backend. On local, confirm the absence: the menu testids (see `F07.menu-open`) have no `share-publicly-button`, and the git control bar shows no `Connect Repo` chip. With Cloud: the menu's **Share publicly** switch (`share-publicly-button`, then `copy-share-link-button`, `open-share-link-button`), the repo chip **Connect Repo** opening the Open repository modal (`close-open-repository-modal`), `/conversations/task-` while a conversation starts, and the `Unavailable for Archives` tooltips (see F25).", "raw": "**Cloud-only (`F07.cloud-only`).** Blocked without a Cloud backend. On local, confirm the absence: the menu testids (see `F07.menu-open`) have no `share-publicly-button`, and the git control bar shows no `Connect Repo` chip. With Cloud: the menu's **Share publicly** switch (`share-publicly-button`, then `copy-share-link-button`, `open-share-link-button`), the repo chip **Connect Repo** opening the Open repository modal (`close-open-repository-modal`), `/conversations/task-` while a conversation starts, and the `Unavailable for Archives` tooltips (see F25).", "source_line": 86, "expected": [ "Share publicly, Connect Repo, `task-` start URLs, paused-sandbox resume, archived read-only state and the shared-conversation redirect exist only on a Cloud backend." ], "notes": [ "Known issue [#18136](https://github.com/OpenHands/OpenHands/issues/18136) (`F07.cloud-only`): Connect Repo on a cloud conversation does not save the repository on the server. Live prerequisite: An authenticated Cloud sandbox/repository provider and a second fresh browser to compare server repository metadata." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F07--phone", "title": "Phone", "ids": [ "F07.phone" ], "body": "Run `control-openhands browser viewport phone`, `control-openhands browser testids 'testid=chat-pane-header'` (adds `sidebar-mobile-menu-toggle`; all header controls visible), open each info dialog from the menu and `control-openhands browser bbox 'testid=skills-modal'` (likewise `hooks-modal`, `system-message-modal`): `insideViewport` is `true` and `pageHorizontalOverflow` `false`. Then `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'` and `control-openhands browser bbox 'testid=conversation-name-context-menu'`: inside the viewport too (`x` about 145, `width` about 237), and `control-openhands browser screenshot --feature F07.phone --name header-menu` shows every label whole (`Show Available Skills`, `Download conversation data`, `Delete Conversation`). `control-openhands browser press Escape` closes it. Return with `control-openhands browser viewport desktop`.", "raw": "**Phone (`F07.phone`).** Run `control-openhands browser viewport phone`, `control-openhands browser testids 'testid=chat-pane-header'` (adds `sidebar-mobile-menu-toggle`; all header controls visible), open each info dialog from the menu and `control-openhands browser bbox 'testid=skills-modal'` (likewise `hooks-modal`, `system-message-modal`): `insideViewport` is `true` and `pageHorizontalOverflow` `false`. Then `control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'` and `control-openhands browser bbox 'testid=conversation-name-context-menu'`: inside the viewport too (`x` about 145, `width` about 237), and `control-openhands browser screenshot --feature F07.phone --name header-menu` shows every label whole (`Show Available Skills`, `Download conversation data`, `Delete Conversation`). `control-openhands browser press Escape` closes it. Return with `control-openhands browser viewport desktop`.", "source_line": 87, "expected": [ "at 390 px the header keeps its controls and the info dialogs fit the viewport." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F07--whitespace-only-header-rename", "title": "Whitespace-only header rename", "ids": [ "F07.rename-inline" ], "body": "After the named-title check, run `control-openhands browser network --clear`, double-click `testid=conversation-name-title`, fill `testid=conversation-name-input` with one space, and `control-openhands browser press Enter`. Read `control-openhands browser network --filter '/api/conversations' --bodies`, then reload and read the title. Expected: restore the previous title with no write. Known failure (reproduced 2026-10-12, [#17061](https://github.com/OpenHands/OpenHands/issues/17061)): Canvas submits `PATCH` with an empty title. Agent Server 1.54.0 rejects it with 422 and restores the prior title; a persistently blank server title was not observed.", "raw": "**Whitespace-only header rename (`F07.rename-inline`).** After the named-title check, run `control-openhands browser network --clear`, double-click `testid=conversation-name-title`, fill `testid=conversation-name-input` with one space, and `control-openhands browser press Enter`. Read `control-openhands browser network --filter '/api/conversations' --bodies`, then reload and read the title. Expected: restore the previous title with no write. Known failure (reproduced 2026-10-12, [#17061](https://github.com/OpenHands/OpenHands/issues/17061)): Canvas submits `PATCH` with an empty title. Agent Server 1.54.0 rejects it with 422 and restores the prior title; a persistently blank server title was not observed.", "source_line": 89, "expected": [ "double-click the title (or menu Rename), Enter or blur saves with a toast; an empty value reverts." ], "notes": [ "Known failure [OpenHands/software-agent-sdk#5629](https://github.com/OpenHands/software-agent-sdk/issues/5629) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A second server's native first load rewrote a newer saved title with its startup copy; the connected Canvas header showed that older title. Recipe: create an idle conversation on A and PATCH its title to `QA_SHARED_BEFORE_B_5629_REPEAT`; run pinned SDK Python `python .agents/skills/verify-openhands/scripts/fixtures/shared-storage-server.py --run --seconds 300` in the background. It starts only a bounded no-editor SDK B sharing this run's conversation directory/cipher/session key; its private `server.json` supplies the port and owned operator/PGID. After B listens, PATCH A to `QA_SHARED_AFTER_B_5629_REPEAT` and confirm disk. Run `control-openhands service stop agent-server --run `, then `python .../fixtures/shared-storage-server.py --run --load-id `: native B `/events/count` returned 200 and disk reverted to BEFORE_B. Restart A only to serve Canvas, choose default-local and doctor; use the Add Backend recipe from [F25](F25-backends-and-cloud.md) to connect B at its recorded port with the private key file, then open its card: the header reads BEFORE_B. SIGTERM only the recorded helper operator and verify its port/PGID gone; the helper also bounds runtime and terminates remaining group descendants. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-repeat-before-b-load-5629.json`, `metadata-repeat-after-b-load-5629.json`, `shared-first-load-reverted-5629.png`. The native first-load request is explicit; stale catalog display alone is insufficient proof of this overwrite.", "Known failure [OpenHands/software-agent-sdk#4800](https://github.com/OpenHands/software-agent-sdk/issues/4800) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A refused rename changed the server's GET title while `meta.json` kept the previous title; graceful shutdown subsequently persisted the rejected title. Recipe: create an idle dummy-profile conversation, then double-click `conversation-name-title`, fill `conversation-name-input` with `QA_META_BEFORE_4800` and press Enter; verify GET/disk. Record the original mode and temporarily `chmod 0500 /private/state/dev_conversations/` to deny an atomic metadata replacement. Rename through that UI to `QA_META_REFUSED_4800`: the permission-denied toast/HTTP 500 appears, GET shows REFUSED, disk still shows BEFORE. Restore the exact original directory mode in `finally`, then `restart`, `doctor`, GET and reload (CLI with `--run `): REFUSED was persisted during graceful shutdown. Only the owned temporary directory is changed; evidence contains title/timestamp fields, not stored secrets. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-before-4800.json`, `metadata-denied-4800.json`, `refused-title-4800.png`, `metadata-after-graceful-restart-4800.json`. Tags/autotitle and ownership-conflict branches remain untested." ], "annotations": [ "Known failure (reproduced 2026-10-12, [#17061](https://github.com/OpenHands/OpenHands/issues/17061)): Canvas submits `PATCH` with an empty title. Agent Server 1.54.0 rejects it with 422 and restores the prior title; a persistently blank server title was not observed." ], "states": [ "failure" ] }, { "name": "F07--already-deleted-conversation", "title": "Already-deleted conversation", "ids": [ "F07.delete-confirm", "F07.rename-inline" ], "body": "Expected: a stale action reports that the conversation is missing, and a stale rename never reports success. Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5800](https://github.com/OpenHands/software-agent-sdk/issues/5800)): confirming Delete after another client removed this disposable conversation returns400 and displays `Bad Request`, instead of a missing-conversation404. The separate stale-rename control receives PATCH200 but re-fetches missing metadata and displays not-found, so its false-success variant is prevented by Canvas. Pause/interrupt were not tested.\n ```sh known-bug F07.delete-confirm already-deleted\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n workspace = fixture / \"workspace\"\n workspace.mkdir()\n print(create(workspace))\n PY\n )\"\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'\n control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'\n control-openhands assert --field status --equals 200 -- api DELETE \"/api/conversations/$QA_ID\" --write\n control-openhands browser network --clear\n control-openhands browser click 'role=dialog >> testid=confirm-button' --observe 'role=status' --observe-ms 1000\n control-openhands browser screenshot --feature F07.delete-confirm --name stale-delete\n control-openhands assert --field recent.0.status --equals 404 --bug-shows 400 -- browser network --filter \"/api/conversations/$QA_ID\" # bug\n ```", "raw": "**Already-deleted conversation (`F07.delete-confirm`, `F07.rename-inline`).** Expected: a stale action reports that the conversation is missing, and a stale rename never reports success. Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5800](https://github.com/OpenHands/software-agent-sdk/issues/5800)): confirming Delete after another client removed this disposable conversation returns400 and displays `Bad Request`, instead of a missing-conversation404. The separate stale-rename control receives PATCH200 but re-fetches missing metadata and displays not-found, so its false-success variant is prevented by Canvas. Pause/interrupt were not tested.\n ```sh known-bug F07.delete-confirm already-deleted\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n workspace = fixture / \"workspace\"\n workspace.mkdir()\n print(create(workspace))\n PY\n )\"\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser click 'testid=chat-pane-header >> testid=ellipsis-button'\n control-openhands browser click 'testid=conversation-name-context-menu >> testid=delete-button'\n control-openhands assert --field status --equals 200 -- api DELETE \"/api/conversations/$QA_ID\" --write\n control-openhands browser network --clear\n control-openhands browser click 'role=dialog >> testid=confirm-button' --observe 'role=status' --observe-ms 1000\n control-openhands browser screenshot --feature F07.delete-confirm --name stale-delete\n control-openhands assert --field recent.0.status --equals 404 --bug-shows 400 -- browser network --filter \"/api/conversations/$QA_ID\" # bug\n ```", "source_line": 91, "expected": [ "Delete Conversation names the title in a confirmation; Cancel keeps it, Confirm Delete removes it and returns to `/conversations`.", "double-click the title (or menu Rename), Enter or blur saves with a toast; an empty value reverts." ], "notes": [ "Known failure [OpenHands/software-agent-sdk#5629](https://github.com/OpenHands/software-agent-sdk/issues/5629) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A second server's native first load rewrote a newer saved title with its startup copy; the connected Canvas header showed that older title. Recipe: create an idle conversation on A and PATCH its title to `QA_SHARED_BEFORE_B_5629_REPEAT`; run pinned SDK Python `python .agents/skills/verify-openhands/scripts/fixtures/shared-storage-server.py --run --seconds 300` in the background. It starts only a bounded no-editor SDK B sharing this run's conversation directory/cipher/session key; its private `server.json` supplies the port and owned operator/PGID. After B listens, PATCH A to `QA_SHARED_AFTER_B_5629_REPEAT` and confirm disk. Run `control-openhands service stop agent-server --run `, then `python .../fixtures/shared-storage-server.py --run --load-id `: native B `/events/count` returned 200 and disk reverted to BEFORE_B. Restart A only to serve Canvas, choose default-local and doctor; use the Add Backend recipe from [F25](F25-backends-and-cloud.md) to connect B at its recorded port with the private key file, then open its card: the header reads BEFORE_B. SIGTERM only the recorded helper operator and verify its port/PGID gone; the helper also bounds runtime and terminates remaining group descendants. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-repeat-before-b-load-5629.json`, `metadata-repeat-after-b-load-5629.json`, `shared-first-load-reverted-5629.png`. The native first-load request is explicit; stale catalog display alone is insufficient proof of this overwrite.", "Known failure [OpenHands/software-agent-sdk#4800](https://github.com/OpenHands/software-agent-sdk/issues/4800) (`F07.rename-inline`), reproduced 2026-10-12 at `fa1f724ab573`, SDK 1.54.0. A refused rename changed the server's GET title while `meta.json` kept the previous title; graceful shutdown subsequently persisted the rejected title. Recipe: create an idle dummy-profile conversation, then double-click `conversation-name-title`, fill `conversation-name-input` with `QA_META_BEFORE_4800` and press Enter; verify GET/disk. Record the original mode and temporarily `chmod 0500 /private/state/dev_conversations/` to deny an atomic metadata replacement. Rename through that UI to `QA_META_REFUSED_4800`: the permission-denied toast/HTTP 500 appears, GET shows REFUSED, disk still shows BEFORE. Restore the exact original directory mode in `finally`, then `restart`, `doctor`, GET and reload (CLI with `--run `): REFUSED was persisted during graceful shutdown. Only the owned temporary directory is changed; evidence contains title/timestamp fields, not stored secrets. Evidence: `backfill-18309-surface-rechecks/F07.rename-inline/metadata-before-4800.json`, `metadata-denied-4800.json`, `refused-title-4800.png`, `metadata-after-graceful-restart-4800.json`. Tags/autotitle and ownership-conflict branches remain untested." ], "annotations": [ "Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5800](https://github.com/OpenHands/software-agent-sdk/issues/5800)): confirming Delete after another client removed this disposable conversation returns400 and displays `Bad Request`, instead of a missing-conversation404. The separate stale-rename control receives PATCH200 but re-fetches missing metadata and displays not-found, so its false-success variant is prevented by Canvas. Pause/interrupt were not tested." ], "states": [ "failure" ] } ] }, { "id": "F08", "title": "Workspace drawer: files and changes", "file": "F08-workspace-files-and-changes.md", "sha256": "50549f873291663706783c2107f4293af3763e9f34b6c29dc9558bc8c6a79076", "overview": "Next to the chat, a conversation has a right-hand drawer with tabs. This family\ncovers the drawer itself (toggle, tab bar, overflow menu with pinning, the phone\npanel page) and its two workspace tabs. **Files** is a read-only browser of the\nconversation's working directory: a path row with copy, a strip of open files,\na collapsible tree, and a Rich/Plain viewer for code, Markdown, HTML, images and\nbinaries. **Commits** lists an \"Uncommitted\" row and the git history, newest\nfirst. Each expands into per-file Monaco diffs with old/diff/new modes. The\nTerminal, Browser, Planner, Task list and Usage tab bodies belong to F27. The\nFiles tab has no upload, download or edit.", "entry": "- Chat header: the panel toggle (`right-panel-toggle`, block-drawer icon, right of the overview `i` button). At desktop width it shows or hides the drawer; at 1024 px or narrower it navigates to the panel page.\n- Drawer tab bar (`tabs-pane-header`): `conversation-tab-files`, `conversation-tab-commits`. The ellipsis (`tabs-pane-header >> testid=ellipsis-button`) opens the menu with `conversation-tabs-menu-open-` and `conversation-tabs-menu-pin-`.\n- Direct URL `/conversations//panel` (the phone panel page at 1024 px or narrower; wider, it redirects to `/conversations/` with the drawer open).\n- Overview (`conversation-overview-toggle`) → Changes row (`conversation-overview-diffs`) → Commits tab with Uncommitted expanded.\n- Files toolbar: `File discovery` (`file-discovery-settings`, left of `files-tab-refresh`) opens the File discovery dialog.\n- Workspace paths in agent messages written as inline code (`markdown-file-path-link`), plus path chips in file-editor tool cards (`file-path-chip`) → Files tab with that file selected.\n- There is no command-menu entry or keyboard shortcut for the drawer or these tabs.", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done) and `control-openhands llm preset deepseek` (three short model turns, deepseek-flash).\n- A git fixture: `control-openhands fixture git-repo` (creates `/workspace/qa-repo` with `README.md`, `src/calc.py`, `src/test_calc.py` and one commit by `QA Fixture`).\n- The conversation runs in that repo. From `/` run `control-openhands browser click 'testid=open-workspace-button'`, `control-openhands browser click 'testid=workspace-dropdown'`, `control-openhands browser click 'role=button[name=\"+ Add Workspace\"]'`. The folder browser opens at `/private/home`, so then run `control-openhands browser click 'testid=folder-browser-up'` twice, `control-openhands browser dblclick 'testid=folder-browser-entry-workspace'`, `control-openhands browser dblclick 'testid=folder-browser-entry-qa-repo'`, `control-openhands browser click 'testid=folder-browser-use'` and `control-openhands browser click 'testid=workspace-launch-button'`. The composer now shows a `qa-repo` chip. Start with `--stay`, which keeps that chip; a plain `conversation start` navigates home and loses it.\n- Seed changes with a prompt file, because the command contains quotes. Write `qa-f08-1.txt` (anywhere outside the workspace) containing: ``In the current workspace directory run exactly this one bash command and then reply only \"done\": printf '# QA notes\\n\\nHello **world**\\n' > qa-notes.md && printf '

QA page

\\n' > qa-page.html && printf 'def sub(a, b):\\n return a - b\\n' >> src/calc.py && rm src/test_calc.py && printf 'PK\\003\\004' > qa-deck.pptx && head -c 64 /dev/urandom > qa-blob.bin && git status --short``. Then run `control-openhands conversation start --stay --prompt \"$(cat qa-f08-1.txt)\" --wait --timeout 240`. Read `` from its `id` (or `control-openhands browser url`); the browser stays on `/conversations/`.\n- Viewport desktop (1440×1000) unless a bullet says phone.", "gotchas": "- When the drawer is closed, its tabs stay mounted at width 0. `browser count 'testid=files-tab'` is `1` even then, so assert with `browser visible` or `right-panel-toggle` `aria-pressed`.\n- The tab row is a `tablist \"Conversation panels\"` of `role=tab` controls, each named after its tab even when inactive (#17612), so `role=tab[name=\"Commits\"]` finds the Commits tab whether or not it is active; `role=button[name=\"Commits\"]` finds nothing. `aria-selected` marks the selected tab. The `conversation-tab-*` test ids still work.\n- The tab row ends with the ellipsis menu (`Customize tabs`), then the `VSCode` button (`drawer-vscode-link`, named `Open in VS Code`) when the runtime reports a running editor. #17660 removed that button and #18048 restored it, so on a checkout between the two the drawer has no editor control at all.\n- The overflow menu is not a `role=menu`. Escape closes it and returns focus to the tab bar's `ellipsis-button`, both in the desktop drawer and on the phone `/panel` page; choosing a row or an outside click (`browser mouse-click`) closes it too.\n- The Files list and the open preview do not follow bash edits: `use-auto-refresh-files-on-edit.ts` refreshes the file queries only for file-editor observations, and its comment says shell observations skip them on purpose (so that every shell command does not reload canvas iframes). This map still expects files the agent writes to appear (`F08.files-auto-refresh`), so the bash case stays a fail row. Click `files-tab-refresh` once before asserting on files the agent made with the shell. Since #18032 that one click also re-reads the open file (the content fetch revalidates instead of using the browser HTTP cache, and the rich preview gets a new `?v=`), and a page reload shows the current body without a click.\n- The chat also fetches the file list (to linkify paths). A conversation page left open while the agent works can therefore show a stale tree the moment the drawer first opens.\n- Binary detection is a NUL-byte sniff of the first 8 KB, and the Office message also needs that sniff to trip. A 4-byte `PK\\003\\004` \"pptx\" or random bytes without a NUL render as text in the plain viewer. Fixtures must contain a NUL.\n- `control-openhands fixture folder` writes a plain folder with `notes.md`, but by the time the conversation has run the folder holds an empty `.git`, so a conversation that leaves `notes.md` in place shows `Uncommitted 1 file` rather than the empty state. Delete it first (as in the recipe).\n- Diff views are Monaco. `testid=file-diff-viewer` and `file-single-viewer` are not on the DOM; read text through `testid=editor-container`. The narrow drawer falls back to an inline (not side-by-side) diff. The view-mode buttons are named `Old version`, `Diff` and `New version`, and `collapse` is named `Expand` or `Collapse` after its state (#18033).\n- `browser eval \"navigator.clipboard.readText()\"` hangs for about 2 minutes (no clipboard permission). Use `control-openhands browser clipboard` to read what Copy wrote.\n- `browser click ... --observe` accepts the usual selector syntax (`testid=...`).\n- Two `.cursor-ew-resize` grips sit in the conversation page (chat/drawer, then the Files tree); scope the tree one with `testid=files-tab-tree-resize-handle`.\n- At phone width, chat path links (and the agent's `canvas_ui_control` tab and file requests, through `src/services/canvas-ui.ts`) navigate to `/conversations//panel` (#18045); at desktop width they open the side drawer and the URL does not change. A `/panel` URL above 1024 px redirects to `/conversations/`, so a bullet that leaves the phone viewport on `/panel` ends on the chat route.\n- If the agent's terminal resets mid-command (\"terminal session was reset\", for example after a `restart`), the agent reruns the command. Check the workspace with `git status` through a second prompt rather than trusting the first observation. Runs no longer share a tmux server (OpenHands/OpenHands#17946).\n- Known issue: [#17567](https://github.com/OpenHands/OpenHands/issues/17567) (Monaco disposed-model page error on diff unmount; it also fires on collapse and on a diff→old switch).\n- Known issue [#18108](https://github.com/OpenHands/OpenHands/issues/18108) (`F08.files-tree`): Files tab shows \"No files in workspace\" on cloud when the agent writes outside /workspace/project. Live prerequisite: An authenticated Enterprise Cloud sandbox, no repository selected, with files outside its default working directory.\n- Known issue [#17773](https://github.com/OpenHands/OpenHands/issues/17773) (`F08.files-tree`): Files tab always shows \"No files in workspace\" on Windows local backend. Live prerequisite: A Windows local Agent Server using cmd.exe; this run is macOS.\n- Known issue [#16949](https://github.com/OpenHands/OpenHands/issues/16949) (`F08.files-refresh`): Keep Files and Changes views source-driven after refresh. Live prerequisite: An authenticated Cloud repository for the refresh-query mismatch, or real file-editor observations causing an in-place M-to-D Git refetch.\n- Known issue [#16716](https://github.com/OpenHands/OpenHands/issues/16716) (`F08.uncommitted-changes`): Diff view is regularly out of sync. Live prerequisite: The original simultaneous-agent/shared-repository state or a deterministic stale-diff trigger; the report only suggests that timing.\n- Known issue [#16689](https://github.com/OpenHands/OpenHands/issues/16689) (`F08.files-load-error`): When `workspace/` paths throw a 401, call the workspace-session endpoint. Live prerequisite: The original fresh-VM URL, auth/cookie state and exact request sequence; the report says the trigger is unclear.\n- Known issue [#16454](https://github.com/OpenHands/OpenHands/issues/16454) (`F08.uncommitted-changes`): Diff list is showing duplicate entries. Live prerequisite: The original Git changes response and update sequence producing duplicate rows; the report provides no trigger.\n- Known issue [#16331](https://github.com/OpenHands/OpenHands/issues/16331) (`F08.files-tree`): file preview panel stops working during streaming. Live prerequisite: A real streaming/tool-running conversation while selecting a second workspace file.\n- Known issue [#15692](https://github.com/OpenHands/OpenHands/issues/15692) (`F08.uncommitted-changes`): Diff still shows after agent has committed and pushed. Live prerequisite: A real agent commit/push terminal observation while the diff remains mounted, plus a disposable Git remote.\n- Known issue [#15567](https://github.com/OpenHands/OpenHands/issues/15567) (`F08.files-load-error`): Workspace file preview returns 401 Unauthorized on non-HTTPS non-loopback hosts. Live prerequisite: A plain-HTTP non-loopback deployment and browser accessing that origin, with workspace-cookie observation.\n- Known issue [#15519](https://github.com/OpenHands/OpenHands/issues/15519) (`F08.files-load-error`): \"files\" tab in Files side panel throws 401 Unauthorized. Live prerequisite: The reported Linux Docker deployment/auth topology; this isolated run uses the macOS source launcher.\n- Known issue [#15501](https://github.com/OpenHands/OpenHands/issues/15501) (`F08.files-rich-plain`): Generated-page JavaScript was blocked by an iframe sandbox. Files Rich intentionally disables scripts; the report does not identify whether Files or Browser was used. Live prerequisite: The original preview panel and file URL or minimal page identifying the intended surface; Files Rich deliberately blocks scripts, while Browser is separate.\n- Known issue [OpenHands/software-agent-sdk#4562](https://github.com/OpenHands/software-agent-sdk/issues/4562) (`F08.files-load-error`): Plain-HTTP non-loopback workspace previews can return401 after the browser rejects the workspace-session cookie. Live prerequisite: A plain-HTTP non-loopback same-origin browser deployment with real workspace-cookie observation.\n- Known issue [#16871](https://github.com/OpenHands/OpenHands/issues/16871) (`F08.vscode-link`): the shared `transformVSCodeUrl` still accepts arbitrary schemes, and the current Cloud `use-unified-vscode-url.ts` branch passes sandbox `exposed_urls` editor values through it without the Local origin gate. Live prerequisite: an authorized Cloud account/sandbox whose editor `exposed_urls` value can be controlled for safe URL-scheme rejection checks. Local inspection on 2026-10-12 at `fa1f724ab573` found no Download via VS Code action in the real card menu: its only production caller does not supply the required `showOptions` prop. The current drawer opens with `noopener,noreferrer`. Neither the old opener exploit nor the remaining Cloud scheme scenario was reproduced. Evidence: `backfill-18309-surface-rechecks/F04.download/vscode-card-action-absent-16871.png` and its ARIA snapshot. Keep this linked to the live editor-link consumer; do not infer the whole issue resolved from the removed card action.\n- Known issue [OpenHands/software-agent-sdk#5845](https://github.com/OpenHands/software-agent-sdk/issues/5845) (`F08.vscode-link`): reserved characters in the workspace path can split the editor URL's unencoded `folder` parameter and open the wrong folder. Live prerequisite: a supported origin serving a genuinely enabled/running editor and an owned conversation in a path containing `&` or `#`. A direct URL-constructor call does not exercise the link; Cloud `exposed_urls` takes another path.\n- Known issue [OpenHands/software-agent-sdk#5784](https://github.com/OpenHands/software-agent-sdk/issues/5784) (`F08.files-rich-plain`, `F08.files-fallbacks`, `F08.files-open-new-window`): a path-prefixed Agent Server can set a workspace cookie whose Path omits the prefix, leaving HTML/image/file previews unauthorized. Live prerequisite: an authenticated path-prefixed owned Local server with `OH_WEB_URL` and a real cookie-authenticated preview; the ordinary root-path launch cannot exercise that cookie Path.", "features": [ { "id": "F08.panel-toggle", "description": "the chat-header toggle (`Show panel`/`Hide panel`, `aria-pressed`) opens and closes the drawer; the first open shows Files." }, { "id": "F08.tab-bar", "description": "the tab bar (Files, Commits, Planner, Terminal, Browser, Usage; Task list only when the agent has one) switches the drawer body, and only the active tab shows its label. Clicking the active tab closes the drawer." }, { "id": "F08.drawer-state-reload", "description": "after a reload the drawer starts closed, and reopening it restores the last selected tab for that conversation." }, { "id": "F08.tabs-menu", "description": "the ellipsis menu at the end of the tab bar lists every tab. Choosing a row opens that tab and closes the menu, and so does clicking outside." }, { "id": "F08.tabs-pin", "description": "the pill on each menu row pins or unpins a tab. Unpinning the active tab selects another pinned one, and the choice persists across reloads." }, { "id": "F08.drawer-resize", "description": "dragging the chat/drawer divider resizes both panes (chat 30–80 %, default 50 %, stored in `desktop-layout-panel-width`)." }, { "id": "F08.vscode-link", "description": "a `VSCode` button at the right of the tab bar opens the editor in a new window; it is hidden when the runtime has no editor." }, { "id": "F08.archived-disabled", "description": "on an archived conversation the toggle is disabled with the tooltip `Unavailable for Archives`." }, { "id": "F08.phone-panel-page", "description": "at 1024 px or narrower the toggle navigates to `/conversations//panel`, a full-screen page with a Back chevron, the same tab bar and the tab body." }, { "id": "F08.panel-direct-url", "description": "opening `/conversations//panel` directly renders the panel page at 1024 px or narrower, and Back returns to the chat; wider, the URL is replaced with `/conversations/` and the drawer opens beside the chat." }, { "id": "F08.files-workspace-path", "description": "`Workspace: ` with the full path in its tooltip; the copy button turns into `Copied to clipboard` for about 2 s." }, { "id": "F08.files-tree", "description": "a tree of folders, collapsed at first, and files. Clicking a file selects it and adds it to the open-file strip." }, { "id": "F08.files-empty-workspace", "description": "a workspace with no files shows `No files in workspace`." }, { "id": "F08.files-tree-toggle", "description": "the list-tree button hides or shows the tree, and the choice persists per conversation. The tree is 224 px wide by default." }, { "id": "F08.files-tree-resize", "description": "dragging the divider between the tree and the preview resizes the tree (160–480 px, default 224) and the width persists across reloads (`files-tab-tree-width` in localStorage)." }, { "id": "F08.files-discovery-settings", "description": "the Files toolbar's `File discovery` button opens a dialog (excluded directory patterns, maximum files, symbolic links) whose Save applies to every conversation on that workspace and backend; an excluded directory leaves the tree, Restore defaults brings the default patterns back, an invalid limit disables Save." }, { "id": "F08.files-open-tabs", "description": "the open-file strip, each tab with a close X. Closing the selected file selects a neighbour. Open files and the selection survive a reload and do not leak into other conversations." }, { "id": "F08.files-no-selection", "description": "with no file selected the pane reads `Select a file to view its contents` and the Rich/Plain toggle is absent." }, { "id": "F08.files-rich-plain", "description": "Rich highlights code, renders Markdown and shows HTML in an iframe sandboxed without scripts. Plain shows highlighted source. The mode persists across reloads." }, { "id": "F08.files-fallbacks", "description": "images render as `` in Rich mode. Files with NUL bytes show `Binary file – preview not available`, and Office files show `Preview isn't available for Word files.` (or PowerPoint/Excel)." }, { "id": "F08.files-load-error", "description": "a selected file that can no longer be read shows the error (`Failed to read : 404`)." }, { "id": "F08.files-open-new-window", "description": "the external-link icon opens the file from the workspace fileserver in a new browser tab." }, { "id": "F08.files-refresh", "description": "`Refresh files` re-reads the file list and the selected file's content." }, { "id": "F08.files-auto-refresh", "description": "files the agent writes appear without a manual refresh." }, { "id": "F08.files-open-from-chat", "description": "clicking a workspace path in an agent (or user) message opens the drawer on Files with that file selected." }, { "id": "F08.files-open-from-tool-chip", "description": "the path chip on a file-editor tool card (absolute path) opens the drawer on Files with that file selected." }, { "id": "F08.commits-states", "description": "with no commits and no changes the Commits tab shows `No commits yet`. While the runtime is inactive it shows `Waiting for runtime to start...`, and while loading `Loading changes...`." }, { "id": "F08.uncommitted-changes", "description": "the top row reads `-`, `Uncommitted` and a file count; expanding it lists the changed paths." }, { "id": "F08.commit-rows", "description": "each commit row shows the short SHA, subject, author (only when there are several authors) and relative time. Rows are single-open accordions, and new agent commits appear without a reload." }, { "id": "F08.commits-cap", "description": "the history loads the latest 50 commits; a longer history ends with `Showing the latest 50 commits`." }, { "id": "F08.diff-view-modes", "description": "expanding a changed file shows its diff; the view-mode buttons (`Old version`, `Diff`, `New version`) switch between old, diff and new without page errors." }, { "id": "F08.diff-markdown-preview", "description": "a Markdown file in new (or old) mode renders as a prose preview." }, { "id": "F08.diff-deleted-file", "description": "a deleted working-tree file reads `This file was deleted. Its previous contents are no longer available for preview.`" }, { "id": "F08.changes-from-overview", "description": "the overview's Changes row opens the drawer on Commits with Uncommitted expanded and closes the overview." } ], "scenarios": [ { "name": "F08--open-the-drawer", "title": "Open the drawer", "ids": [ "F08.panel-toggle" ], "body": "Run `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`false`) and `control-openhands browser tooltip 'testid=right-panel-toggle'` (`Show panel`). Then run `control-openhands browser click 'testid=right-panel-toggle'`, followed by `control-openhands browser attr 'testid=right-panel-toggle' aria-label` and `control-openhands browser visible 'testid=files-tab'`. The label is `Hide panel`, `aria-pressed` is `true`, and Files is visible with the workspace path and tree (`browser screenshot --feature F08.panel-toggle --name drawer-open-files`).", "raw": "**Open the drawer (`F08.panel-toggle`).** Run `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`false`) and `control-openhands browser tooltip 'testid=right-panel-toggle'` (`Show panel`). Then run `control-openhands browser click 'testid=right-panel-toggle'`, followed by `control-openhands browser attr 'testid=right-panel-toggle' aria-label` and `control-openhands browser visible 'testid=files-tab'`. The label is `Hide panel`, `aria-pressed` is `true`, and Files is visible with the workspace path and tree (`browser screenshot --feature F08.panel-toggle --name drawer-open-files`).", "source_line": 72, "expected": [ "the chat-header toggle (`Show panel`/`Hide panel`, `aria-pressed`) opens and closes the drawer; the first open shows Files." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--switch-and-close-by-tab", "title": "Switch and close by tab", "ids": [ "F08.tab-bar" ], "body": "Run `control-openhands browser click 'testid=conversation-tab-commits'` and `control-openhands browser wait 'testid=commit-list' --state visible --timeout 10000` (the first open shows `Loading changes...` for a moment, so an immediate `browser visible` can read `false`). Click `testid=conversation-tab-commits` again. `right-panel-toggle` `aria-pressed` becomes `false` and `control-openhands browser visible 'testid=tabs-pane-header'` is `false`. Reopen with `right-panel-toggle`.", "raw": "**Switch and close by tab (`F08.tab-bar`).** Run `control-openhands browser click 'testid=conversation-tab-commits'` and `control-openhands browser wait 'testid=commit-list' --state visible --timeout 10000` (the first open shows `Loading changes...` for a moment, so an immediate `browser visible` can read `false`). Click `testid=conversation-tab-commits` again. `right-panel-toggle` `aria-pressed` becomes `false` and `control-openhands browser visible 'testid=tabs-pane-header'` is `false`. Reopen with `right-panel-toggle`.", "source_line": 73, "expected": [ "the tab bar (Files, Commits, Planner, Terminal, Browser, Usage; Task list only when the agent has one) switches the drawer body, and only the active tab shows its label. Clicking the active tab closes the drawer." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--reload", "title": "Reload", "ids": [ "F08.drawer-state-reload" ], "body": "With Commits selected, run `control-openhands browser reload`, then `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`false`; the drawer is closed). Click `testid=right-panel-toggle`; `control-openhands browser visible 'testid=commit-list'` is `true`.", "raw": "**Reload (`F08.drawer-state-reload`).** With Commits selected, run `control-openhands browser reload`, then `control-openhands browser attr 'testid=right-panel-toggle' aria-pressed` (`false`; the drawer is closed). Click `testid=right-panel-toggle`; `control-openhands browser visible 'testid=commit-list'` is `true`.", "source_line": 74, "expected": [ "after a reload the drawer starts closed, and reopening it restores the last selected tab for that conversation." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--overflow-menu", "title": "Overflow menu", "ids": [ "F08.tabs-menu" ], "body": "Run `control-openhands browser click 'testid=tabs-pane-header >> testid=ellipsis-button'` and `control-openhands browser testids --filter conversation-tabs-menu`; there is one open row and one pin row per tab. Run `control-openhands browser click 'testid=conversation-tabs-menu-open-usage'`; then `control-openhands browser count 'testid=conversation-tabs-menu-open-usage'` is `0` (menu closed) and the bar's active label is `Usage`. Reopen the menu and run `control-openhands browser mouse-click 560 600` (over the chat); the menu count is `0`.", "raw": "**Overflow menu (`F08.tabs-menu`).** Run `control-openhands browser click 'testid=tabs-pane-header >> testid=ellipsis-button'` and `control-openhands browser testids --filter conversation-tabs-menu`; there is one open row and one pin row per tab. Run `control-openhands browser click 'testid=conversation-tabs-menu-open-usage'`; then `control-openhands browser count 'testid=conversation-tabs-menu-open-usage'` is `0` (menu closed) and the bar's active label is `Usage`. Reopen the menu and run `control-openhands browser mouse-click 560 600` (over the chat); the menu count is `0`.", "source_line": 75, "expected": [ "the ellipsis menu at the end of the tab bar lists every tab. Choosing a row opens that tab and closes the menu, and so does clicking outside." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--pin-unpin", "title": "Pin/unpin", "ids": [ "F08.tabs-pin" ], "body": "With Usage active, open the menu and run `control-openhands browser click 'testid=conversation-tabs-menu-pin-usage'`, then `control-openhands browser attr 'testid=conversation-tabs-menu-pin-usage' aria-label` (`Pin tab to bar`; `aria-pressed` is `false`). Close the menu with `browser mouse-click 560 600`, then run `control-openhands browser count 'testid=tabs-pane-header >> testid=conversation-tab-usage'`. The count is `0` and Planner is now active. Run `control-openhands browser reload` and reopen the drawer; the count is still `0`. To restore, open the menu, click `testid=conversation-tabs-menu-pin-usage` (label `Unpin tab from bar`), then `control-openhands browser click 'testid=conversation-tabs-menu-open-files'`. The usage tab count is `1` again.", "raw": "**Pin/unpin (`F08.tabs-pin`).** With Usage active, open the menu and run `control-openhands browser click 'testid=conversation-tabs-menu-pin-usage'`, then `control-openhands browser attr 'testid=conversation-tabs-menu-pin-usage' aria-label` (`Pin tab to bar`; `aria-pressed` is `false`). Close the menu with `browser mouse-click 560 600`, then run `control-openhands browser count 'testid=tabs-pane-header >> testid=conversation-tab-usage'`. The count is `0` and Planner is now active. Run `control-openhands browser reload` and reopen the drawer; the count is still `0`. To restore, open the menu, click `testid=conversation-tabs-menu-pin-usage` (label `Unpin tab from bar`), then `control-openhands browser click 'testid=conversation-tabs-menu-open-files'`. The usage tab count is `1` again.", "source_line": 76, "expected": [ "the pill on each menu row pins or unpins a tab. Unpinning the active tab selects another pinned one, and the choice persists across reloads." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--resize", "title": "Resize", "ids": [ "F08.drawer-resize" ], "body": "The divider has no test id; its grip is the first `.cursor-ew-resize` on the page (the second is the Files tree divider). With the drawer open, `control-openhands browser bbox 'testid=chat-interface'` and `control-openhands browser bbox 'testid=tabs-pane-header'` are about 570 px wide each and `control-openhands browser eval \"localStorage.getItem('desktop-layout-panel-width')\"` is `\"50\"`. Run `control-openhands browser drag '.cursor-ew-resize >> nth=0' --by 200,0`: the chat is 770 px, the drawer about 369 px and the key about `67.5`. `--by 900,0` clamps at `\"80\"`, `--by -1200,0` at `\"30\"` (chat 342 px), and after `browser reload` plus the toggle the chat is still 342 px. Restore with `control-openhands browser drag '.cursor-ew-resize >> nth=0' --by 228,0` (chat 570 px, key `\"50\"`).", "raw": "**Resize (`F08.drawer-resize`).** The divider has no test id; its grip is the first `.cursor-ew-resize` on the page (the second is the Files tree divider). With the drawer open, `control-openhands browser bbox 'testid=chat-interface'` and `control-openhands browser bbox 'testid=tabs-pane-header'` are about 570 px wide each and `control-openhands browser eval \"localStorage.getItem('desktop-layout-panel-width')\"` is `\"50\"`. Run `control-openhands browser drag '.cursor-ew-resize >> nth=0' --by 200,0`: the chat is 770 px, the drawer about 369 px and the key about `67.5`. `--by 900,0` clamps at `\"80\"`, `--by -1200,0` at `\"30\"` (chat 342 px), and after `browser reload` plus the toggle the chat is still 342 px. Restore with `control-openhands browser drag '.cursor-ew-resize >> nth=0' --by 228,0` (chat 570 px, key `\"50\"`).", "source_line": 77, "expected": [ "dragging the chat/drawer divider resizes both panes (chat 30–80 %, default 50 %, stored in `desktop-layout-panel-width`)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--vs-code-link", "title": "VS Code link", "ids": [ "F08.vscode-link" ], "body": "Blocked in a runtime without a running VS Code server: `control-openhands browser count 'testid=drawer-vscode-link'` is `0` (hidden, as designed), and `control-openhands api GET /api/vscode/status` explains why (`enabled: false`, or `running: false` when the agent-server has no editor binary, as in these stacks; see F26 Gotchas). It needs a runtime with `enable_vscode` and a running editor, on an origin that serves `/vscode/`.", "raw": "**VS Code link (`F08.vscode-link`).** Blocked in a runtime without a running VS Code server: `control-openhands browser count 'testid=drawer-vscode-link'` is `0` (hidden, as designed), and `control-openhands api GET /api/vscode/status` explains why (`enabled: false`, or `running: false` when the agent-server has no editor binary, as in these stacks; see F26 Gotchas). It needs a runtime with `enable_vscode` and a running editor, on an origin that serves `/vscode/`.", "source_line": 78, "expected": [ "a `VSCode` button at the right of the tab bar opens the editor in a new window; it is hidden when the runtime has no editor." ], "notes": [ "Known issue [#16871](https://github.com/OpenHands/OpenHands/issues/16871) (`F08.vscode-link`): the shared `transformVSCodeUrl` still accepts arbitrary schemes, and the current Cloud `use-unified-vscode-url.ts` branch passes sandbox `exposed_urls` editor values through it without the Local origin gate. Live prerequisite: an authorized Cloud account/sandbox whose editor `exposed_urls` value can be controlled for safe URL-scheme rejection checks. Local inspection on 2026-10-12 at `fa1f724ab573` found no Download via VS Code action in the real card menu: its only production caller does not supply the required `showOptions` prop. The current drawer opens with `noopener,noreferrer`. Neither the old opener exploit nor the remaining Cloud scheme scenario was reproduced. Evidence: `backfill-18309-surface-rechecks/F04.download/vscode-card-action-absent-16871.png` and its ARIA snapshot. Keep this linked to the live editor-link consumer; do not infer the whole issue resolved from the removed card action.", "Known issue [OpenHands/software-agent-sdk#5845](https://github.com/OpenHands/software-agent-sdk/issues/5845) (`F08.vscode-link`): reserved characters in the workspace path can split the editor URL's unencoded `folder` parameter and open the wrong folder. Live prerequisite: a supported origin serving a genuinely enabled/running editor and an owned conversation in a path containing `&` or `#`. A direct URL-constructor call does not exercise the link; Cloud `exposed_urls` takes another path." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F08--archived", "title": "Archived", "ids": [ "F08.archived-disabled" ], "body": "Blocked locally: archived status comes from a Cloud sandbox (`sandbox_status`). On such a conversation, `browser enabled 'testid=right-panel-toggle'` should be `false` and `browser tooltip 'testid=right-panel-toggle'` should read `Unavailable for Archives`.", "raw": "**Archived (`F08.archived-disabled`).** Blocked locally: archived status comes from a Cloud sandbox (`sandbox_status`). On such a conversation, `browser enabled 'testid=right-panel-toggle'` should be `false` and `browser tooltip 'testid=right-panel-toggle'` should read `Unavailable for Archives`.", "source_line": 79, "expected": [ "on an archived conversation the toggle is disabled with the tooltip `Unavailable for Archives`." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F08--workspace-path", "title": "Workspace path", "ids": [ "F08.files-workspace-path" ], "body": "Run `control-openhands browser attr 'testid=files-tab-workspace-path-value' title` (ends in `/workspace/qa-repo`). Click `testid=files-tab-workspace-path >> testid=copy-to-clipboard`. Immediately afterwards, `control-openhands browser attr 'testid=files-tab-workspace-path >> testid=copy-to-clipboard' aria-label` is `Copied to clipboard` and `browser enabled` on the same button is `false`. `control-openhands browser clipboard` returns the full path. About 3 s later the label is `Copy to clipboard` again.", "raw": "**Workspace path (`F08.files-workspace-path`).** Run `control-openhands browser attr 'testid=files-tab-workspace-path-value' title` (ends in `/workspace/qa-repo`). Click `testid=files-tab-workspace-path >> testid=copy-to-clipboard`. Immediately afterwards, `control-openhands browser attr 'testid=files-tab-workspace-path >> testid=copy-to-clipboard' aria-label` is `Copied to clipboard` and `browser enabled` on the same button is `false`. `control-openhands browser clipboard` returns the full path. About 3 s later the label is `Copy to clipboard` again.", "source_line": 80, "expected": [ "`Workspace: ` with the full path in its tooltip; the copy button turns into `Copied to clipboard` for about 2 s." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--tree", "title": "Tree", "ids": [ "F08.files-tree" ], "body": "Run `control-openhands browser click 'testid=files-tab-refresh'` first (see Gotchas: files the agent made with bash are missing until a refresh). Then `control-openhands browser attr 'testid=file-tree-dir-src' aria-expanded` (`false`), `control-openhands browser click 'testid=file-tree-dir-src'` (now `true`) and `control-openhands browser click 'testid=file-tree-file-src/calc.py'`. `control-openhands browser text 'testid=file-content-viewer-highlighted'` shows the 4 lines `def add`…`return a - b`, and `testid=file-quick-row-item-src/calc.py` exists.", "raw": "**Tree (`F08.files-tree`).** Run `control-openhands browser click 'testid=files-tab-refresh'` first (see Gotchas: files the agent made with bash are missing until a refresh). Then `control-openhands browser attr 'testid=file-tree-dir-src' aria-expanded` (`false`), `control-openhands browser click 'testid=file-tree-dir-src'` (now `true`) and `control-openhands browser click 'testid=file-tree-file-src/calc.py'`. `control-openhands browser text 'testid=file-content-viewer-highlighted'` shows the 4 lines `def add`…`return a - b`, and `testid=file-quick-row-item-src/calc.py` exists.", "source_line": 81, "expected": [ "a tree of folders, collapsed at first, and files. Clicking a file selects it and adds it to the open-file strip." ], "notes": [ "Known issue [#18108](https://github.com/OpenHands/OpenHands/issues/18108) (`F08.files-tree`): Files tab shows \"No files in workspace\" on cloud when the agent writes outside /workspace/project. Live prerequisite: An authenticated Enterprise Cloud sandbox, no repository selected, with files outside its default working directory.", "Known issue [#17773](https://github.com/OpenHands/OpenHands/issues/17773) (`F08.files-tree`): Files tab always shows \"No files in workspace\" on Windows local backend. Live prerequisite: A Windows local Agent Server using cmd.exe; this run is macOS.", "Known issue [#16331](https://github.com/OpenHands/OpenHands/issues/16331) (`F08.files-tree`): file preview panel stops working during streaming. Live prerequisite: A real streaming/tool-running conversation while selecting a second workspace file." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F08--tree-toggle", "title": "Tree toggle", "ids": [ "F08.files-tree-toggle" ], "body": "Run `control-openhands browser bbox 'testid=files-tab-tree'` (width `224`). Then `control-openhands browser click 'testid=file-quick-row-tree-toggle'`, `control-openhands browser attr 'testid=file-quick-row-tree-toggle' aria-label` (`Show file tree`) and `control-openhands browser count 'testid=files-tab-tree'` (`0`). Reload and reopen the drawer: the count is still `0`. Click the toggle again to restore (count `1`).", "raw": "**Tree toggle (`F08.files-tree-toggle`).** Run `control-openhands browser bbox 'testid=files-tab-tree'` (width `224`). Then `control-openhands browser click 'testid=file-quick-row-tree-toggle'`, `control-openhands browser attr 'testid=file-quick-row-tree-toggle' aria-label` (`Show file tree`) and `control-openhands browser count 'testid=files-tab-tree'` (`0`). Reload and reopen the drawer: the count is still `0`. Click the toggle again to restore (count `1`).", "source_line": 82, "expected": [ "the list-tree button hides or shows the tree, and the choice persists per conversation. The tree is 224 px wide by default." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--tree-resize", "title": "Tree resize", "ids": [ "F08.files-tree-resize" ], "body": "Run `control-openhands browser drag 'testid=files-tab-tree-resize-handle >> .cursor-ew-resize' --by 100,0`; `browser bbox 'testid=files-tab-tree'` is `324` wide and `control-openhands browser eval \"localStorage.getItem('files-tab-tree-width')\"` is `\"324\"`. `--by 500,0` clamps at `480`, `--by -600,0` at `160`, and the width survives `browser reload` plus the toggle. Restore with `--by 64,0` (`224`).", "raw": "**Tree resize (`F08.files-tree-resize`).** Run `control-openhands browser drag 'testid=files-tab-tree-resize-handle >> .cursor-ew-resize' --by 100,0`; `browser bbox 'testid=files-tab-tree'` is `324` wide and `control-openhands browser eval \"localStorage.getItem('files-tab-tree-width')\"` is `\"324\"`. `--by 500,0` clamps at `480`, `--by -600,0` at `160`, and the width survives `browser reload` plus the toggle. Restore with `--by 64,0` (`224`).", "source_line": 83, "expected": [ "dragging the divider between the tree and the preview resizes the tree (160–480 px, default 224) and the width persists across reloads (`files-tab-tree-width` in localStorage)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--file-discovery", "title": "File discovery", "ids": [ "F08.files-discovery-settings" ], "body": "With the drawer open on Files (open it with `right-panel-toggle` and click `testid=conversation-tab-files` only if Files is not already the active tab: clicking the active tab closes the drawer), run `control-openhands browser click 'testid=file-discovery-settings'` and `control-openhands browser snapshot 'role=dialog[name=\"File discovery\"]'`: `Applies to this workspace on this backend, including other conversations using it.`, textbox `Excluded directories` (`.git`, `node_modules`, `.venv`, … `target`), spinbutton `Maximum files (0 for unlimited)` `2000`, checkbox `Include symbolic links to files`, buttons `Restore defaults`, `Cancel`, `Save`. With `D='role=dialog[name=\"File discovery\"]'`: `control-openhands browser fill \"$D >> role=spinbutton\" '-1'` makes `control-openhands browser enabled \"$D >> role=button[name=\\\"Save\\\"]\"` `false`; `2000` makes it `true`. Run `control-openhands browser fill \"$D >> role=textbox\" \"$(printf '.git\\nnode_modules\\nsrc')\"` and click Save: the dialog closes, `control-openhands api GET /api/settings` holds `workspace_file_discovery` keyed by the workspace path with `excludedPatterns` `.git`, `node_modules`, `src`, and after `control-openhands browser click 'testid=files-tab-refresh'` `control-openhands browser count 'testid=file-tree-dir-src'` is `0`, also after `browser reload` and reopening the drawer (`browser screenshot --feature F08.files-discovery-settings --name dialog` before saving). Restore: reopen the dialog, click `Restore defaults` (the textbox returns to the default list; the dialog stays open), then `Save`; after Refresh the `file-tree-dir-src` count is `1`. `control-openhands browser press Escape` closes the dialog without saving. At phone width the dialog fits the viewport: on the 390 px `/panel` page (`control-openhands browser viewport phone`, `browser goto /conversations//panel`, `browser click 'testid=conversation-tab-files'`, then `file-discovery-settings`) `control-openhands browser bbox 'testid=file-discovery-settings-modal'` is about 351 px wide at x about 20 with `insideViewport` `true` (520 px at desktop), and `browser screenshot --feature F08.files-discovery-settings --name phone-fits` shows the title, the textbox and `Restore defaults`, `Cancel`, `Save` whole. Measure the `file-discovery-settings-modal` test id: `role=dialog` is the full-screen backdrop. Focus starts in the dialog (the Excluded directories textbox) and Tab cycles its fields and buttons without leaving it.", "raw": "**File discovery (`F08.files-discovery-settings`).** With the drawer open on Files (open it with `right-panel-toggle` and click `testid=conversation-tab-files` only if Files is not already the active tab: clicking the active tab closes the drawer), run `control-openhands browser click 'testid=file-discovery-settings'` and `control-openhands browser snapshot 'role=dialog[name=\"File discovery\"]'`: `Applies to this workspace on this backend, including other conversations using it.`, textbox `Excluded directories` (`.git`, `node_modules`, `.venv`, … `target`), spinbutton `Maximum files (0 for unlimited)` `2000`, checkbox `Include symbolic links to files`, buttons `Restore defaults`, `Cancel`, `Save`. With `D='role=dialog[name=\"File discovery\"]'`: `control-openhands browser fill \"$D >> role=spinbutton\" '-1'` makes `control-openhands browser enabled \"$D >> role=button[name=\\\"Save\\\"]\"` `false`; `2000` makes it `true`. Run `control-openhands browser fill \"$D >> role=textbox\" \"$(printf '.git\\nnode_modules\\nsrc')\"` and click Save: the dialog closes, `control-openhands api GET /api/settings` holds `workspace_file_discovery` keyed by the workspace path with `excludedPatterns` `.git`, `node_modules`, `src`, and after `control-openhands browser click 'testid=files-tab-refresh'` `control-openhands browser count 'testid=file-tree-dir-src'` is `0`, also after `browser reload` and reopening the drawer (`browser screenshot --feature F08.files-discovery-settings --name dialog` before saving). Restore: reopen the dialog, click `Restore defaults` (the textbox returns to the default list; the dialog stays open), then `Save`; after Refresh the `file-tree-dir-src` count is `1`. `control-openhands browser press Escape` closes the dialog without saving. At phone width the dialog fits the viewport: on the 390 px `/panel` page (`control-openhands browser viewport phone`, `browser goto /conversations//panel`, `browser click 'testid=conversation-tab-files'`, then `file-discovery-settings`) `control-openhands browser bbox 'testid=file-discovery-settings-modal'` is about 351 px wide at x about 20 with `insideViewport` `true` (520 px at desktop), and `browser screenshot --feature F08.files-discovery-settings --name phone-fits` shows the title, the textbox and `Restore defaults`, `Cancel`, `Save` whole. Measure the `file-discovery-settings-modal` test id: `role=dialog` is the full-screen backdrop. Focus starts in the dialog (the Excluded directories textbox) and Tab cycles its fields and buttons without leaving it.", "source_line": 84, "expected": [ "the Files toolbar's `File discovery` button opens a dialog (excluded directory patterns, maximum files, symbolic links) whose Save applies to every conversation on that workspace and backend; an excluded directory leaves the tree, Restore defaults brings the default patterns back, an invalid limit disables Save." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--rich-plain", "title": "Rich/Plain", "ids": [ "F08.files-rich-plain" ], "body": "Click `testid=file-tree-file-qa-notes.md`; `testid=file-content-viewer-markdown` exists. Click `testid=file-tree-file-qa-page.html`, then run `control-openhands browser attr 'testid=file-content-viewer-iframe' sandbox` (`allow-same-origin`) and `control-openhands browser eval \"document.querySelector('[data-testid=file-content-viewer-iframe]').contentDocument.body.innerText\"` (`QA page`). Run `control-openhands browser click 'testid=files-tab-content-mode-toggle-option-plain'`; `control-openhands browser text 'testid=files-tab-content >> testid=file-content-viewer-highlighted'` reads `1

QA page

`. After `browser reload` and reopening, `control-openhands browser attr 'testid=files-tab-content-mode-toggle-option-plain' aria-checked` is `true`. Switch back with `testid=files-tab-content-mode-toggle-option-rich`.", "raw": "**Rich/Plain (`F08.files-rich-plain`).** Click `testid=file-tree-file-qa-notes.md`; `testid=file-content-viewer-markdown` exists. Click `testid=file-tree-file-qa-page.html`, then run `control-openhands browser attr 'testid=file-content-viewer-iframe' sandbox` (`allow-same-origin`) and `control-openhands browser eval \"document.querySelector('[data-testid=file-content-viewer-iframe]').contentDocument.body.innerText\"` (`QA page`). Run `control-openhands browser click 'testid=files-tab-content-mode-toggle-option-plain'`; `control-openhands browser text 'testid=files-tab-content >> testid=file-content-viewer-highlighted'` reads `1

QA page

`. After `browser reload` and reopening, `control-openhands browser attr 'testid=files-tab-content-mode-toggle-option-plain' aria-checked` is `true`. Switch back with `testid=files-tab-content-mode-toggle-option-rich`.", "source_line": 85, "expected": [ "Rich highlights code, renders Markdown and shows HTML in an iframe sandboxed without scripts. Plain shows highlighted source. The mode persists across reloads." ], "notes": [ "Known issue [#15501](https://github.com/OpenHands/OpenHands/issues/15501) (`F08.files-rich-plain`): Generated-page JavaScript was blocked by an iframe sandbox. Files Rich intentionally disables scripts; the report does not identify whether Files or Browser was used. Live prerequisite: The original preview panel and file URL or minimal page identifying the intended surface; Files Rich deliberately blocks scripts, while Browser is separate.", "Known issue [OpenHands/software-agent-sdk#5784](https://github.com/OpenHands/software-agent-sdk/issues/5784) (`F08.files-rich-plain`, `F08.files-fallbacks`, `F08.files-open-new-window`): a path-prefixed Agent Server can set a workspace cookie whose Path omits the prefix, leaving HTML/image/file previews unauthorized. Live prerequisite: an authenticated path-prefixed owned Local server with `OH_WEB_URL` and a real cookie-authenticated preview; the ordinary root-path launch cannot exercise that cookie Path." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F08--open-in-new-window", "title": "Open in new window", "ids": [ "F08.files-open-new-window" ], "body": "With `qa-page.html` selected, run `control-openhands browser click 'testid=files-tab-open-in-new-window'` and `control-openhands browser tabs`. Page 1's URL is `/api/conversations//workspace/qa-page.html?v=`. Run `control-openhands browser tab 1`, `control-openhands browser text 'body'` (`QA page`) and `control-openhands browser close-tab 1`.", "raw": "**Open in new window (`F08.files-open-new-window`).** With `qa-page.html` selected, run `control-openhands browser click 'testid=files-tab-open-in-new-window'` and `control-openhands browser tabs`. Page 1's URL is `/api/conversations//workspace/qa-page.html?v=`. Run `control-openhands browser tab 1`, `control-openhands browser text 'body'` (`QA page`) and `control-openhands browser close-tab 1`.", "source_line": 86, "expected": [ "the external-link icon opens the file from the workspace fileserver in a new browser tab." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5784](https://github.com/OpenHands/software-agent-sdk/issues/5784) (`F08.files-rich-plain`, `F08.files-fallbacks`, `F08.files-open-new-window`): a path-prefixed Agent Server can set a workspace cookie whose Path omits the prefix, leaving HTML/image/file previews unauthorized. Live prerequisite: an authenticated path-prefixed owned Local server with `OH_WEB_URL` and a real cookie-authenticated preview; the ordinary root-path launch cannot exercise that cookie Path." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F08--open-file-strip", "title": "Open-file strip", "ids": [ "F08.files-open-tabs" ], "body": "With several files opened, run `control-openhands browser attr 'testid=file-quick-row-item-qa-page.html' title` (the path). Select `qa-notes.md` with `control-openhands browser click 'testid=file-quick-row-item-qa-notes.md'`, then close it with `control-openhands browser click 'testid=file-quick-row-close-qa-notes.md'`. `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=file-quick-row-item-]')].filter(e=>e.getAttribute('aria-selected')==='true').map(e=>e.dataset.testid)\"` names the neighbour. Run `browser reload`, reopen the drawer and `control-openhands browser text 'testid=file-quick-row'`: the same files remain. Opening a different conversation (see Commits empty state) shows no quick-row items: `browser text 'testid=file-quick-row'` is empty. `browser testids` omits strip tabs scrolled out of view; count them with `browser eval \"[...document.querySelectorAll('[data-testid^=file-quick-row-item-]')].length\"`.", "raw": "**Open-file strip (`F08.files-open-tabs`).** With several files opened, run `control-openhands browser attr 'testid=file-quick-row-item-qa-page.html' title` (the path). Select `qa-notes.md` with `control-openhands browser click 'testid=file-quick-row-item-qa-notes.md'`, then close it with `control-openhands browser click 'testid=file-quick-row-close-qa-notes.md'`. `control-openhands browser eval \"[...document.querySelectorAll('[data-testid^=file-quick-row-item-]')].filter(e=>e.getAttribute('aria-selected')==='true').map(e=>e.dataset.testid)\"` names the neighbour. Run `browser reload`, reopen the drawer and `control-openhands browser text 'testid=file-quick-row'`: the same files remain. Opening a different conversation (see Commits empty state) shows no quick-row items: `browser text 'testid=file-quick-row'` is empty. `browser testids` omits strip tabs scrolled out of view; count them with `browser eval \"[...document.querySelectorAll('[data-testid^=file-quick-row-item-]')].length\"`.", "source_line": 87, "expected": [ "the open-file strip, each tab with a close X. Closing the selected file selects a neighbour. Open files and the selection survive a reload and do not leak into other conversations." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--no-selection", "title": "No selection", "ids": [ "F08.files-no-selection" ], "body": "Close every open file with its `file-quick-row-close-` button. `control-openhands browser text 'testid=files-tab-content'` reads `Select a file to view its contents`, and `control-openhands browser count 'testid=files-tab-content-mode-toggle'` is `0`.", "raw": "**No selection (`F08.files-no-selection`).** Close every open file with its `file-quick-row-close-` button. `control-openhands browser text 'testid=files-tab-content'` reads `Select a file to view its contents`, and `control-openhands browser count 'testid=files-tab-content-mode-toggle'` is `0`.", "source_line": 88, "expected": [ "with no file selected the pane reads `Select a file to view its contents` and the Rich/Plain toggle is absent." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--second-agent-turn-commit-binaries-auto-refresh", "title": "Second agent turn: commit, binaries, auto-refresh", "ids": [ "F08.files-auto-refresh", "F08.files-refresh" ], "body": "Keep Files open with `qa-notes.md` selected in Rich mode. Write `qa-f08-2.txt` with: ``In the current workspace directory run exactly this one bash command and then reply only \"done\": printf 'BIN\\000\\001' > qa-nul.bin && printf 'PK\\003\\004\\000\\000' > qa-real.docx && python3 -c \"import base64;open('qa-dot.png','wb').write(base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=='))\" && printf '\\n- edited by agent\\n' >> qa-notes.md && git add qa-notes.md && git -c user.name=QA-Agent -c user.email=qa@example.invalid commit -qm 'QA agent commit' && git log --oneline``. Run `control-openhands conversation start --stay --prompt \"$(cat qa-f08-2.txt)\" --wait --timeout 240`, then `control-openhands browser testids 'testid=files-tab-tree'` and `control-openhands browser text 'testid=file-content-viewer-markdown'`. Expected: `qa-nul.bin`, `qa-real.docx` and `qa-dot.png` are listed, and the preview ends with `edited by agent`. Known failure (reproduced 2026-10-06): neither changes until Refresh, because the auto-refresh hook skips the file queries for shell commands on purpose (see Gotchas). Repro candidate, not yet filed; a report would ask to revisit that choice. Click `testid=files-tab-refresh` once: the list gains the three files and the open preview ends with `edited by agent`.", "raw": "**Second agent turn: commit, binaries, auto-refresh (`F08.files-auto-refresh`, `F08.files-refresh`).** Keep Files open with `qa-notes.md` selected in Rich mode. Write `qa-f08-2.txt` with: ``In the current workspace directory run exactly this one bash command and then reply only \"done\": printf 'BIN\\000\\001' > qa-nul.bin && printf 'PK\\003\\004\\000\\000' > qa-real.docx && python3 -c \"import base64;open('qa-dot.png','wb').write(base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=='))\" && printf '\\n- edited by agent\\n' >> qa-notes.md && git add qa-notes.md && git -c user.name=QA-Agent -c user.email=qa@example.invalid commit -qm 'QA agent commit' && git log --oneline``. Run `control-openhands conversation start --stay --prompt \"$(cat qa-f08-2.txt)\" --wait --timeout 240`, then `control-openhands browser testids 'testid=files-tab-tree'` and `control-openhands browser text 'testid=file-content-viewer-markdown'`. Expected: `qa-nul.bin`, `qa-real.docx` and `qa-dot.png` are listed, and the preview ends with `edited by agent`. Known failure (reproduced 2026-10-06): neither changes until Refresh, because the auto-refresh hook skips the file queries for shell commands on purpose (see Gotchas). Repro candidate, not yet filed; a report would ask to revisit that choice. Click `testid=files-tab-refresh` once: the list gains the three files and the open preview ends with `edited by agent`.", "source_line": 89, "expected": [ "files the agent writes appear without a manual refresh.", "`Refresh files` re-reads the file list and the selected file's content." ], "notes": [ "Known issue [#16949](https://github.com/OpenHands/OpenHands/issues/16949) (`F08.files-refresh`): Keep Files and Changes views source-driven after refresh. Live prerequisite: An authenticated Cloud repository for the refresh-query mismatch, or real file-editor observations causing an in-place M-to-D Git refetch." ], "annotations": [ "Known failure (reproduced 2026-10-06): neither changes until Refresh, because the auto-refresh hook skips the file queries for shell commands on purpose (see Gotchas). Repro candidate, not yet filed; a report would ask to revisit that choice. Click `testid=files-tab-refresh` once: the list gains the three files and the open preview ends with `edited by agent`." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F08--fallbacks", "title": "Fallbacks", "ids": [ "F08.files-fallbacks" ], "body": "After a refresh, click `testid=file-tree-file-qa-nul.bin` (`testid=file-content-viewer-binary-fallback`, text `Binary file – preview not available`). Then click `testid=file-tree-file-qa-real.docx` (`testid=file-content-viewer-unsupported-document`, `Preview isn't available for Word files.`). Then click `testid=file-tree-file-qa-dot.png` and run `control-openhands browser eval \"document.querySelector('[data-testid=file-content-viewer-image] img').naturalWidth\"` (`1`). In Plain mode the PNG shows the binary fallback. `qa-deck.pptx` and `qa-blob.bin` from the first prompt have no NUL byte and render as text (see Gotchas).", "raw": "**Fallbacks (`F08.files-fallbacks`).** After a refresh, click `testid=file-tree-file-qa-nul.bin` (`testid=file-content-viewer-binary-fallback`, text `Binary file – preview not available`). Then click `testid=file-tree-file-qa-real.docx` (`testid=file-content-viewer-unsupported-document`, `Preview isn't available for Word files.`). Then click `testid=file-tree-file-qa-dot.png` and run `control-openhands browser eval \"document.querySelector('[data-testid=file-content-viewer-image] img').naturalWidth\"` (`1`). In Plain mode the PNG shows the binary fallback. `qa-deck.pptx` and `qa-blob.bin` from the first prompt have no NUL byte and render as text (see Gotchas).", "source_line": 90, "expected": [ "images render as `` in Rich mode. Files with NUL bytes show `Binary file – preview not available`, and Office files show `Preview isn't available for Word files.` (or PowerPoint/Excel)." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5784](https://github.com/OpenHands/software-agent-sdk/issues/5784) (`F08.files-rich-plain`, `F08.files-fallbacks`, `F08.files-open-new-window`): a path-prefixed Agent Server can set a workspace cookie whose Path omits the prefix, leaving HTML/image/file previews unauthorized. Live prerequisite: an authenticated path-prefixed owned Local server with `OH_WEB_URL` and a real cookie-authenticated preview; the ordinary root-path launch cannot exercise that cookie Path." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F08--file-editor-edits-auto-refresh-deleted-file-errors", "title": "File-editor edits auto-refresh; deleted file errors", "ids": [ "F08.files-auto-refresh", "F08.files-load-error" ], "body": "Keep Files open with `testid=file-tree-file-qa-blob.bin` selected. Run `control-openhands conversation start --stay --prompt 'Do two steps, then reply only \"done\". Step 1: run the bash command: rm qa-blob.bin. Step 2: use your file editor tool (not bash) to create the file qa-editor.txt in the current workspace directory with the single line: created by file editor' --wait --timeout 240`. Without clicking Refresh, `control-openhands browser count 'testid=file-tree-file-qa-editor.txt'` is `1` and `browser count 'testid=file-tree-file-qa-blob.bin'` is `0` (the editor observation refetches the whole list). `qa-blob.bin` stays in the open-file strip, and the same observation re-reads the open file: `control-openhands browser text 'testid=file-content-viewer-error'` reads `Failed to read qa-blob.bin: 404` and `control-openhands browser network --filter qa-blob` shows the 404 (`browser screenshot --feature F08.files-load-error --name deleted-open-file`).", "raw": "**File-editor edits auto-refresh; deleted file errors (`F08.files-auto-refresh`, `F08.files-load-error`).** Keep Files open with `testid=file-tree-file-qa-blob.bin` selected. Run `control-openhands conversation start --stay --prompt 'Do two steps, then reply only \"done\". Step 1: run the bash command: rm qa-blob.bin. Step 2: use your file editor tool (not bash) to create the file qa-editor.txt in the current workspace directory with the single line: created by file editor' --wait --timeout 240`. Without clicking Refresh, `control-openhands browser count 'testid=file-tree-file-qa-editor.txt'` is `1` and `browser count 'testid=file-tree-file-qa-blob.bin'` is `0` (the editor observation refetches the whole list). `qa-blob.bin` stays in the open-file strip, and the same observation re-reads the open file: `control-openhands browser text 'testid=file-content-viewer-error'` reads `Failed to read qa-blob.bin: 404` and `control-openhands browser network --filter qa-blob` shows the 404 (`browser screenshot --feature F08.files-load-error --name deleted-open-file`).", "source_line": 91, "expected": [ "files the agent writes appear without a manual refresh.", "a selected file that can no longer be read shows the error (`Failed to read : 404`)." ], "notes": [ "Known issue [#16689](https://github.com/OpenHands/OpenHands/issues/16689) (`F08.files-load-error`): When `workspace/` paths throw a 401, call the workspace-session endpoint. Live prerequisite: The original fresh-VM URL, auth/cookie state and exact request sequence; the report says the trigger is unclear.", "Known issue [#15567](https://github.com/OpenHands/OpenHands/issues/15567) (`F08.files-load-error`): Workspace file preview returns 401 Unauthorized on non-HTTPS non-loopback hosts. Live prerequisite: A plain-HTTP non-loopback deployment and browser accessing that origin, with workspace-cookie observation.", "Known issue [#15519](https://github.com/OpenHands/OpenHands/issues/15519) (`F08.files-load-error`): \"files\" tab in Files side panel throws 401 Unauthorized. Live prerequisite: The reported Linux Docker deployment/auth topology; this isolated run uses the macOS source launcher.", "Known issue [OpenHands/software-agent-sdk#4562](https://github.com/OpenHands/software-agent-sdk/issues/4562) (`F08.files-load-error`): Plain-HTTP non-loopback workspace previews can return401 after the browser rejects the workspace-session cookie. Live prerequisite: A plain-HTTP non-loopback same-origin browser deployment with real workspace-cookie observation." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F08--open-from-chat", "title": "Open from chat", "ids": [ "F08.files-open-from-chat" ], "body": "Run `control-openhands conversation start --stay --prompt 'Without running any tool, reply with exactly this text and nothing else: See `src/calc.py` for the code.' --wait --timeout 180`. Close the drawer, then run `control-openhands browser click 'testid=agent-message >> nth=-1 >> testid=markdown-file-path-link'`. `right-panel-toggle` `aria-pressed` turns `true`, `testid=files-tab` is visible and `file-quick-row-item-src/calc.py` is the selected tab. The same path in the prompt bubble works too: close the drawer and click `'testid=user-message >> testid=markdown-file-path-link'`. Phone: select another file (`control-openhands browser click 'testid=file-quick-row-item-qa-notes.md'`), run `control-openhands browser viewport phone`, then the same agent-message click with `--expect-url '/panel$'`: the URL becomes `/conversations//panel`, `browser visible 'testid=files-tab'` is `true` and `control-openhands browser attr 'testid=file-quick-row-item-src/calc.py' aria-selected` is `true` (`browser screenshot --feature F08.files-open-from-chat --name phone-after-click`). Return with `control-openhands browser viewport desktop`: the app replaces `/panel` with `/conversations/` (`control-openhands browser wait-url '/conversations/[^/]+$'`) and shows the drawer beside the chat.", "raw": "**Open from chat (`F08.files-open-from-chat`).** Run `control-openhands conversation start --stay --prompt 'Without running any tool, reply with exactly this text and nothing else: See `src/calc.py` for the code.' --wait --timeout 180`. Close the drawer, then run `control-openhands browser click 'testid=agent-message >> nth=-1 >> testid=markdown-file-path-link'`. `right-panel-toggle` `aria-pressed` turns `true`, `testid=files-tab` is visible and `file-quick-row-item-src/calc.py` is the selected tab. The same path in the prompt bubble works too: close the drawer and click `'testid=user-message >> testid=markdown-file-path-link'`. Phone: select another file (`control-openhands browser click 'testid=file-quick-row-item-qa-notes.md'`), run `control-openhands browser viewport phone`, then the same agent-message click with `--expect-url '/panel$'`: the URL becomes `/conversations//panel`, `browser visible 'testid=files-tab'` is `true` and `control-openhands browser attr 'testid=file-quick-row-item-src/calc.py' aria-selected` is `true` (`browser screenshot --feature F08.files-open-from-chat --name phone-after-click`). Return with `control-openhands browser viewport desktop`: the app replaces `/panel` with `/conversations/` (`control-openhands browser wait-url '/conversations/[^/]+$'`) and shows the drawer beside the chat.", "source_line": 92, "expected": [ "clicking a workspace path in an agent (or user) message opens the drawer on Files with that file selected." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--tool-card-path-chip", "title": "Tool-card path chip", "ids": [ "F08.files-open-from-tool-chip" ], "body": "After the file-editor turn above, close the drawer and expand its tool group and card: `control-openhands browser click 'text=2 actions completed'`, then `control-openhands browser click 'role=region[name=\"Collapse actions\"] >> role=button[name=\"Expand\"] >> nth=-1'` (the `Create qa-editor.txt` card). `control-openhands browser text 'testid=file-path-chip'` is the absolute `/workspace/qa-repo/qa-editor.txt`. Run `control-openhands browser click 'testid=file-path-chip'`: `right-panel-toggle` `aria-pressed` turns `true`, the selected strip tab is `file-quick-row-item-qa-editor.txt` (relative) and `browser text 'testid=files-tab-content'` ends with `created by file editor`.", "raw": "**Tool-card path chip (`F08.files-open-from-tool-chip`).** After the file-editor turn above, close the drawer and expand its tool group and card: `control-openhands browser click 'text=2 actions completed'`, then `control-openhands browser click 'role=region[name=\"Collapse actions\"] >> role=button[name=\"Expand\"] >> nth=-1'` (the `Create qa-editor.txt` card). `control-openhands browser text 'testid=file-path-chip'` is the absolute `/workspace/qa-repo/qa-editor.txt`. Run `control-openhands browser click 'testid=file-path-chip'`: `right-panel-toggle` `aria-pressed` turns `true`, the selected strip tab is `file-quick-row-item-qa-editor.txt` (relative) and `browser text 'testid=files-tab-content'` ends with `created by file editor`.", "source_line": 93, "expected": [ "the path chip on a file-editor tool card (absolute path) opens the drawer on Files with that file selected." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--commits-list", "title": "Commits list", "ids": [ "F08.uncommitted-changes", "F08.commit-rows" ], "body": "Click `testid=conversation-tab-commits` and run `control-openhands browser text 'testid=commit-list'`. After the first prompt it reads `-`, `Uncommitted`, `6 files`, then `4bf8272`-style SHA, `Initial fixture commit` and `… ago`; after the second prompt `QA agent commit` appears above it, with author names `QA-Agent` and `QA Fixture` shown because there are now two authors. Run `control-openhands browser click 'testid=uncommitted-changes-row-toggle'` and `control-openhands browser text 'testid=uncommitted-changes-row-content'` (the changed paths; `aria-expanded` is `true`). Run `control-openhands browser click 'testid=commit-row >> has-text=Initial fixture commit >> testid=commit-row-toggle'`; Uncommitted's `aria-expanded` turns `false`, and `control-openhands browser text 'testid=commit-row-content'` lists `README.md`, `src/calc.py`, `src/test_calc.py`. After the third prompt the Uncommitted count is `8 files`.", "raw": "**Commits list (`F08.uncommitted-changes`, `F08.commit-rows`).** Click `testid=conversation-tab-commits` and run `control-openhands browser text 'testid=commit-list'`. After the first prompt it reads `-`, `Uncommitted`, `6 files`, then `4bf8272`-style SHA, `Initial fixture commit` and `… ago`; after the second prompt `QA agent commit` appears above it, with author names `QA-Agent` and `QA Fixture` shown because there are now two authors. Run `control-openhands browser click 'testid=uncommitted-changes-row-toggle'` and `control-openhands browser text 'testid=uncommitted-changes-row-content'` (the changed paths; `aria-expanded` is `true`). Run `control-openhands browser click 'testid=commit-row >> has-text=Initial fixture commit >> testid=commit-row-toggle'`; Uncommitted's `aria-expanded` turns `false`, and `control-openhands browser text 'testid=commit-row-content'` lists `README.md`, `src/calc.py`, `src/test_calc.py`. After the third prompt the Uncommitted count is `8 files`.", "source_line": 94, "expected": [ "the top row reads `-`, `Uncommitted` and a file count; expanding it lists the changed paths.", "each commit row shows the short SHA, subject, author (only when there are several authors) and relative time. Rows are single-open accordions, and new agent commits appear without a reload." ], "notes": [ "Known issue [#16716](https://github.com/OpenHands/OpenHands/issues/16716) (`F08.uncommitted-changes`): Diff view is regularly out of sync. Live prerequisite: The original simultaneous-agent/shared-repository state or a deterministic stale-diff trigger; the report only suggests that timing.", "Known issue [#16454](https://github.com/OpenHands/OpenHands/issues/16454) (`F08.uncommitted-changes`): Diff list is showing duplicate entries. Live prerequisite: The original Git changes response and update sequence producing duplicate rows; the report provides no trigger.", "Known issue [#15692](https://github.com/OpenHands/OpenHands/issues/15692) (`F08.uncommitted-changes`): Diff still shows after agent has committed and pushed. Live prerequisite: A real agent commit/push terminal observation while the diff remains mounted, plus a disposable Git remote." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F08--diff-modes", "title": "Diff modes", "ids": [ "F08.diff-view-modes" ], "body": "In the expanded Uncommitted row, run `control-openhands browser click 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/calc.py >> testid=collapse'`. `control-openhands browser text 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/calc.py >> testid=editor-container'` shows the added `def sub` lines, and `view-mode-diff` has `aria-pressed` `true`. The row's buttons are named: `control-openhands browser snapshot 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/calc.py'` lists `button \"Old version\"`, `button \"Diff\" [pressed]`, `button \"New version\"` and `button \"Collapse\"` (the same `collapse` button reads `Expand` while the row is collapsed), so `… >> role=button[name=\"Old version\"]` works as well as the test id. Click `… >> testid=view-mode-old` (2 lines), then `… >> testid=view-mode-new` (4 lines), then `view-mode-diff`. Run `control-openhands browser errors --app-only` before and after: each collapse and each diff→old switch adds `TextModel got disposed before DiffEditorWidget model got reset` (fail, #17567).", "raw": "**Diff modes (`F08.diff-view-modes`).** In the expanded Uncommitted row, run `control-openhands browser click 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/calc.py >> testid=collapse'`. `control-openhands browser text 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/calc.py >> testid=editor-container'` shows the added `def sub` lines, and `view-mode-diff` has `aria-pressed` `true`. The row's buttons are named: `control-openhands browser snapshot 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/calc.py'` lists `button \"Old version\"`, `button \"Diff\" [pressed]`, `button \"New version\"` and `button \"Collapse\"` (the same `collapse` button reads `Expand` while the row is collapsed), so `… >> role=button[name=\"Old version\"]` works as well as the test id. Click `… >> testid=view-mode-old` (2 lines), then `… >> testid=view-mode-new` (4 lines), then `view-mode-diff`. Run `control-openhands browser errors --app-only` before and after: each collapse and each diff→old switch adds `TextModel got disposed before DiffEditorWidget model got reset` (fail, #17567).", "source_line": 95, "expected": [ "expanding a changed file shows its diff; the view-mode buttons (`Old version`, `Diff`, `New version`) switch between old, diff and new without page errors." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--markdown-preview", "title": "Markdown preview", "ids": [ "F08.diff-markdown-preview" ], "body": "In this order `qa-notes.md` is already committed, so check it in the `QA agent commit` row: `control-openhands browser click 'testid=commit-row >> has-text=QA agent commit >> testid=commit-row-toggle'`, then with `F='testid=commit-row >> has-text=QA agent commit >> testid=file-diff-viewer-outer >> has-text=qa-notes.md'` click `\"$F >> testid=collapse\"` and `\"$F >> testid=view-mode-new\"`. `control-openhands browser count \"$F >> testid=markdown-preview\"` is `1`, rendering `

QA notes

`, **world** in bold and the `edited by agent` list item. (Before the second prompt the same check works under `testid=uncommitted-changes-row-content`.) Re-expand Uncommitted with `uncommitted-changes-row-toggle` for the next bullet.", "raw": "**Markdown preview (`F08.diff-markdown-preview`).** In this order `qa-notes.md` is already committed, so check it in the `QA agent commit` row: `control-openhands browser click 'testid=commit-row >> has-text=QA agent commit >> testid=commit-row-toggle'`, then with `F='testid=commit-row >> has-text=QA agent commit >> testid=file-diff-viewer-outer >> has-text=qa-notes.md'` click `\"$F >> testid=collapse\"` and `\"$F >> testid=view-mode-new\"`. `control-openhands browser count \"$F >> testid=markdown-preview\"` is `1`, rendering `

QA notes

`, **world** in bold and the `edited by agent` list item. (Before the second prompt the same check works under `testid=uncommitted-changes-row-content`.) Re-expand Uncommitted with `uncommitted-changes-row-toggle` for the next bullet.", "source_line": 96, "expected": [ "a Markdown file in new (or old) mode renders as a prose preview." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--deleted-file", "title": "Deleted file", "ids": [ "F08.diff-deleted-file" ], "body": "Expand `src/test_calc.py` in Uncommitted. `control-openhands browser text 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/test_calc.py >> testid=file-deleted-message'` reads `This file was deleted. Its previous contents are no longer available for preview.`", "raw": "**Deleted file (`F08.diff-deleted-file`).** Expand `src/test_calc.py` in Uncommitted. `control-openhands browser text 'testid=uncommitted-changes-row-content >> testid=file-diff-viewer-outer >> has-text=src/test_calc.py >> testid=file-deleted-message'` reads `This file was deleted. Its previous contents are no longer available for preview.`", "source_line": 97, "expected": [ "a deleted working-tree file reads `This file was deleted. Its previous contents are no longer available for preview.`" ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--from-the-overview", "title": "From the overview", "ids": [ "F08.changes-from-overview" ], "body": "With the drawer closed, run `control-openhands browser click 'testid=conversation-overview-toggle'` and `control-openhands browser click 'testid=conversation-overview-diffs'` (its siblings `conversation-overview-diffs-additions`/`-deletions` show `+N`/`-N`; deleted files are not counted). `right-panel-toggle` `aria-pressed` is `true`, `testid=commit-list` is visible, `uncommitted-changes-row-toggle` has `aria-expanded` `true` and `testid=conversation-overview-panel` is hidden. After `browser reload` and the toggle, Commits is still selected.", "raw": "**From the overview (`F08.changes-from-overview`).** With the drawer closed, run `control-openhands browser click 'testid=conversation-overview-toggle'` and `control-openhands browser click 'testid=conversation-overview-diffs'` (its siblings `conversation-overview-diffs-additions`/`-deletions` show `+N`/`-N`; deleted files are not counted). `right-panel-toggle` `aria-pressed` is `true`, `testid=commit-list` is visible, `uncommitted-changes-row-toggle` has `aria-expanded` `true` and `testid=conversation-overview-panel` is hidden. After `browser reload` and the toggle, Commits is still selected.", "source_line": 98, "expected": [ "the overview's Changes row opens the drawer on Commits with Uncommitted expanded and closes the overview." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--empty-states", "title": "Empty states", "ids": [ "F08.commits-states", "F08.files-empty-workspace" ], "body": "Run `control-openhands fixture folder`, which makes `/workspace/qa-folder`, a plain folder (no `.git`) with `notes.md`; by the time the conversation has run it holds an empty `.git` (seen live; not created by the prompt), so it ends up a repo with no commits. Run `control-openhands browser goto /` (the browser is on the qa-repo conversation) and pick it with the workspace picker as in Preconditions (`folder-browser-entry-qa-folder`). Then run `control-openhands conversation start --stay --prompt 'Run exactly this bash command, then reply only \"done\": rm notes.md && git status --short' --wait --timeout 180`. Open the drawer and click `testid=conversation-tab-commits`; `control-openhands browser count 'text=\"No commits yet\"'` is `1` (`browser screenshot --feature F08.commits-states --name no-commits`). On Files, click `testid=files-tab-refresh`; `control-openhands browser text 'testid=files-tab-tree'` reads `No files in workspace`. `Loading changes...` (`testid=commits-tab-status`) shows briefly on the first Commits open after a load: on `/conversations/` with Files selected, `browser reload`, open the drawer, then `control-openhands browser click 'testid=conversation-tab-commits' --observe 'testid=commits-tab-status' --observe-ms 4000` records `Loading changes...` for a few ms. `Waiting for runtime to start...` (agent state init/loading/error) was not reached: `service stop agent-server` and `restart` both left `No commits yet` on screen.", "raw": "**Empty states (`F08.commits-states`, `F08.files-empty-workspace`).** Run `control-openhands fixture folder`, which makes `/workspace/qa-folder`, a plain folder (no `.git`) with `notes.md`; by the time the conversation has run it holds an empty `.git` (seen live; not created by the prompt), so it ends up a repo with no commits. Run `control-openhands browser goto /` (the browser is on the qa-repo conversation) and pick it with the workspace picker as in Preconditions (`folder-browser-entry-qa-folder`). Then run `control-openhands conversation start --stay --prompt 'Run exactly this bash command, then reply only \"done\": rm notes.md && git status --short' --wait --timeout 180`. Open the drawer and click `testid=conversation-tab-commits`; `control-openhands browser count 'text=\"No commits yet\"'` is `1` (`browser screenshot --feature F08.commits-states --name no-commits`). On Files, click `testid=files-tab-refresh`; `control-openhands browser text 'testid=files-tab-tree'` reads `No files in workspace`. `Loading changes...` (`testid=commits-tab-status`) shows briefly on the first Commits open after a load: on `/conversations/` with Files selected, `browser reload`, open the drawer, then `control-openhands browser click 'testid=conversation-tab-commits' --observe 'testid=commits-tab-status' --observe-ms 4000` records `Loading changes...` for a few ms. `Waiting for runtime to start...` (agent state init/loading/error) was not reached: `service stop agent-server` and `restart` both left `No commits yet` on screen.", "source_line": 99, "expected": [ "with no commits and no changes the Commits tab shows `No commits yet`. While the runtime is inactive it shows `Waiting for runtime to start...`, and while loading `Loading changes...`.", "a workspace with no files shows `No files in workspace`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--commit-cap", "title": "Commit cap", "ids": [ "F08.commits-cap" ], "body": "Arrange more than 50 commits in the fixture (read-only for the UI): `for i in $(seq 1 50); do git -C \"$OH_VERIFY_RUN/workspace/qa-repo\" -c user.name=QA-Bulk -c user.email=qa@example.invalid commit -q --allow-empty -m \"QA bulk $i\"; done`. On the qa-repo conversation (`control-openhands browser goto /conversations/`; Empty states left the qa-folder conversation) run `browser reload`, open the drawer on Commits and wait for `testid=commit-list`. `control-openhands browser text 'testid=commit-list-cap-notice'` reads `Showing the latest 50 commits` and `browser count 'testid=commit-row'` is `50`. Do this last: it changes the commit list the other bullets read.", "raw": "**Commit cap (`F08.commits-cap`).** Arrange more than 50 commits in the fixture (read-only for the UI): `for i in $(seq 1 50); do git -C \"$OH_VERIFY_RUN/workspace/qa-repo\" -c user.name=QA-Bulk -c user.email=qa@example.invalid commit -q --allow-empty -m \"QA bulk $i\"; done`. On the qa-repo conversation (`control-openhands browser goto /conversations/`; Empty states left the qa-folder conversation) run `browser reload`, open the drawer on Commits and wait for `testid=commit-list`. `control-openhands browser text 'testid=commit-list-cap-notice'` reads `Showing the latest 50 commits` and `browser count 'testid=commit-row'` is `50`. Do this last: it changes the commit list the other bullets read.", "source_line": 100, "expected": [ "the history loads the latest 50 commits; a longer history ends with `Showing the latest 50 commits`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--phone-panel-page", "title": "Phone panel page", "ids": [ "F08.phone-panel-page" ], "body": "Run `control-openhands browser viewport phone` on `/conversations/`, then `control-openhands browser click 'testid=right-panel-toggle' --expect-url '/panel$'`. `control-openhands browser bbox 'testid=files-tab'` gives `insideViewport` `true` and `pageHorizontalOverflow` `false`, with Commits diffs usable at 390 px (`browser screenshot --feature F08.phone-panel-page --name panel`). `control-openhands browser attr 'testid=conversation-mobile-panel-back' aria-label` is `Back`. Run `control-openhands browser click 'testid=conversation-mobile-panel-back' --expect-url '/conversations/[^/]+$'`; the chat is visible. Return with `control-openhands browser viewport desktop`.", "raw": "**Phone panel page (`F08.phone-panel-page`).** Run `control-openhands browser viewport phone` on `/conversations/`, then `control-openhands browser click 'testid=right-panel-toggle' --expect-url '/panel$'`. `control-openhands browser bbox 'testid=files-tab'` gives `insideViewport` `true` and `pageHorizontalOverflow` `false`, with Commits diffs usable at 390 px (`browser screenshot --feature F08.phone-panel-page --name panel`). `control-openhands browser attr 'testid=conversation-mobile-panel-back' aria-label` is `Back`. Run `control-openhands browser click 'testid=conversation-mobile-panel-back' --expect-url '/conversations/[^/]+$'`; the chat is visible. Return with `control-openhands browser viewport desktop`.", "source_line": 101, "expected": [ "at 1024 px or narrower the toggle navigates to `/conversations//panel`, a full-screen page with a Back chevron, the same tab bar and the tab body." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--direct-panel-url", "title": "Direct panel URL", "ids": [ "F08.panel-direct-url" ], "body": "At desktop run `control-openhands browser goto /conversations//panel`. The app replaces the URL with `/conversations/` (the `goto` result already shows it; `control-openhands browser url` confirms), `control-openhands browser count 'testid=conversation-mobile-panel-back'` is `0`, `browser visible 'testid=chat-interface'` is `true` and `right-panel-toggle` `aria-pressed` is `true` (the drawer opens beside the chat; `browser screenshot --feature F08.panel-direct-url --name desktop-redirect`). Then run `control-openhands browser viewport phone` and the same `goto`: the URL keeps `/panel`, the `conversation-mobile-panel-back` count is `1` and `chat-interface` is not visible. `control-openhands browser click 'testid=conversation-mobile-panel-back' --expect-url '/conversations/[^/]+$'` returns to the chat. Return with `control-openhands browser viewport desktop`.", "raw": "**Direct panel URL (`F08.panel-direct-url`).** At desktop run `control-openhands browser goto /conversations//panel`. The app replaces the URL with `/conversations/` (the `goto` result already shows it; `control-openhands browser url` confirms), `control-openhands browser count 'testid=conversation-mobile-panel-back'` is `0`, `browser visible 'testid=chat-interface'` is `true` and `right-panel-toggle` `aria-pressed` is `true` (the drawer opens beside the chat; `browser screenshot --feature F08.panel-direct-url --name desktop-redirect`). Then run `control-openhands browser viewport phone` and the same `goto`: the URL keeps `/panel`, the `conversation-mobile-panel-back` count is `1` and `chat-interface` is not visible. `control-openhands browser click 'testid=conversation-mobile-panel-back' --expect-url '/conversations/[^/]+$'` returns to the chat. Return with `control-openhands browser viewport desktop`.", "source_line": 102, "expected": [ "opening `/conversations//panel` directly renders the panel page at 1024 px or narrower, and Back returns to the chat; wider, the URL is replaced with `/conversations/` and the drawer opens beside the chat." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F08--errors", "title": "Errors", "ids": [], "body": "After each group run `control-openhands browser errors --app-only`. The Monaco disposal error above belongs to this family. 404s on `/api/llm/balance` and `.agents_tmp/` downloads come from the Usage and Planner tabs (F27).", "raw": "**Errors.** After each group run `control-openhands browser errors --app-only`. The Monaco disposal error above belongs to this family. 404s on `/api/llm/balance` and `.agents_tmp/` downloads come from the Usage and Planner tabs (F27).", "source_line": 103, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F08--rich-fenced-code", "title": "Rich fenced code", "ids": [ "F08.files-rich-plain" ], "body": "Arrange a Markdown file in the selected workspace containing a heading and a fenced Python `print(\"qa\")` block. Open it from Files in Rich mode and take `browser screenshot --feature F08.files-rich-plain --name fenced-code`. Known failure (reproduced 2026-10-12, [#16802](https://github.com/OpenHands/OpenHands/issues/16802)): the code renders in nested dark/gray containers with repeated padding. This is a layout-only observation; the measured `pre` and prose ancestor had zero border width, so it is not a claim of visible CSS borders.", "raw": "**Rich fenced code (`F08.files-rich-plain`).** Arrange a Markdown file in the selected workspace containing a heading and a fenced Python `print(\"qa\")` block. Open it from Files in Rich mode and take `browser screenshot --feature F08.files-rich-plain --name fenced-code`. Known failure (reproduced 2026-10-12, [#16802](https://github.com/OpenHands/OpenHands/issues/16802)): the code renders in nested dark/gray containers with repeated padding. This is a layout-only observation; the measured `pre` and prose ancestor had zero border width, so it is not a claim of visible CSS borders.", "source_line": 105, "expected": [ "Rich highlights code, renders Markdown and shows HTML in an iframe sandboxed without scripts. Plain shows highlighted source. The mode persists across reloads." ], "notes": [ "Known issue [#15501](https://github.com/OpenHands/OpenHands/issues/15501) (`F08.files-rich-plain`): Generated-page JavaScript was blocked by an iframe sandbox. Files Rich intentionally disables scripts; the report does not identify whether Files or Browser was used. Live prerequisite: The original preview panel and file URL or minimal page identifying the intended surface; Files Rich deliberately blocks scripts, while Browser is separate.", "Known issue [OpenHands/software-agent-sdk#5784](https://github.com/OpenHands/software-agent-sdk/issues/5784) (`F08.files-rich-plain`, `F08.files-fallbacks`, `F08.files-open-new-window`): a path-prefixed Agent Server can set a workspace cookie whose Path omits the prefix, leaving HTML/image/file previews unauthorized. Live prerequisite: an authenticated path-prefixed owned Local server with `OH_WEB_URL` and a real cookie-authenticated preview; the ordinary root-path launch cannot exercise that cookie Path." ], "annotations": [ "Known failure (reproduced 2026-10-12, [#16802](https://github.com/OpenHands/OpenHands/issues/16802)): the code renders in nested dark/gray containers with repeated padding. This is a layout-only observation; the measured `pre` and prose ancestor had zero border width, so it is not a claim of visible CSS borders." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F08--git-filename-with-a-space", "title": "Git filename with a space", "ids": [ "F08.uncommitted-changes" ], "body": "In a disposable workspace Git fixture, create and commit `my notes.md`, then modify it (plain shell Git/file commands are fixture setup, not UI proof). Open its empty Local conversation's Commits tab and read `browser network --filter '/git/'`. Expected: the modified path and other changes stay listed. Known failure (reproduced 2026-10-12, [OpenHands/software-agent-sdk#5642](https://github.com/OpenHands/software-agent-sdk/issues/5642)): the scoped changes call returns 400 and hides the Uncommitted changes. Only the space-name case was tested; the same issue's non-ASCII and rename cases still require their own checks.", "raw": "**Git filename with a space (`F08.uncommitted-changes`).** In a disposable workspace Git fixture, create and commit `my notes.md`, then modify it (plain shell Git/file commands are fixture setup, not UI proof). Open its empty Local conversation's Commits tab and read `browser network --filter '/git/'`. Expected: the modified path and other changes stay listed. Known failure (reproduced 2026-10-12, [OpenHands/software-agent-sdk#5642](https://github.com/OpenHands/software-agent-sdk/issues/5642)): the scoped changes call returns 400 and hides the Uncommitted changes. Only the space-name case was tested; the same issue's non-ASCII and rename cases still require their own checks.", "source_line": 106, "expected": [ "the top row reads `-`, `Uncommitted` and a file count; expanding it lists the changed paths." ], "notes": [ "Known issue [#16716](https://github.com/OpenHands/OpenHands/issues/16716) (`F08.uncommitted-changes`): Diff view is regularly out of sync. Live prerequisite: The original simultaneous-agent/shared-repository state or a deterministic stale-diff trigger; the report only suggests that timing.", "Known issue [#16454](https://github.com/OpenHands/OpenHands/issues/16454) (`F08.uncommitted-changes`): Diff list is showing duplicate entries. Live prerequisite: The original Git changes response and update sequence producing duplicate rows; the report provides no trigger.", "Known issue [#15692](https://github.com/OpenHands/OpenHands/issues/15692) (`F08.uncommitted-changes`): Diff still shows after agent has committed and pushed. Live prerequisite: A real agent commit/push terminal observation while the diff remains mounted, plus a disposable Git remote." ], "annotations": [ "Known failure (reproduced 2026-10-12, [OpenHands/software-agent-sdk#5642](https://github.com/OpenHands/software-agent-sdk/issues/5642)): the scoped changes call returns 400 and hides the Uncommitted changes. Only the space-name case was tested; the same issue's non-ASCII and rename cases still require their own checks." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F08--file-listing-above-one-mib", "title": "File listing above one MiB", "ids": [ "F08.files-tree" ], "body": "In a disposable fixture, arrange 1800 tiny files beneath three 220-character directory segments so the sorted relative-path listing exceeds one MiB while remaining below the 2000-file cap. Keep a normal tracked README at the root. Open Files, click `testid=files-tab-refresh`, then count `role=button[name=\"README.md\"][exact]` and inspect the tree. Known failure (reproduced 2026-10-12, [OpenHands/software-agent-sdk#5638](https://github.com/OpenHands/software-agent-sdk/issues/5638)): the README disappears and a spurious partial-path folder appears because the command API returns the tail output chunk only. The greater-than-100-chunk/null-exit case was not driven.", "raw": "**File listing above one MiB (`F08.files-tree`).** In a disposable fixture, arrange 1800 tiny files beneath three 220-character directory segments so the sorted relative-path listing exceeds one MiB while remaining below the 2000-file cap. Keep a normal tracked README at the root. Open Files, click `testid=files-tab-refresh`, then count `role=button[name=\"README.md\"][exact]` and inspect the tree. Known failure (reproduced 2026-10-12, [OpenHands/software-agent-sdk#5638](https://github.com/OpenHands/software-agent-sdk/issues/5638)): the README disappears and a spurious partial-path folder appears because the command API returns the tail output chunk only. The greater-than-100-chunk/null-exit case was not driven.", "source_line": 107, "expected": [ "a tree of folders, collapsed at first, and files. Clicking a file selects it and adds it to the open-file strip." ], "notes": [ "Known issue [#18108](https://github.com/OpenHands/OpenHands/issues/18108) (`F08.files-tree`): Files tab shows \"No files in workspace\" on cloud when the agent writes outside /workspace/project. Live prerequisite: An authenticated Enterprise Cloud sandbox, no repository selected, with files outside its default working directory.", "Known issue [#17773](https://github.com/OpenHands/OpenHands/issues/17773) (`F08.files-tree`): Files tab always shows \"No files in workspace\" on Windows local backend. Live prerequisite: A Windows local Agent Server using cmd.exe; this run is macOS.", "Known issue [#16331](https://github.com/OpenHands/OpenHands/issues/16331) (`F08.files-tree`): file preview panel stops working during streaming. Live prerequisite: A real streaming/tool-running conversation while selecting a second workspace file." ], "annotations": [ "Known failure (reproduced 2026-10-12, [OpenHands/software-agent-sdk#5638](https://github.com/OpenHands/software-agent-sdk/issues/5638)): the README disappears and a spurious partial-path folder appears because the command API returns the tail output chunk only. The greater-than-100-chunk/null-exit case was not driven." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F08--subdirectory-git-paths", "title": "Subdirectory Git paths", "ids": [ "F08.uncommitted-changes", "F08.diff-view-modes" ], "body": "Expected: a workspace at `repo/src` lists changes beneath that folder relative to it, and its diffs open. Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5843](https://github.com/OpenHands/software-agent-sdk/issues/5843)): tracked paths retain the repository prefix while untracked paths are relative to `src`; opening displayed `src/app.py` requests `repo/src/src/app.py` and returns400. The ordinary repository-root control lists correct paths. The recipe creates only disposable, idle offline conversations.\n ```sh known-bug F08.diff-view-modes subdirectory-paths\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n repo = fixture / \"repo\"\n (repo / \"src\").mkdir(parents=True)\n git(repo, \"init\", \"-q\", \"-b\", \"main\")\n (repo / \"README.md\").write_text(\"root before\\n\")\n (repo / \"src/app.py\").write_text(\"x = 1\\n\")\n git(repo, \"add\", \".\")\n git(repo, \"-c\", \"user.name=QA\", \"-c\", \"user.email=qa@example.invalid\", \"commit\", \"-qm\", \"Owned fixture\")\n (repo / \"README.md\").write_text(\"root after\\n\")\n (repo / \"src/app.py\").write_text(\"x = 2\\n\")\n (repo / \"src/new.py\").write_text(\"new\\n\")\n print(create(repo / \"src\"))\n PY\n )\"\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser click 'testid=right-panel-toggle'\n control-openhands browser click 'testid=conversation-tab-commits'\n control-openhands browser wait-text Uncommitted --timeout 10000\n control-openhands browser click 'testid=uncommitted-changes-row-toggle'\n control-openhands browser network --clear\n control-openhands browser click 'testid=file-diff-viewer-outer >> has-text=src/app.py >> testid=collapse' --observe 'testid=uncommitted-changes-row-content' --observe-ms 1000\n control-openhands browser screenshot --feature F08.diff-view-modes --name subdir-mixed-paths\n control-openhands assert --field recent.0.status --equals 200 --bug-shows 400 -- browser network --filter '/git/diff' # bug\n ```", "raw": "**Subdirectory Git paths (`F08.uncommitted-changes`, `F08.diff-view-modes`).** Expected: a workspace at `repo/src` lists changes beneath that folder relative to it, and its diffs open. Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5843](https://github.com/OpenHands/software-agent-sdk/issues/5843)): tracked paths retain the repository prefix while untracked paths are relative to `src`; opening displayed `src/app.py` requests `repo/src/src/app.py` and returns400. The ordinary repository-root control lists correct paths. The recipe creates only disposable, idle offline conversations.\n ```sh known-bug F08.diff-view-modes subdirectory-paths\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n repo = fixture / \"repo\"\n (repo / \"src\").mkdir(parents=True)\n git(repo, \"init\", \"-q\", \"-b\", \"main\")\n (repo / \"README.md\").write_text(\"root before\\n\")\n (repo / \"src/app.py\").write_text(\"x = 1\\n\")\n git(repo, \"add\", \".\")\n git(repo, \"-c\", \"user.name=QA\", \"-c\", \"user.email=qa@example.invalid\", \"commit\", \"-qm\", \"Owned fixture\")\n (repo / \"README.md\").write_text(\"root after\\n\")\n (repo / \"src/app.py\").write_text(\"x = 2\\n\")\n (repo / \"src/new.py\").write_text(\"new\\n\")\n print(create(repo / \"src\"))\n PY\n )\"\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser click 'testid=right-panel-toggle'\n control-openhands browser click 'testid=conversation-tab-commits'\n control-openhands browser wait-text Uncommitted --timeout 10000\n control-openhands browser click 'testid=uncommitted-changes-row-toggle'\n control-openhands browser network --clear\n control-openhands browser click 'testid=file-diff-viewer-outer >> has-text=src/app.py >> testid=collapse' --observe 'testid=uncommitted-changes-row-content' --observe-ms 1000\n control-openhands browser screenshot --feature F08.diff-view-modes --name subdir-mixed-paths\n control-openhands assert --field recent.0.status --equals 200 --bug-shows 400 -- browser network --filter '/git/diff' # bug\n ```", "source_line": 109, "expected": [ "the top row reads `-`, `Uncommitted` and a file count; expanding it lists the changed paths.", "expanding a changed file shows its diff; the view-mode buttons (`Old version`, `Diff`, `New version`) switch between old, diff and new without page errors." ], "notes": [ "Known issue [#16716](https://github.com/OpenHands/OpenHands/issues/16716) (`F08.uncommitted-changes`): Diff view is regularly out of sync. Live prerequisite: The original simultaneous-agent/shared-repository state or a deterministic stale-diff trigger; the report only suggests that timing.", "Known issue [#16454](https://github.com/OpenHands/OpenHands/issues/16454) (`F08.uncommitted-changes`): Diff list is showing duplicate entries. Live prerequisite: The original Git changes response and update sequence producing duplicate rows; the report provides no trigger.", "Known issue [#15692](https://github.com/OpenHands/OpenHands/issues/15692) (`F08.uncommitted-changes`): Diff still shows after agent has committed and pushed. Live prerequisite: A real agent commit/push terminal observation while the diff remains mounted, plus a disposable Git remote." ], "annotations": [ "Known failure (reproduced 2026-10-16, SDK 1.55.0, [OpenHands/software-agent-sdk#5843](https://github.com/OpenHands/software-agent-sdk/issues/5843)): tracked paths retain the repository prefix while untracked paths are relative to `src`; opening displayed `src/app.py` requests `repo/src/src/app.py` and returns400. The ordinary repository-root control lists correct paths. The recipe creates only disposable, idle offline conversations." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F08--plain-parent-containing-a-repository", "title": "Plain parent containing a repository", "ids": [ "F08.uncommitted-changes" ], "body": "Expected: a plain workspace holding one dirty repository exposes that repository's changes. Known failure (reproduced 2026-10-16, pinned SDK 1.55.0, [OpenHands/software-agent-sdk#5641](https://github.com/OpenHands/software-agent-sdk/issues/5641)): the changes request returns200 with no changes and Commits reads `No commits yet` although the child repository is dirty. The upstream fix is closed but not included in the current pin. The recipe removes only its own verified empty parent `.git`; never remove a user's repository.\n ```sh known-bug F08.uncommitted-changes nested-repository\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n import shutil\n parent = fixture / \"parent\"\n child = parent / \"proj\"\n child.mkdir(parents=True)\n git(child, \"init\", \"-q\", \"-b\", \"main\")\n (child / \"README.md\").write_text(\"before\\n\")\n git(child, \"add\", \".\")\n git(child, \"-c\", \"user.name=QA\", \"-c\", \"user.email=qa@example.invalid\", \"commit\", \"-qm\", \"Owned nested fixture\")\n (child / \"README.md\").write_text(\"after\\n\")\n (child / \"notes.txt\").write_text(\"untracked\\n\")\n conversation_id = create(parent)\n empty_git = parent / \".git\"\n if empty_git.exists():\n assert empty_git.is_dir() and not empty_git.is_symlink()\n top = pathlib.Path(subprocess.check_output([\"git\", \"-C\", str(parent), \"rev-parse\", \"--show-toplevel\"], text=True).strip())\n assert top.resolve() == parent.resolve()\n assert not subprocess.check_output([\"git\", \"-C\", str(parent), \"ls-files\"], text=True).strip()\n shutil.rmtree(empty_git)\n print(conversation_id)\n PY\n )\"\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser click 'testid=right-panel-toggle'\n control-openhands browser click 'testid=conversation-tab-commits'\n control-openhands browser wait '#conversation-tab-panel main'\n control-openhands browser wait 'testid=commits-tab-status' --state detached --timeout 10000\n control-openhands assert --field recent.0.status --equals 200 -- browser network --filter '/git/changes'\n control-openhands browser screenshot --feature F08.uncommitted-changes --name nested-repository\n control-openhands assert --field count --equals 1 --bug-shows 0 -- browser count 'testid=uncommitted-changes-row' # bug\n ```", "raw": "**Plain parent containing a repository (`F08.uncommitted-changes`).** Expected: a plain workspace holding one dirty repository exposes that repository's changes. Known failure (reproduced 2026-10-16, pinned SDK 1.55.0, [OpenHands/software-agent-sdk#5641](https://github.com/OpenHands/software-agent-sdk/issues/5641)): the changes request returns200 with no changes and Commits reads `No commits yet` although the child repository is dirty. The upstream fix is closed but not included in the current pin. The recipe removes only its own verified empty parent `.git`; never remove a user's repository.\n ```sh known-bug F08.uncommitted-changes nested-repository\n control-openhands browser viewport desktop\n QA_ID=\"$(python3 - <<'PY'\n import json, os, pathlib, subprocess, tempfile\n base = pathlib.Path(os.environ[\"OH_VERIFY_RUN\"]) / \"workspace\"\n base.mkdir(exist_ok=True)\n fixture = pathlib.Path(tempfile.mkdtemp(prefix=\"qa-surface-\", dir=base))\n llm = {\"model\":\"openai/gpt-4o\", \"base_url\":\"http://127.0.0.1:1/v1\", \"api_key\":\"qa-not-real\", \"num_retries\":0, \"timeout\":2}\n def create(workspace, parent=None, tags=None):\n payload = {\"agent\":{\"kind\":\"Agent\", \"llm\":llm, \"tools\":[]}, \"workspace\":{\"working_dir\":str(workspace)}, \"autotitle\":False}\n if parent: payload[\"parent_conversation_id\"] = parent\n if tags: payload[\"tags\"] = tags\n request = fixture / \"request.json\"\n request.write_text(json.dumps(payload))\n result = subprocess.run([\"control-openhands\", \"api\", \"POST\", \"/api/conversations\", \"--write\", \"--data\", \"@\"+str(request), \"--pick\", \"id\"], check=True, capture_output=True, text=True)\n return json.loads(result.stdout)[\"value\"]\n def git(directory, *args):\n subprocess.run([\"git\", \"-C\", str(directory), *args], check=True, capture_output=True)\n import shutil\n parent = fixture / \"parent\"\n child = parent / \"proj\"\n child.mkdir(parents=True)\n git(child, \"init\", \"-q\", \"-b\", \"main\")\n (child / \"README.md\").write_text(\"before\\n\")\n git(child, \"add\", \".\")\n git(child, \"-c\", \"user.name=QA\", \"-c\", \"user.email=qa@example.invalid\", \"commit\", \"-qm\", \"Owned nested fixture\")\n (child / \"README.md\").write_text(\"after\\n\")\n (child / \"notes.txt\").write_text(\"untracked\\n\")\n conversation_id = create(parent)\n empty_git = parent / \".git\"\n if empty_git.exists():\n assert empty_git.is_dir() and not empty_git.is_symlink()\n top = pathlib.Path(subprocess.check_output([\"git\", \"-C\", str(parent), \"rev-parse\", \"--show-toplevel\"], text=True).strip())\n assert top.resolve() == parent.resolve()\n assert not subprocess.check_output([\"git\", \"-C\", str(parent), \"ls-files\"], text=True).strip()\n shutil.rmtree(empty_git)\n print(conversation_id)\n PY\n )\"\n control-openhands browser goto \"/conversations/$QA_ID\"\n control-openhands browser click 'testid=right-panel-toggle'\n control-openhands browser click 'testid=conversation-tab-commits'\n control-openhands browser wait '#conversation-tab-panel main'\n control-openhands browser wait 'testid=commits-tab-status' --state detached --timeout 10000\n control-openhands assert --field recent.0.status --equals 200 -- browser network --filter '/git/changes'\n control-openhands browser screenshot --feature F08.uncommitted-changes --name nested-repository\n control-openhands assert --field count --equals 1 --bug-shows 0 -- browser count 'testid=uncommitted-changes-row' # bug\n ```", "source_line": 152, "expected": [ "the top row reads `-`, `Uncommitted` and a file count; expanding it lists the changed paths." ], "notes": [ "Known issue [#16716](https://github.com/OpenHands/OpenHands/issues/16716) (`F08.uncommitted-changes`): Diff view is regularly out of sync. Live prerequisite: The original simultaneous-agent/shared-repository state or a deterministic stale-diff trigger; the report only suggests that timing.", "Known issue [#16454](https://github.com/OpenHands/OpenHands/issues/16454) (`F08.uncommitted-changes`): Diff list is showing duplicate entries. Live prerequisite: The original Git changes response and update sequence producing duplicate rows; the report provides no trigger.", "Known issue [#15692](https://github.com/OpenHands/OpenHands/issues/15692) (`F08.uncommitted-changes`): Diff still shows after agent has committed and pushed. Live prerequisite: A real agent commit/push terminal observation while the diff remains mounted, plus a disposable Git remote." ], "annotations": [ "Known failure (reproduced 2026-10-16, pinned SDK 1.55.0, [OpenHands/software-agent-sdk#5641](https://github.com/OpenHands/software-agent-sdk/issues/5641)): the changes request returns200 with no changes and Commits reads `No commits yet` although the child repository is dirty. The upstream fix is closed but not included in the current pin. The recipe removes only its own verified empty parent `.git`; never remove a user's repository." ], "states": [ "failure", "issue", "blocked" ] } ] }, { "id": "F09", "title": "Settings shell and navigation", "file": "F09-settings-shell.md", "sha256": "d177599f878ea879eda909be2770766747eed9cdeea5c6650c79518f849bedf5", "overview": "Settings is one shell around eight pages (Agent, LLM, Model Router, Condenser,\nAgent Context, Verification, Application, Secrets). On desktop a pinned left\ncolumn lists the pages, highlights the current one and ends with an \"Agent\nCanvas\" version card and a \"synced from … backend\" note; each page gets a title\nand a grey subline. Below 1024 px (phones and tablets) `/settings` is a hub list\ninstead, at tablet widths (768–1023 px) each subpage has a compact Settings selector, and at phone width every page shows a Back chevron to the hub. Users reach Settings from the sidebar gear,\nthe collapsed rail, the command menu and several deep links (LLM banner, getting\nstarted checklist, composer pickers). The individual pages are other families\n(F10–F16; Secrets is F14).", "entry": "- Sidebar footer gear next to the backend selector (`backend-selector-settings-link`), on desktop or inside the phone drawer (`sidebar-mobile-menu-toggle`, then `sidebar-mobile-drawer >> backend-selector-settings-link`).\n- Collapsed sidebar: the gear icon at the bottom of the rail (`collapsed-settings-link`).\n- Command menu (`Control+k`/`Meta+k` or `command-menu-trigger`): group **Settings** with Settings, Agent settings, LLM profiles, Model Router, Condenser settings, Agent Context, Verification settings, Application settings, Secrets settings.\n- Deep links: Home banner **Set up LLM** (`home-llm-not-configured-action`, only while no LLM key is configured); getting-started checklist **Add LLM API key** and **Customize your agent**; composer LLM pill (`chat-input-llm-profile`) → **LLM Profiles**; composer **+** (`chat-plus-button`) → **Switch agent profile** → **Manage agent profiles**. Application settings also links \"Manage LLM profiles\" (F15's page).\n- Direct URLs: `/settings`, `/settings/agents`, `/settings/llm`, `/settings/meta-llm`, `/settings/condenser`, `/settings/agent-context`, `/settings/verification`, `/settings/app`, `/settings/secrets`, and the legacy `/settings/agent`.\n- Inside Settings: the left nav (`sidebar-settings-/settings/`) on desktop; the hub list (`settings-mobile-hub`) plus the top-bar Back chevron (`sidebar-mobile-back-button`) on phones.", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done), desktop viewport.\n- Run the banner and checklist deep links (`F09.entry-deep-links`, first bullet) **before** configuring a model: the banner only exists while no LLM key is set.\n- The composer picker deep links need a configured profile: `control-openhands llm preset deepseek` (no model call is made).\n- `F09.cloud-links` positive case and `F09.locked-cloud-landing` need a Cloud backend/account (blocked in a local run). `F09.hidden-llm-redirect` is blocked: the frontend hardcodes `hide_llm_settings: false`. The update states depend on the browser reaching `registry.npmjs.org`: with access and a current client you get `F09.update-up-to-date`; `F09.update-available` and the version tile also need an npm `latest` newer than the client (blocked while the client is the latest release); the failure text of `F09.update-modal` needs the registry to be unreachable (no mocks).", "gotchas": "- `browser click` returns the URL *before* client-side navigation and redirects settle (it prints `/` or `/settings`). Always follow a navigating click with `browser wait-url '$'` before asserting.\n- At phone width there are two `backend-selector-settings-link` elements (hidden desktop sidebar plus the drawer); scope with `testid=sidebar-mobile-drawer >> ...` or the click fails in strict mode. The hub's nav links share testids with the desktop nav, so scope them with `testid=settings-mobile-hub >> ...`.\n- The page does not scroll inside the settings `
`; the app shell column does. `browser scroll --by N` only scrolls the element into view and ignores `--by`; hover the content first, then `browser scroll --by N` (mouse wheel at the pointer).\n- Only `/settings/app` is taller than a 1000 px viewport in a fresh run; use it for the pinned-nav check.\n- Visiting `/settings/agents` ticks the checklist item \"Customize your agent\" for good (struck through, still a working link, so the bullet order above is fine), and configuring a model hides the Home banner. Run the banner deep link before `llm preset` on a fresh run.\n- Registry reachability depends on the sandbox and changes between runs. When the browser cannot reach `registry.npmjs.org` (seen as `ERR_CERT_AUTHORITY_INVALID` through a proxy), the version badge and sidebar tile stay hidden and the dialog shows the failure state; that is environment, not product. When it can, a checkout at the latest release shows \"Up to date\" and the failure text cannot be produced. Check `browser network` for the registry request before deciding which update rows are `blocked`.\n- The update dialog's `role=\"dialog\"` wrapper has no accessible name, so `role=dialog[name=...]` does not match it; use `testid=agent-canvas-update-modal`.\n- The `/settings` → `/settings/agents` redirect only happens at ≥1024 px. Below that (phones and tablets) `/settings` is the hub; do not expect a redirect there.\n- From 768 to 1023 px the sidebar rail stays and the phone top bar is gone; the Settings selector above each subpage provides sibling navigation. The rail's gear (`backend-selector-settings-link`) still returns to the hub.\n- The phone hub is indented 32 px while sub-pages start at 16 px (hub padding inside the page padding); that is current layout, not overflow.\n- The \"LLM Profiles\" rename in `useSettingsNavItems` targets a `/settings` nav item that does not exist, so it never applies; the LLM nav item always reads \"LLM\".\n- Known issue [#17508](https://github.com/OpenHands/OpenHands/issues/17508) (`F09.update-available`): After installing a newer package, an older running bundle can keep showing its own version and an update banner. Live prerequisite: Isolated old/new package installs with the old process still serving, or the original stale bundle/cache capture after update.\n- The public `SettingsNavigation` export renders the legacy `SettingsMobileDrawer` with a half-opacity black scrim (`bg-black/50`). Check its modern and fallback compiled CSS as a build contract. The routed phone hub uses `SettingsMobileHub`, so its navigation recipe does not exercise the exported drawer; visual verification of that export needs an embedding host.", "features": [ { "id": "F09.entry-gear", "description": "the sidebar gear opens Settings on the Agent page; the gear is marked active on every settings URL; at phone width the drawer's gear opens the hub and closes the drawer." }, { "id": "F09.entry-collapsed-rail", "description": "with the sidebar collapsed, the rail's Settings icon opens the Agent page." }, { "id": "F09.entry-command-menu", "description": "the command menu's Settings group (Settings, Agent settings, LLM profiles, Model Router, Condenser, Agent Context, Verification, Application, Secrets settings) opens the matching page." }, { "id": "F09.command-menu-coverage", "description": "every page in the settings nav, including Model Router and Agent Context, can be found in the command menu and opens from it." }, { "id": "F09.entry-deep-links", "description": "the Home \"Set up LLM\" banner, the checklist items \"Add LLM API key\" and \"Customize your agent\", the composer LLM picker's \"LLM Profiles\" link and the + menu's \"Manage agent profiles\" link each land on `/settings/llm` or `/settings/agents`." }, { "id": "F09.index-redirect", "description": "on desktop, `/settings` replaces itself with `/settings/agents` (Back skips `/settings`)." }, { "id": "F09.locked-cloud-landing", "description": "on a canvas locked to a Cloud host, `/settings` lands on `/settings/app` instead." }, { "id": "F09.legacy-agent-redirect", "description": "the retired `/settings/agent` URL replaces itself with `/settings/agents`." }, { "id": "F09.hidden-llm-redirect", "description": "with the `hide_llm_settings` feature flag, the LLM link disappears and `/settings/llm` redirects to `/settings/agents`." }, { "id": "F09.desktop-nav", "description": "the desktop nav lists the eight pages in order, exactly one link is `aria-current=\"page\"`, and the nav stays pinned while the page content scrolls." }, { "id": "F09.page-header", "description": "each page shows its nav title as an H2 with a grey subline; Secrets draws its own header instead, and the Agent profile editor hides the shell header until you leave it." }, { "id": "F09.landmarks", "description": "the settings screen exposes a single `main` landmark, on desktop pages and on the phone hub." }, { "id": "F09.phone-hub", "description": "below 1024 px (phones and tablets), `/settings` stays put and shows a \"Settings\" heading, the eight links, the version card and the synced note, with no page header and no horizontal overflow." }, { "id": "F09.hub-resize-redirect", "description": "widening the window to ≥1024 px while the phone hub is open replaces `/settings` with `/settings/agents`; narrowing on a sub-page keeps its URL and adds the Back chevron." }, { "id": "F09.breakpoint-sweep", "description": "at 767, 768, 820, 1023, 1024 and 1440 px, `/settings` is the hub below 1024 px and redirects to the desktop nav from 1024 px up, and the Application page's `main` is at least 340 px wide with no horizontal overflow. Below 1024 px the settings `
` fills the column down to the viewport bottom and scrolls itself, so a popover inside it (the tablet selector) is not clipped on a short page (#18256)." }, { "id": "F09.tablet-selector", "description": "at 768–1023 px, a Settings selector above the subpage opens the same visible section links as the desktop sidebar, including Cloud links when available. Selecting a sibling navigates directly and closes it; Tab reaches links, Escape closes and returns focus, and outside clicks dismiss it. The current section uses a slash-boundary match, so a trailing slash does not hide the selector or its selected link. It is absent on the hub, phones and desktops." }, { "id": "F09.phone-back", "description": "at phone width each settings page shows a Back chevron (label \"Settings\") in the top bar that returns to the hub." }, { "id": "F09.synced-badge", "description": "the nav footer reads \"These settings are synced from Local backend ()\" for the active backend." }, { "id": "F09.cloud-links", "description": "on a Cloud backend only, the nav adds \"Integrations\" and \"All Cloud Settings\" external links; on a local backend neither exists." }, { "id": "F09.update-card", "description": "the \"Agent Canvas / Version x.y.z\" card opens a dialog; its \"Up to date\" / \"Update\" badge appears only once the npm check succeeded." }, { "id": "F09.update-modal", "description": "the dialog shows the version, a \"Check for updates\" button that re-runs the check, and an inline status; a failed check reads \"Couldn't check for updates. Try again later.\"; Close and Escape dismiss it." }, { "id": "F09.update-up-to-date", "description": "when the npm check succeeds and the client is current, the card badge reads \"Up to date\", the dialog status reads \"You're running the latest version.\" with a \"What's new\" link to the GitHub releases page (new tab), and \"Check for updates\" briefly shows \"Checking for updates…\"." }, { "id": "F09.update-available", "description": "when npm has a newer version, the dialog shows \"Version … is available.\", a \"What's new\" link, and npm/Docker command tabs with a copy button." }, { "id": "F09.update-modal-phone", "description": "the update dialog fits a 390 px (and a 320 px) viewport with its title and Close button on screen, and Close dismisses it." }, { "id": "F09.sidebar-version-tile", "description": "the main sidebar shows an \"Agent Canvas\" version tile only when an update is available." }, { "id": "F09.unknown-subpath", "description": "an unknown `/settings/` URL shows the app's \"Page not found\" page inside the app shell (sidebar and a Home link), without the settings shell." } ], "scenarios": [ { "name": "F09--gear-entry-and-index-redirect", "title": "Gear entry and index redirect", "ids": [ "F09.entry-gear", "F09.index-redirect" ], "body": "From `/`, run `control-openhands browser click 'testid=backend-selector-settings-link'` then `control-openhands browser wait-url '/settings/agents$'`. Then `control-openhands browser attr 'testid=sidebar-settings-/settings/agents' aria-current` is `page` and `control-openhands browser attr 'testid=backend-selector-settings-link' data-active` is `true` (it is `false` again on `/`). Run `control-openhands browser back` then `control-openhands browser url`: the URL is `/`, not `/settings`, which proves the redirect replaced history. Direct entry: `control-openhands browser goto /settings` returns `\"url\": \".../settings/agents\"`.", "raw": "**Gear entry and index redirect (`F09.entry-gear`, `F09.index-redirect`).** From `/`, run `control-openhands browser click 'testid=backend-selector-settings-link'` then `control-openhands browser wait-url '/settings/agents$'`. Then `control-openhands browser attr 'testid=sidebar-settings-/settings/agents' aria-current` is `page` and `control-openhands browser attr 'testid=backend-selector-settings-link' data-active` is `true` (it is `false` again on `/`). Run `control-openhands browser back` then `control-openhands browser url`: the URL is `/`, not `/settings`, which proves the redirect replaced history. Direct entry: `control-openhands browser goto /settings` returns `\"url\": \".../settings/agents\"`.", "source_line": 62, "expected": [ "the sidebar gear opens Settings on the Agent page; the gear is marked active on every settings URL; at phone width the drawer's gear opens the hub and closes the drawer.", "on desktop, `/settings` replaces itself with `/settings/agents` (Back skips `/settings`)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--collapsed-rail", "title": "Collapsed rail", "ids": [ "F09.entry-collapsed-rail" ], "body": "From `/` (`control-openhands browser goto /`; the previous bullet ends on `/settings/agents`, where the URL wait would pass at once) run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, `control-openhands browser click 'testid=collapsed-settings-link'`, `control-openhands browser wait-url '/settings/agents$'` and `control-openhands browser screenshot --feature F09.entry-collapsed-rail --name rail`. The screenshot shows the icon rail, the settings nav with Agent highlighted. Restore with `control-openhands browser click 'testid=sidebar-collapse-toggle'` (the label flips back to \"Collapse sidebar\").", "raw": "**Collapsed rail (`F09.entry-collapsed-rail`).** From `/` (`control-openhands browser goto /`; the previous bullet ends on `/settings/agents`, where the URL wait would pass at once) run `control-openhands browser click 'testid=sidebar-collapse-toggle'`, `control-openhands browser click 'testid=collapsed-settings-link'`, `control-openhands browser wait-url '/settings/agents$'` and `control-openhands browser screenshot --feature F09.entry-collapsed-rail --name rail`. The screenshot shows the icon rail, the settings nav with Agent highlighted. Restore with `control-openhands browser click 'testid=sidebar-collapse-toggle'` (the label flips back to \"Collapse sidebar\").", "source_line": 63, "expected": [ "with the sidebar collapsed, the rail's Settings icon opens the Agent page." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--command-menu", "title": "Command menu", "ids": [ "F09.entry-command-menu" ], "body": "First rest the pointer off the list with `control-openhands browser hover 'testid=command-menu-trigger'`: a hover can move the menu's selection, and Enter runs the selected option (F02 Gotchas). For each title run `control-openhands browser goto /`, `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' 'LLM profiles'`, `control-openhands browser press Enter`, `control-openhands browser wait-url '/settings/llm$'`. Expected targets: `Settings` → `/settings/agents`, `Agent settings` → `/settings/agents`, `LLM profiles` → `/settings/llm`, `Model Router` → `/settings/meta-llm`, `Condenser settings` → `/settings/condenser`, `Agent Context` → `/settings/agent-context`, `Verification settings` → `/settings/verification`, `Application settings` → `/settings/app`, `Secrets settings` → `/settings/secrets`. `control-openhands browser snapshot 'testid=command-menu >> role=listbox'` after typing `LLM profiles` shows two options under the \"Settings\" group text: `LLM profiles` `[selected]` and `Model Router`, whose subline mentions LLM profiles. Enter runs the selected one.", "raw": "**Command menu (`F09.entry-command-menu`).** First rest the pointer off the list with `control-openhands browser hover 'testid=command-menu-trigger'`: a hover can move the menu's selection, and Enter runs the selected option (F02 Gotchas). For each title run `control-openhands browser goto /`, `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' 'LLM profiles'`, `control-openhands browser press Enter`, `control-openhands browser wait-url '/settings/llm$'`. Expected targets: `Settings` → `/settings/agents`, `Agent settings` → `/settings/agents`, `LLM profiles` → `/settings/llm`, `Model Router` → `/settings/meta-llm`, `Condenser settings` → `/settings/condenser`, `Agent Context` → `/settings/agent-context`, `Verification settings` → `/settings/verification`, `Application settings` → `/settings/app`, `Secrets settings` → `/settings/secrets`. `control-openhands browser snapshot 'testid=command-menu >> role=listbox'` after typing `LLM profiles` shows two options under the \"Settings\" group text: `LLM profiles` `[selected]` and `Model Router`, whose subline mentions LLM profiles. Enter runs the selected one.", "source_line": 64, "expected": [ "the command menu's Settings group (Settings, Agent settings, LLM profiles, Model Router, Condenser, Agent Context, Verification, Application, Secrets settings) opens the matching page." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--command-menu-coverage", "title": "Command menu coverage", "ids": [ "F09.command-menu-coverage" ], "body": "Run `control-openhands browser goto /`, `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' 'Model Router'` and `control-openhands browser snapshot 'testid=command-menu >> role=listbox'`. It shows one option under the \"Settings\" group text: `Model Router Meta-profiles that route each task to the right LLM profile. Go` `[selected]`. Run `control-openhands browser press Enter` and `control-openhands browser wait-url '/settings/meta-llm$'`; `control-openhands browser count 'testid=command-menu'` is `0`. Repeat from `/` with `Agent Context`: the option reads `Agent Context Let the agent keep notes and recall them in new conversations. Go` and the URL becomes `/settings/agent-context`. With the previous bullet, every page in the settings nav has opened from the menu.", "raw": "**Command menu coverage (`F09.command-menu-coverage`).** Run `control-openhands browser goto /`, `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' 'Model Router'` and `control-openhands browser snapshot 'testid=command-menu >> role=listbox'`. It shows one option under the \"Settings\" group text: `Model Router Meta-profiles that route each task to the right LLM profile. Go` `[selected]`. Run `control-openhands browser press Enter` and `control-openhands browser wait-url '/settings/meta-llm$'`; `control-openhands browser count 'testid=command-menu'` is `0`. Repeat from `/` with `Agent Context`: the option reads `Agent Context Let the agent keep notes and recall them in new conversations. Go` and the URL becomes `/settings/agent-context`. With the previous bullet, every page in the settings nav has opened from the menu.", "source_line": 65, "expected": [ "every page in the settings nav, including Model Router and Agent Context, can be found in the command menu and opens from it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--deep-links-without-a-model", "title": "Deep links without a model", "ids": [ "F09.entry-deep-links" ], "body": "On `/` with no LLM key (`control-openhands browser goto /`): `control-openhands browser click 'testid=home-llm-not-configured-action'` then `control-openhands browser wait-url '/settings/'` → `/settings/llm`. `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-configure-llm'`, `control-openhands browser wait-url '/settings/'` → `/settings/llm`. `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-customize-agent'`, `control-openhands browser wait-url '/settings/'` → `/settings/agents`.", "raw": "**Deep links without a model (`F09.entry-deep-links`).** On `/` with no LLM key (`control-openhands browser goto /`): `control-openhands browser click 'testid=home-llm-not-configured-action'` then `control-openhands browser wait-url '/settings/'` → `/settings/llm`. `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-configure-llm'`, `control-openhands browser wait-url '/settings/'` → `/settings/llm`. `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-onboarding-checklist-item-customize-agent'`, `control-openhands browser wait-url '/settings/'` → `/settings/agents`.", "source_line": 66, "expected": [ "the Home \"Set up LLM\" banner, the checklist items \"Add LLM API key\" and \"Customize your agent\", the composer LLM picker's \"LLM Profiles\" link and the + menu's \"Manage agent profiles\" link each land on `/settings/llm` or `/settings/agents`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--deep-links-from-the-composer", "title": "Deep links from the composer", "ids": [ "F09.entry-deep-links" ], "body": "After `control-openhands llm preset deepseek`, run `control-openhands browser goto /`, `control-openhands browser click 'testid=chat-input-llm-profile'`, `control-openhands browser click 'testid=chat-input-llm-profile-popover >> role=link[name=\"LLM Profiles\"]'`, `control-openhands browser wait-url '/settings/llm$'`; `control-openhands browser count 'testid=chat-input-llm-profile-popover'` is `0`. Then `control-openhands browser goto /`, `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser hover 'testid=switch-agent-profile-button'`, `control-openhands browser click 'testid=agent-profile-submenu >> role=link[name=\"Manage agent profiles\"]'`, `control-openhands browser wait-url '/settings/agents$'`; `control-openhands browser count 'testid=tools-context-menu'` is `0`.", "raw": "**Deep links from the composer (`F09.entry-deep-links`).** After `control-openhands llm preset deepseek`, run `control-openhands browser goto /`, `control-openhands browser click 'testid=chat-input-llm-profile'`, `control-openhands browser click 'testid=chat-input-llm-profile-popover >> role=link[name=\"LLM Profiles\"]'`, `control-openhands browser wait-url '/settings/llm$'`; `control-openhands browser count 'testid=chat-input-llm-profile-popover'` is `0`. Then `control-openhands browser goto /`, `control-openhands browser click 'testid=chat-plus-button'`, `control-openhands browser hover 'testid=switch-agent-profile-button'`, `control-openhands browser click 'testid=agent-profile-submenu >> role=link[name=\"Manage agent profiles\"]'`, `control-openhands browser wait-url '/settings/agents$'`; `control-openhands browser count 'testid=tools-context-menu'` is `0`.", "source_line": 67, "expected": [ "the Home \"Set up LLM\" banner, the checklist items \"Add LLM API key\" and \"Customize your agent\", the composer LLM picker's \"LLM Profiles\" link and the + menu's \"Manage agent profiles\" link each land on `/settings/llm` or `/settings/agents`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--legacy-url", "title": "Legacy URL", "ids": [ "F09.legacy-agent-redirect" ], "body": "Run `control-openhands browser goto /`, then `control-openhands browser goto /settings/agent`; the result already reads `.../settings/agents`. `control-openhands browser back` returns to `/`.", "raw": "**Legacy URL (`F09.legacy-agent-redirect`).** Run `control-openhands browser goto /`, then `control-openhands browser goto /settings/agent`; the result already reads `.../settings/agents`. `control-openhands browser back` returns to `/`.", "source_line": 68, "expected": [ "the retired `/settings/agent` URL replaces itself with `/settings/agents`." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--desktop-nav", "title": "Desktop nav", "ids": [ "F09.desktop-nav" ], "body": "On any settings page (`control-openhands browser goto /settings/agents`; Legacy URL ends on `/`) run `control-openhands browser snapshot 'testid=settings-navbar-desktop'`: links Agent, LLM, Model Router, Condenser, Agent Context, Verification, Application, Secrets (with `/url`s), then `button \"Agent Canvas Version …\"` and the synced text. Click each one, e.g. `control-openhands browser click 'testid=sidebar-settings-/settings/meta-llm'`, `control-openhands browser wait-url '/settings/meta-llm$'`, `control-openhands browser attr 'testid=sidebar-settings-/settings/meta-llm' aria-current` (`page`), and `control-openhands browser count 'testid=settings-navbar-desktop >> [aria-current=\"page\"]'` (`1`). Pinned: `control-openhands browser goto /settings/app` (the only page taller than the viewport), `control-openhands browser bbox 'testid=settings-navbar-desktop'` (`y` 32), `control-openhands browser hover 'testid=settings-page-subtitle'`, `control-openhands browser scroll --by 1000`, then `control-openhands browser bbox 'testid=settings-page-subtitle'` (negative `y`, content scrolled) and `control-openhands browser bbox 'testid=settings-navbar-desktop'` (still `y` 32); `control-openhands browser screenshot --feature F09.desktop-nav --name scrolled` shows Git Settings on the right and the full nav on the left.", "raw": "**Desktop nav (`F09.desktop-nav`).** On any settings page (`control-openhands browser goto /settings/agents`; Legacy URL ends on `/`) run `control-openhands browser snapshot 'testid=settings-navbar-desktop'`: links Agent, LLM, Model Router, Condenser, Agent Context, Verification, Application, Secrets (with `/url`s), then `button \"Agent Canvas Version …\"` and the synced text. Click each one, e.g. `control-openhands browser click 'testid=sidebar-settings-/settings/meta-llm'`, `control-openhands browser wait-url '/settings/meta-llm$'`, `control-openhands browser attr 'testid=sidebar-settings-/settings/meta-llm' aria-current` (`page`), and `control-openhands browser count 'testid=settings-navbar-desktop >> [aria-current=\"page\"]'` (`1`). Pinned: `control-openhands browser goto /settings/app` (the only page taller than the viewport), `control-openhands browser bbox 'testid=settings-navbar-desktop'` (`y` 32), `control-openhands browser hover 'testid=settings-page-subtitle'`, `control-openhands browser scroll --by 1000`, then `control-openhands browser bbox 'testid=settings-page-subtitle'` (negative `y`, content scrolled) and `control-openhands browser bbox 'testid=settings-navbar-desktop'` (still `y` 32); `control-openhands browser screenshot --feature F09.desktop-nav --name scrolled` shows Git Settings on the right and the full nav on the left.", "source_line": 69, "expected": [ "the desktop nav lists the eight pages in order, exactly one link is `aria-current=\"page\"`, and the nav stays pinned while the page content scrolls." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--page-header", "title": "Page header", "ids": [ "F09.page-header" ], "body": "After each nav click, `control-openhands browser text 'testid=settings-screen >> header'` reads title + subline: Agent \"Create and manage reusable agent setups.\", LLM \"Model, API key, and options for the agent.\", Model Router \"Meta-profiles that route each task to the right LLM profile.\", Condenser \"Summarize long chats to stay within context limits.\", Agent Context \"Let the agent keep notes and recall them in new conversations.\", Verification \"Confirmation prompts and security checks on actions.\", Application \"Language, theme, notifications, and Git identity.\". On Secrets, `control-openhands browser count 'testid=settings-screen >> header >> h2'` is `0` (the page draws its own \"Secrets\" heading). Editor: on `/settings/agents` (`control-openhands browser goto /settings/agents`) run `control-openhands browser click 'testid=add-agent-profile'`; `control-openhands browser count 'testid=settings-page-subtitle'` is `0` and `control-openhands browser text 'testid=settings-screen >> h2'` is `Add agent profile`. `control-openhands browser click 'testid=back-to-agent-profiles'` brings the subtitle back (count `1`); opening the editor again and clicking `testid=sidebar-settings-/settings/condenser` also shows the Condenser subline.", "raw": "**Page header (`F09.page-header`).** After each nav click, `control-openhands browser text 'testid=settings-screen >> header'` reads title + subline: Agent \"Create and manage reusable agent setups.\", LLM \"Model, API key, and options for the agent.\", Model Router \"Meta-profiles that route each task to the right LLM profile.\", Condenser \"Summarize long chats to stay within context limits.\", Agent Context \"Let the agent keep notes and recall them in new conversations.\", Verification \"Confirmation prompts and security checks on actions.\", Application \"Language, theme, notifications, and Git identity.\". On Secrets, `control-openhands browser count 'testid=settings-screen >> header >> h2'` is `0` (the page draws its own \"Secrets\" heading). Editor: on `/settings/agents` (`control-openhands browser goto /settings/agents`) run `control-openhands browser click 'testid=add-agent-profile'`; `control-openhands browser count 'testid=settings-page-subtitle'` is `0` and `control-openhands browser text 'testid=settings-screen >> h2'` is `Add agent profile`. `control-openhands browser click 'testid=back-to-agent-profiles'` brings the subtitle back (count `1`); opening the editor again and clicking `testid=sidebar-settings-/settings/condenser` also shows the Condenser subline.", "source_line": 70, "expected": [ "each page shows its nav title as an H2 with a grey subline; Secrets draws its own header instead, and the Agent profile editor hides the shell header until you leave it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--landmarks", "title": "Landmarks", "ids": [ "F09.landmarks" ], "body": "Run `control-openhands browser goto /settings/agents`, `control-openhands browser count 'role=main'` (`1`) and `control-openhands browser eval \"document.querySelector('[data-testid=settings-screen]').tagName\"` (`DIV`: the shell wrapper is not a landmark; the layout's own `
` is). `control-openhands browser goto /settings/app` and the same `count` is `1`. At phone width (`control-openhands browser viewport phone`, `control-openhands browser goto /settings`) the hub also counts `1`; return with `control-openhands browser viewport desktop`.", "raw": "**Landmarks (`F09.landmarks`).** Run `control-openhands browser goto /settings/agents`, `control-openhands browser count 'role=main'` (`1`) and `control-openhands browser eval \"document.querySelector('[data-testid=settings-screen]').tagName\"` (`DIV`: the shell wrapper is not a landmark; the layout's own `
` is). `control-openhands browser goto /settings/app` and the same `count` is `1`. At phone width (`control-openhands browser viewport phone`, `control-openhands browser goto /settings`) the hub also counts `1`; return with `control-openhands browser viewport desktop`.", "source_line": 71, "expected": [ "the settings screen exposes a single `main` landmark, on desktop pages and on the phone hub." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--phone-hub-and-back", "title": "Phone hub and Back", "ids": [ "F09.phone-hub", "F09.phone-back" ], "body": "Run `control-openhands browser viewport phone`, `control-openhands browser goto /settings` (URL stays `/settings`), `control-openhands browser snapshot 'testid=settings-mobile-hub'` (heading \"Settings\", the eight links, the version button, the synced text), `control-openhands browser count 'testid=settings-page-subtitle'` (`0`), `control-openhands browser bbox 'testid=settings-mobile-hub'` (`pageHorizontalOverflow` `false`) and `control-openhands browser screenshot --feature F09.phone-hub --name hub`. Then `control-openhands browser click 'testid=settings-mobile-hub >> testid=sidebar-settings-/settings/condenser'`, `control-openhands browser wait-url '/settings/condenser$'`, `control-openhands browser attr 'testid=sidebar-mobile-back-button' aria-label` (`Settings`), `control-openhands browser click 'testid=sidebar-mobile-back-button'`, `control-openhands browser wait-url '/settings$'` and `control-openhands browser visible 'testid=settings-mobile-hub'` (`true`). Drawer entry: `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-mobile-menu-toggle'`, `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=backend-selector-settings-link'`, `control-openhands browser wait-url '/settings$'` and `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state detached --timeout 5000` pass (the drawer closes as it animates out; an immediate `visible` can still read `true`). Return with `control-openhands browser viewport desktop`.", "raw": "**Phone hub and Back (`F09.phone-hub`, `F09.phone-back`).** Run `control-openhands browser viewport phone`, `control-openhands browser goto /settings` (URL stays `/settings`), `control-openhands browser snapshot 'testid=settings-mobile-hub'` (heading \"Settings\", the eight links, the version button, the synced text), `control-openhands browser count 'testid=settings-page-subtitle'` (`0`), `control-openhands browser bbox 'testid=settings-mobile-hub'` (`pageHorizontalOverflow` `false`) and `control-openhands browser screenshot --feature F09.phone-hub --name hub`. Then `control-openhands browser click 'testid=settings-mobile-hub >> testid=sidebar-settings-/settings/condenser'`, `control-openhands browser wait-url '/settings/condenser$'`, `control-openhands browser attr 'testid=sidebar-mobile-back-button' aria-label` (`Settings`), `control-openhands browser click 'testid=sidebar-mobile-back-button'`, `control-openhands browser wait-url '/settings$'` and `control-openhands browser visible 'testid=settings-mobile-hub'` (`true`). Drawer entry: `control-openhands browser goto /`, `control-openhands browser click 'testid=sidebar-mobile-menu-toggle'`, `control-openhands browser click 'testid=sidebar-mobile-drawer >> testid=backend-selector-settings-link'`, `control-openhands browser wait-url '/settings$'` and `control-openhands browser wait 'testid=sidebar-mobile-drawer' --state detached --timeout 5000` pass (the drawer closes as it animates out; an immediate `visible` can still read `true`). Return with `control-openhands browser viewport desktop`.", "source_line": 72, "expected": [ "below 1024 px (phones and tablets), `/settings` stays put and shows a \"Settings\" heading, the eight links, the version card and the synced note, with no page header and no horizontal overflow.", "at phone width each settings page shows a Back chevron (label \"Settings\") in the top bar that returns to the hub." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--resize-from-the-hub", "title": "Resize from the hub", "ids": [ "F09.hub-resize-redirect" ], "body": "Run `control-openhands browser goto /`, `control-openhands browser viewport phone`, `control-openhands browser goto /settings` and `control-openhands browser visible 'testid=settings-mobile-hub'` (`true`). Then `control-openhands browser viewport desktop` and `control-openhands browser wait-url '/settings/agents$'`: the hub is gone (`count 'testid=settings-mobile-hub'` is `0`) and `control-openhands browser back` lands on `/` (the redirect replaced `/settings`). Reverse: `control-openhands browser goto /settings/condenser`, `control-openhands browser viewport phone`, `control-openhands browser url` (still `/settings/condenser`) and `control-openhands browser visible 'testid=sidebar-mobile-back-button'` (`true`). Tablets get the hub too, and the boundary is 1024 px: run `control-openhands browser viewport tablet` (820 px), `control-openhands browser goto /settings` (URL stays `/settings`) and `control-openhands browser visible 'testid=settings-mobile-hub'` (`true`); then `control-openhands browser viewport 1023x900`, `control-openhands browser goto /settings` and the same `visible` (`true`); then `control-openhands browser viewport 1024x900` and `control-openhands browser wait-url '/settings/agents$'` (`browser visible 'testid=settings-navbar-desktop'` is `true`). Return with `control-openhands browser viewport desktop`.", "raw": "**Resize from the hub (`F09.hub-resize-redirect`).** Run `control-openhands browser goto /`, `control-openhands browser viewport phone`, `control-openhands browser goto /settings` and `control-openhands browser visible 'testid=settings-mobile-hub'` (`true`). Then `control-openhands browser viewport desktop` and `control-openhands browser wait-url '/settings/agents$'`: the hub is gone (`count 'testid=settings-mobile-hub'` is `0`) and `control-openhands browser back` lands on `/` (the redirect replaced `/settings`). Reverse: `control-openhands browser goto /settings/condenser`, `control-openhands browser viewport phone`, `control-openhands browser url` (still `/settings/condenser`) and `control-openhands browser visible 'testid=sidebar-mobile-back-button'` (`true`). Tablets get the hub too, and the boundary is 1024 px: run `control-openhands browser viewport tablet` (820 px), `control-openhands browser goto /settings` (URL stays `/settings`) and `control-openhands browser visible 'testid=settings-mobile-hub'` (`true`); then `control-openhands browser viewport 1023x900`, `control-openhands browser goto /settings` and the same `visible` (`true`); then `control-openhands browser viewport 1024x900` and `control-openhands browser wait-url '/settings/agents$'` (`browser visible 'testid=settings-navbar-desktop'` is `true`). Return with `control-openhands browser viewport desktop`.", "source_line": 73, "expected": [ "widening the window to ≥1024 px while the phone hub is open replaces `/settings` with `/settings/agents`; narrowing on a sub-page keeps its URL and adds the Back chevron." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--widths-around-the-breakpoint", "title": "Widths around the breakpoint", "ids": [ "F09.breakpoint-sweep" ], "body": "For each width `W` in `767`, `768`, `820`, `1023`, `1024`, `1440` run `control-openhands browser viewport Wx1024` and `control-openhands browser goto /settings`. Below 1024 px the URL stays `/settings`, `control-openhands browser visible 'testid=settings-mobile-hub'` is `true` and `control-openhands browser bbox 'testid=settings-mobile-hub'` has `pageHorizontalOverflow` `false` (725, 426, 478 and 681 px wide); from 1024 px up `control-openhands browser wait-url '/settings/agents$'` passes and `control-openhands browser visible 'testid=settings-navbar-desktop'` is `true`. Then `control-openhands browser goto /settings/app` and `control-openhands browser bbox 'role=main'`: `width` is at least 340 (767, 468, 520, 723, 424 and 840 today: full width at 767 px, `W − 300` beside the rail up to 1023 px, `W − 600` beside the rail and the settings nav from 1024 px), `scrollWidth` equals `clientWidth` and `pageHorizontalOverflow` is `false` (`insideViewport` is `false` only because the page is taller than 1024 px). Take `control-openhands browser screenshot --feature F09.breakpoint-sweep --name app-767` and `--name app-1024`. Height: at `390x844`, `767x1024`, `820x1180` and `1023x900`, on `/settings/secrets` (a short page) `control-openhands browser eval \"Math.round(document.querySelector('[data-testid=settings-screen] main').getBoundingClientRect().bottom)+'/'+innerHeight\"` reads the viewport height twice (`1180/1180` at 820x1180), and on `/settings/app` `control-openhands browser eval \"(()=>{const m=document.querySelector('[data-testid=settings-screen] main');return m.scrollHeight>m.clientHeight})()\"` is `true` (the `
` scrolls, not an outer column). At 1440x1000 the desktop layout is unchanged (the Secrets `
` ends at its content). Finish with `control-openhands browser viewport desktop`.", "raw": "**Widths around the breakpoint (`F09.breakpoint-sweep`).** For each width `W` in `767`, `768`, `820`, `1023`, `1024`, `1440` run `control-openhands browser viewport Wx1024` and `control-openhands browser goto /settings`. Below 1024 px the URL stays `/settings`, `control-openhands browser visible 'testid=settings-mobile-hub'` is `true` and `control-openhands browser bbox 'testid=settings-mobile-hub'` has `pageHorizontalOverflow` `false` (725, 426, 478 and 681 px wide); from 1024 px up `control-openhands browser wait-url '/settings/agents$'` passes and `control-openhands browser visible 'testid=settings-navbar-desktop'` is `true`. Then `control-openhands browser goto /settings/app` and `control-openhands browser bbox 'role=main'`: `width` is at least 340 (767, 468, 520, 723, 424 and 840 today: full width at 767 px, `W − 300` beside the rail up to 1023 px, `W − 600` beside the rail and the settings nav from 1024 px), `scrollWidth` equals `clientWidth` and `pageHorizontalOverflow` is `false` (`insideViewport` is `false` only because the page is taller than 1024 px). Take `control-openhands browser screenshot --feature F09.breakpoint-sweep --name app-767` and `--name app-1024`. Height: at `390x844`, `767x1024`, `820x1180` and `1023x900`, on `/settings/secrets` (a short page) `control-openhands browser eval \"Math.round(document.querySelector('[data-testid=settings-screen] main').getBoundingClientRect().bottom)+'/'+innerHeight\"` reads the viewport height twice (`1180/1180` at 820x1180), and on `/settings/app` `control-openhands browser eval \"(()=>{const m=document.querySelector('[data-testid=settings-screen] main');return m.scrollHeight>m.clientHeight})()\"` is `true` (the `
` scrolls, not an outer column). At 1440x1000 the desktop layout is unchanged (the Secrets `
` ends at its content). Finish with `control-openhands browser viewport desktop`.", "source_line": 74, "expected": [ "at 767, 768, 820, 1023, 1024 and 1440 px, `/settings` is the hub below 1024 px and redirects to the desktop nav from 1024 px up, and the Application page's `main` is at least 340 px wide with no horizontal overflow. Below 1024 px the settings `
` fills the column down to the viewport bottom and scrolls itself, so a popover inside it (the tablet selector) is not clipped on a short page (#18256)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--tablet-selector", "title": "Tablet selector", "ids": [ "F09.tablet-selector" ], "body": "Run `control-openhands browser viewport tablet`, `control-openhands browser goto /settings/app`, then `control-openhands browser click 'testid=settings-compact-navigation >> role=button'`. `control-openhands browser snapshot 'testid=settings-compact-navigation >> role=navigation'` lists every visible settings destination with Application current. Select `control-openhands browser click 'testid=settings-compact-navigation >> testid=sidebar-settings-/settings/secrets' --expect-url '/settings/secrets$'`; the selector now shows Secrets and its button has `aria-expanded=false`. Open again, `browser press Tab`, `browser press Escape`: `browser eval \"document.activeElement.getAttribute('aria-expanded')\"` is `false` (focus returned to the trigger). Capture `browser screenshot --feature F09.tablet-selector --name settings`. Repeat at 768 and 1023 px; at 767 and 1024 px `browser count 'testid=settings-compact-navigation'` is `0`. On `/settings` below 1024 px the hub remains the list. At tablet width, visit `/settings/llm/`: the trigger reads `Settings: LLM`, the page heading is LLM, and opening the selector marks LLM current. A nested URL such as `/settings/llm/example` is not a registered page and shows the global Not Found screen (F09.unknown-subpath).", "raw": "**Tablet selector (`F09.tablet-selector`).** Run `control-openhands browser viewport tablet`, `control-openhands browser goto /settings/app`, then `control-openhands browser click 'testid=settings-compact-navigation >> role=button'`. `control-openhands browser snapshot 'testid=settings-compact-navigation >> role=navigation'` lists every visible settings destination with Application current. Select `control-openhands browser click 'testid=settings-compact-navigation >> testid=sidebar-settings-/settings/secrets' --expect-url '/settings/secrets$'`; the selector now shows Secrets and its button has `aria-expanded=false`. Open again, `browser press Tab`, `browser press Escape`: `browser eval \"document.activeElement.getAttribute('aria-expanded')\"` is `false` (focus returned to the trigger). Capture `browser screenshot --feature F09.tablet-selector --name settings`. Repeat at 768 and 1023 px; at 767 and 1024 px `browser count 'testid=settings-compact-navigation'` is `0`. On `/settings` below 1024 px the hub remains the list. At tablet width, visit `/settings/llm/`: the trigger reads `Settings: LLM`, the page heading is LLM, and opening the selector marks LLM current. A nested URL such as `/settings/llm/example` is not a registered page and shows the global Not Found screen (F09.unknown-subpath).", "source_line": 75, "expected": [ "at 768–1023 px, a Settings selector above the subpage opens the same visible section links as the desktop sidebar, including Cloud links when available. Selecting a sibling navigates directly and closes it; Tab reaches links, Escape closes and returns focus, and outside clicks dismiss it. The current section uses a slash-boundary match, so a trailing slash does not hide the selector or its selected link. It is absent on the hub, phones and desktops." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--synced-badge", "title": "Synced badge", "ids": [ "F09.synced-badge" ], "body": "Run `control-openhands browser text 'testid=backend-synced-settings-badge'` and `control-openhands browser eval 'location.origin'`. The text is `These settings are synced from Local backend ()` with the same origin.", "raw": "**Synced badge (`F09.synced-badge`).** Run `control-openhands browser text 'testid=backend-synced-settings-badge'` and `control-openhands browser eval 'location.origin'`. The text is `These settings are synced from Local backend ()` with the same origin.", "source_line": 76, "expected": [ "the nav footer reads \"These settings are synced from Local backend ()\" for the active backend." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--cloud-links", "title": "Cloud links", "ids": [ "F09.cloud-links" ], "body": "On a local backend `control-openhands browser count 'testid=settings-integrations-link'` and `control-openhands browser count 'testid=settings-cloud-link'` are both `0`. Positive case (blocked without a Cloud account): with a Cloud backend active both exist; \"All Cloud Settings\" points at `/settings?org=` and opens in a new tab unless the canvas is locked to Cloud.", "raw": "**Cloud links (`F09.cloud-links`).** On a local backend `control-openhands browser count 'testid=settings-integrations-link'` and `control-openhands browser count 'testid=settings-cloud-link'` are both `0`. Positive case (blocked without a Cloud account): with a Cloud backend active both exist; \"All Cloud Settings\" points at `/settings?org=` and opens in a new tab unless the canvas is locked to Cloud.", "source_line": 77, "expected": [ "on a Cloud backend only, the nav adds \"Integrations\" and \"All Cloud Settings\" external links; on a local backend neither exists." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F09--update-card-and-dialog", "title": "Update card and dialog", "ids": [ "F09.update-card", "F09.update-modal", "F09.update-up-to-date" ], "body": "On a settings page run `control-openhands browser text 'testid=agent-canvas-update-toggle'` (`Agent Canvas`, `Version`, the client version, plus the badge text when present), `control-openhands browser count 'testid=agent-canvas-update-badge'` (`0` while the npm check fails; `1` with \"Up to date\" or \"Update\" when it succeeds), then `control-openhands browser click 'testid=agent-canvas-update-toggle'`, `control-openhands browser wait 'testid=agent-canvas-update-modal'` and `control-openhands browser text 'testid=agent-canvas-update-status'`. With registry access and a current client the status is `You're running the latest version. What's new`, and `control-openhands browser attr 'testid=agent-canvas-update-release-notes' href` is `https://github.com/OpenHands/OpenHands/releases` (`target` `_blank`). Without registry access the status is `Couldn't check for updates. Try again later.` Re-check: run `control-openhands browser network --clear`, then `control-openhands browser click 'testid=agent-canvas-update-check-button' --observe 'testid=agent-canvas-update-modal' --observe-ms 2000` (the observed states include `Checking for updates…` before the final status), then `control-openhands browser network`: exactly one new `GET https://registry.npmjs.org/@openhands/agent-canvas/latest` proves the button re-checked (`browser errors` does not list successful requests). `control-openhands browser enabled 'testid=agent-canvas-update-check-button'` is `true` again. Close with `control-openhands browser click 'testid=close-agent-canvas-update-modal'` (modal count `0`); reopen and `control-openhands browser press Escape` also closes it.", "raw": "**Update card and dialog (`F09.update-card`, `F09.update-modal`, `F09.update-up-to-date`).** On a settings page run `control-openhands browser text 'testid=agent-canvas-update-toggle'` (`Agent Canvas`, `Version`, the client version, plus the badge text when present), `control-openhands browser count 'testid=agent-canvas-update-badge'` (`0` while the npm check fails; `1` with \"Up to date\" or \"Update\" when it succeeds), then `control-openhands browser click 'testid=agent-canvas-update-toggle'`, `control-openhands browser wait 'testid=agent-canvas-update-modal'` and `control-openhands browser text 'testid=agent-canvas-update-status'`. With registry access and a current client the status is `You're running the latest version. What's new`, and `control-openhands browser attr 'testid=agent-canvas-update-release-notes' href` is `https://github.com/OpenHands/OpenHands/releases` (`target` `_blank`). Without registry access the status is `Couldn't check for updates. Try again later.` Re-check: run `control-openhands browser network --clear`, then `control-openhands browser click 'testid=agent-canvas-update-check-button' --observe 'testid=agent-canvas-update-modal' --observe-ms 2000` (the observed states include `Checking for updates…` before the final status), then `control-openhands browser network`: exactly one new `GET https://registry.npmjs.org/@openhands/agent-canvas/latest` proves the button re-checked (`browser errors` does not list successful requests). `control-openhands browser enabled 'testid=agent-canvas-update-check-button'` is `true` again. Close with `control-openhands browser click 'testid=close-agent-canvas-update-modal'` (modal count `0`); reopen and `control-openhands browser press Escape` also closes it.", "source_line": 78, "expected": [ "the \"Agent Canvas / Version x.y.z\" card opens a dialog; its \"Up to date\" / \"Update\" badge appears only once the npm check succeeded.", "the dialog shows the version, a \"Check for updates\" button that re-runs the check, and an inline status; a failed check reads \"Couldn't check for updates. Try again later.\"; Close and Escape dismiss it.", "when the npm check succeeds and the client is current, the card badge reads \"Up to date\", the dialog status reads \"You're running the latest version.\" with a \"What's new\" link to the GitHub releases page (new tab), and \"Check for updates\" briefly shows \"Checking for updates…\"." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--update-available", "title": "Update available", "ids": [ "F09.update-available" ], "body": "Blocked unless the browser reaches the registry and npm `latest` is newer than the client (a checkout at the latest release shows `F09.update-up-to-date` instead). With both, and a newer npm `latest` than the client, the status shows `Version is available.` with `testid=agent-canvas-update-release-notes` (\"What's new\"), the npm tab `testid=agent-canvas-update-command-npm` reads `npm install -g @openhands/agent-canvas@latest`, the Docker tab reads `docker pull ghcr.io/openhands/agent-canvas:latest`, and `testid=copy-to-clipboard` turns into a check mark for 2 s.", "raw": "**Update available (`F09.update-available`).** Blocked unless the browser reaches the registry and npm `latest` is newer than the client (a checkout at the latest release shows `F09.update-up-to-date` instead). With both, and a newer npm `latest` than the client, the status shows `Version is available.` with `testid=agent-canvas-update-release-notes` (\"What's new\"), the npm tab `testid=agent-canvas-update-command-npm` reads `npm install -g @openhands/agent-canvas@latest`, the Docker tab reads `docker pull ghcr.io/openhands/agent-canvas:latest`, and `testid=copy-to-clipboard` turns into a check mark for 2 s.", "source_line": 79, "expected": [ "when npm has a newer version, the dialog shows \"Version … is available.\", a \"What's new\" link, and npm/Docker command tabs with a copy button." ], "notes": [ "Known issue [#17508](https://github.com/OpenHands/OpenHands/issues/17508) (`F09.update-available`): After installing a newer package, an older running bundle can keep showing its own version and an update banner. Live prerequisite: Isolated old/new package installs with the old process still serving, or the original stale bundle/cache capture after update." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F09--update-dialog-at-phone-width", "title": "Update dialog at phone width", "ids": [ "F09.update-modal-phone" ], "body": "Run `control-openhands browser viewport phone`, `control-openhands browser goto /settings`, `control-openhands browser click 'testid=settings-mobile-hub >> testid=agent-canvas-update-toggle'`, `control-openhands browser wait 'testid=agent-canvas-update-modal'`, `control-openhands browser bbox 'testid=agent-canvas-update-modal'` and `control-openhands browser bbox 'testid=close-agent-canvas-update-modal'`, then `control-openhands browser screenshot --feature F09.update-modal-phone --name dialog`. Both boxes have `insideViewport` `true` and `pageHorizontalOverflow` `false`: the dialog is 351 px wide at `x` 19.5 and Close is at `x` 329.5; the screenshot shows the whole \"Agent Canvas\" title and the X. `control-openhands browser click 'testid=close-agent-canvas-update-modal'` closes it (`control-openhands browser count 'testid=agent-canvas-update-modal'` is `0`). Run `control-openhands browser viewport narrow` (320×700) and repeat the `goto`, `click`, `wait` and both `bbox` commands: the dialog is 288 px wide at `x` 16 and Close is at `x` 263, both inside the viewport; `control-openhands browser press Escape` closes it. Then `control-openhands browser viewport desktop`.", "raw": "**Update dialog at phone width (`F09.update-modal-phone`).** Run `control-openhands browser viewport phone`, `control-openhands browser goto /settings`, `control-openhands browser click 'testid=settings-mobile-hub >> testid=agent-canvas-update-toggle'`, `control-openhands browser wait 'testid=agent-canvas-update-modal'`, `control-openhands browser bbox 'testid=agent-canvas-update-modal'` and `control-openhands browser bbox 'testid=close-agent-canvas-update-modal'`, then `control-openhands browser screenshot --feature F09.update-modal-phone --name dialog`. Both boxes have `insideViewport` `true` and `pageHorizontalOverflow` `false`: the dialog is 351 px wide at `x` 19.5 and Close is at `x` 329.5; the screenshot shows the whole \"Agent Canvas\" title and the X. `control-openhands browser click 'testid=close-agent-canvas-update-modal'` closes it (`control-openhands browser count 'testid=agent-canvas-update-modal'` is `0`). Run `control-openhands browser viewport narrow` (320×700) and repeat the `goto`, `click`, `wait` and both `bbox` commands: the dialog is 288 px wide at `x` 16 and Close is at `x` 263, both inside the viewport; `control-openhands browser press Escape` closes it. Then `control-openhands browser viewport desktop`.", "source_line": 80, "expected": [ "the update dialog fits a 390 px (and a 320 px) viewport with its title and Close button on screen, and Close dismisses it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--sidebar-version-tile", "title": "Sidebar version tile", "ids": [ "F09.sidebar-version-tile" ], "body": "On `/` run `control-openhands browser count 'testid=agent-canvas-version-tile'`. It is `0` unless the npm check found a newer version, also when the check succeeded with \"Up to date\" (positive case blocked without registry access or while the client is the latest release).", "raw": "**Sidebar version tile (`F09.sidebar-version-tile`).** On `/` run `control-openhands browser count 'testid=agent-canvas-version-tile'`. It is `0` unless the npm check found a newer version, also when the check succeeded with \"Up to date\" (positive case blocked without registry access or while the client is the latest release).", "source_line": 81, "expected": [ "the main sidebar shows an \"Agent Canvas\" version tile only when an update is available." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F09--unknown-sub-page", "title": "Unknown sub-page", "ids": [ "F09.unknown-subpath" ], "body": "Run `control-openhands browser goto /settings/does-not-exist` and `control-openhands browser snapshot 'testid=not-found-screen'`: `heading \"Page not found\"`, the paragraph `This address does not match any page. Check the URL, or go back to the home page.` and `link \"Home\"`. The settings shell is absent (`control-openhands browser count 'testid=settings-screen'` and `control-openhands browser count 'testid=settings-navbar-desktop'` are `0`), while the app sidebar stays (`control-openhands browser count 'aside[data-collapsed]'` is `1`). `control-openhands browser screenshot --feature F09.unknown-subpath --name not-found` shows the message beside the sidebar. Leave with `control-openhands browser click 'testid=not-found-home-link' --expect-url '/$'`.", "raw": "**Unknown sub-page (`F09.unknown-subpath`).** Run `control-openhands browser goto /settings/does-not-exist` and `control-openhands browser snapshot 'testid=not-found-screen'`: `heading \"Page not found\"`, the paragraph `This address does not match any page. Check the URL, or go back to the home page.` and `link \"Home\"`. The settings shell is absent (`control-openhands browser count 'testid=settings-screen'` and `control-openhands browser count 'testid=settings-navbar-desktop'` are `0`), while the app sidebar stays (`control-openhands browser count 'aside[data-collapsed]'` is `1`). `control-openhands browser screenshot --feature F09.unknown-subpath --name not-found` shows the message beside the sidebar. Leave with `control-openhands browser click 'testid=not-found-home-link' --expect-url '/$'`.", "source_line": 82, "expected": [ "an unknown `/settings/` URL shows the app's \"Page not found\" page inside the app shell (sidebar and a Home link), without the settings shell." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F09--after-the-family", "title": "After the family", "ids": [], "body": "Run `control-openhands browser errors --clear` before the family and `control-openhands browser errors --app-only` after it. Expected: `pageErrors` `0` and `appErrors` `0`; the unknown sub-page adds no errors. The Agent profile editor logs `[i18n] Missing translation for key \"SCHEMA$TOOL_CONCURRENCY_LIMIT$…\"` console warnings, counted under `warnings`, not as errors (the Agent page's family). A registry request failure, when the sandbox blocks npm, is external.", "raw": "**After the family.** Run `control-openhands browser errors --clear` before the family and `control-openhands browser errors --app-only` after it. Expected: `pageErrors` `0` and `appErrors` `0`; the unknown sub-page adds no errors. The Agent profile editor logs `[i18n] Missing translation for key \"SCHEMA$TOOL_CONCURRENCY_LIMIT$…\"` console warnings, counted under `warnings`, not as errors (the Agent page's family). A registry request failure, when the sandbox blocks npm, is external.", "source_line": 83, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F09--contract-f09-locked-cloud-landing", "title": "Locked cloud landing", "ids": [ "F09.locked-cloud-landing" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "on a canvas locked to a Cloud host, `/settings` lands on `/settings/app` instead." ], "notes": [], "annotations": [], "states": [ "contract" ] }, { "name": "F09--contract-f09-hidden-llm-redirect", "title": "Hidden llm redirect", "ids": [ "F09.hidden-llm-redirect" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "with the `hide_llm_settings` feature flag, the LLM link disappears and `/settings/llm` redirects to `/settings/agents`." ], "notes": [], "annotations": [], "states": [ "contract" ] } ] }, { "id": "F10", "title": "LLM profiles", "file": "F10-llm-profiles.md", "sha256": "4594c48dc9f877bab6330eee6f1b4b16355b97e6523df66a6a52cd0ef8434d80", "overview": "An LLM profile is a named model configuration: provider and model (or a custom\nmodel and base URL), an API key or a ChatGPT subscription, an optional\nprovider-connection link, and every advanced LLM option. Under Settings → LLM a\nuser lists profiles (grouped by provider connection once any are linked), adds\nand edits them in an embedded form whose Save first runs a one-token validation\ncall, renames, duplicates, deletes and sets one as **Default**, and bulk-adds\nmodels from a provider. The Default profile is the model new conversations run\non. Provider connections themselves (add, edit, delete) are F11.", "entry": "- Sidebar settings gear (`backend-selector-settings-link`, opens `/settings/agents`), then **LLM** in the settings navigation (`sidebar-settings-/settings/llm`).\n- Direct URL `/settings/llm`.\n- Command menu (`Control+k` or `Meta+k`, or `command-menu-trigger` in the sidebar): search \"LLM\", choose **LLM profiles** (\"Manage models, providers, and API keys.\").\n- On the list: **Add LLM Profile** (`add-llm-profile`), **Add from provider connections** (`add-models-from-provider`), each row's `...` (`profile-menu-trigger`), and a provider connection row's `...` → **Bulk add** (`provider-connection-add-models`).\n- The create/edit editor has no URL of its own: it replaces the list in place at `/settings/llm`; **Back** (`back-to-profiles`) and **Cancel** (`cancel-profile-btn`) return to the list.", "given": "Preconditions:\n\n- Baseline state (launched, doctored, `onboard --skip` done). Do not run `llm preset deepseek` first: a fresh run has no profiles, which the empty state and \"first profile becomes Default\" need.\n- A DeepSeek key for real saves, either in the environment as `DEEPSEEK_API_KEY` or in a file. Pass it only with `browser fill ... --value-env DEEPSEEK_API_KEY` or `--value-file `; never on the command line.\n- No profile or connection named `QA_*`, `deepseek-chat` or `deepseek-v3` exists (`control-openhands api GET /api/profiles`, `control-openhands api GET /api/llm/provider-connections`).\n- Select a row by exact name with the CSS form `'[data-testid=profile-row]:has([title=\"\"])'`; `has-text=QA_pro` also matches `QA_pro-copy`.\n- Read toasts with `control-openhands browser wait-text ''` (or `browser toasts`) right after the click; they disappear after a few seconds.\n- Blocked here: completing `F10.subscription` sign-in (needs a ChatGPT Plus/Pro account), `F10.cloud-readonly` (needs a Cloud org member account), `F10.cloud-provider-pagination` (needs a Cloud organization backend where the user can manage profiles), and `F10.schema-unavailable` (needs an agent server without the schema endpoints, for example an older `launch --sdk-ref`). The `F10.list-states` error is arranged by the Load error bullet (a linked profile plus this run's corrupted connections store).", "gotchas": "- Save always runs a real completion against the provider (except for connection-linked profiles). A bad or missing key fails fast with the provider's message; budget one tiny call per Save.\n- The editor's starting tab is inferred from the values it loads. Edit opens on whichever tab the profile's non-default values need (`QA_renamed` and `QA_adv` opened on Basic, `QA_broken` on All). A new editor opens on Basic only while the agent's LLM settings are all defaults: once the Default profile has a temperature (the schema step) it opens on All, and this survives deleting every profile, because the agent settings keep the last applied values. Click the tab you need before touching its fields, or `llm-provider-input` / `llm-custom-model-input` / `llm-api-key-help-anchor` will not exist.\n- `control-openhands service stop agent-server` cannot show this page's error states: the whole app switches to the disconnected-backend screen.\n- In Edit, `llm-api-key-input` holds the stored *encrypted* key token (about 140 masked characters), not an empty field. Never run `browser value` on it: it prints the token. Leave it untouched to keep the key.\n- An open provider or model listbox intercepts clicks on the editor's buttons (`cancel-profile-btn`, `save-profile-btn` time out with \"subtree intercepts pointer events\"): `browser press Escape` first.\n- Provider and model options render in a portal outside the editor and modals: select `testid=provider-item-` unscoped. Only verified providers have `provider-item-*` IDs and only Other models have `model-item-*` IDs; use `role=option[name=\"...\"][exact]` for the rest. `browser testids --filter model-item` lists only items scrolled into view; `browser count` finds the rest.\n- The auto-filled Profile Name never reuses an existing profile's name: picking a model whose name is already a profile's (for example `deepseek-flash` after `llm preset deepseek`) empties the field, which disables Save. Recipes fill `profile-name-input` with a `QA_*` name anyway.\n- A duplicate profile name is flagged: `aria-invalid` `true` and a taken-name alert. The alert has its own wording, `An LLM profile with this name already exists. …`.\n- The Rename modal does not check duplicates client-side (Rename stays enabled); the server's 409 comes back as the toast `Profile '' already exists`, and `browser errors --app-only` counts one `http-error` and one `console.error` for it.\n- Set as default (and deleting the Default) rewrites the agent's active LLM settings (`control-openhands llm show`). Deleting every profile leaves the last applied model and key in agent settings.\n- The app never leaves profiles without a Default: `src/routes/root-layout.tsx` runs `useEnsureActiveProfile` (`src/hooks/use-ensure-active-profile.ts`), which activates the first profile with a key (else the first profile) whenever profiles exist but none is Default, whether never activated or because the Default was deleted (this is what promotes a profile in `F10.delete-default`). So `llm set --no-activate` keeps the Default only while another profile is Default: on a run whose only profile is the arranged one, `control-openhands api GET /api/profiles --pick active_profile` flips from `null` to it at the next page load (`browser reload`) and the agent's LLM settings follow (`llm show`).\n- Profiles created by Add models are named from the model (`deepseek-chat`, …), not `QA_*`. The chooser hides a model when a profile with that *name* exists, not when a profile uses that model. Clean them up before the connection: the server refuses to delete a connection while profiles reference it.\n- A profile linked to a missing connection is grouped under a header showing the raw connection id (`00000000…`) instead of a name.\n- The All tab logs many `[i18n] Missing translation for key \"SCHEMA$LLM$...\"` console warnings; they are not page errors.\n- The list follows the server's order, which sorts by profile file name: `QA_pro-copy-1` lists before `QA_pro-copy` (`-` sorts before `.json`'s `.`). Do not assert a strict alphabetical order on names that are prefixes of each other.\n- Toasts last only a few seconds. A `browser screenshot` taken after a slow command can miss them, so assert with `browser wait-text` or `browser toasts`.\n- A new profile keeps the inherited LLM options the editor shows, such as temperature.\n- `Subscription models are not loaded yet.` uses the translated `SETTINGS$SUBSCRIPTION_MODELS_NOT_LOADED` copy in settings and onboarding. It shows when a subscription profile is saved before `GET /api/llm/subscription/openai/models` answers; checking that branch needs the real request to be pending or failed.\n- Known issue [#17785](https://github.com/OpenHands/OpenHands/issues/17785): Saving DeepSeek-v4-flash on Windows Desktop can fail with missing cache_creation_tokens instead of saving. The reported path needs a Windows Desktop 1.24.0 environment with its SDK dependency resolution and an approved DeepSeek key; this macOS source run cannot establish that result. (`F10.create-validation`).\n- Known issue [#17706](https://github.com/OpenHands/OpenHands/issues/17706): A freshly resolved npm-launcher environment can reject openhands/claude-* conversations because LiteLLM sends prompt_cache_key. Reproduction needs an approved OpenHands proxy key with an Anthropic model and the affected fresh transitive dependency resolution; a keyless initialization cannot prove provider acceptance. (`F10.default-used`).\n- Known issue [#17391](https://github.com/OpenHands/OpenHands/issues/17391): ChatGPT Business subscription login can be refused or use a fallback model with a 404. Completing the entitlement and selected-model checks needs an approved ChatGPT Business workspace login and the reported Docker subscription runtime. (`F10.subscription`).\n- Known issue [OpenHands/software-agent-sdk#5181](https://github.com/OpenHands/software-agent-sdk/issues/5181): The SDK can turn an advertised output limit into a requested default cap and silently drop an explicitly enabled reasoning_effort. The narrowed maintainer scope needs an authenticated OpenRouter/proxy route exposing lower-limit endpoints and capability metadata, or a real endpoint accepting the explicit override; a synthetic empty-metadata response would not establish production behavior. (`F10.schema-validation`).\n- Known issue [OpenHands/software-agent-sdk#4934](https://github.com/OpenHands/software-agent-sdk/issues/4934): Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (`F10.schema-validation`, `F10.default-used`).\n- Known issue [OpenHands/software-agent-sdk#4613](https://github.com/OpenHands/software-agent-sdk/issues/4613): A namespaced model such as openai/openai/example-model can have its inner provider prefix stripped again in Responses API and reach the provider as example-model. Reproduction needs an approved OpenAI-compatible endpoint serving the namespaced model and its exact stored API-mode settings; a loopback connection refusal cannot prove the outbound identifier. (`F10.custom-model`).\n- Known issue [OpenHands/software-agent-sdk#4511](https://github.com/OpenHands/software-agent-sdk/issues/4511): Disabling prompt caching can still emit cache_control on preexisting cache-marked tool messages. The report only supplies a Python serializer reproduction; a Canvas check needs the exact persisted/tool-message precondition and a provider request that exposes the cache markers, with approved model access. (`F10.schema-validation`).\n- Known issue [OpenHands/software-agent-sdk#4250](https://github.com/OpenHands/software-agent-sdk/issues/4250): A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (`F10.custom-model`, `F10.default-used`).\n- Known issue [OpenHands/software-agent-sdk#4247](https://github.com/OpenHands/software-agent-sdk/issues/4247): An LM Studio configuration can fail with LLM Provider NOT provided although its models endpoint works. Reproduction needs the reachable LM Studio service and the exact model ID/provider-qualified config, which the report does not supply as text; the reported CLI/Docker SDK 1.16.1 installation is not this source stack. (`F10.custom-model`).\n- Known issue [OpenHands/software-agent-sdk#5890](https://github.com/OpenHands/software-agent-sdk/issues/5890): Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (`F10.edit`, `F10.default-used`).\n- Known issue [OpenHands/software-agent-sdk#5862](https://github.com/OpenHands/software-agent-sdk/issues/5862): DeepSeek invalid-key401 responses can be classified as bad-request errors. Blocked by this pass's loopback-only provider routing: a real invalid-key validation request to DeepSeek must be authorized to capture its actual401 shape. No usable key or successful inference is needed, but a loopback connection error does not prove this classification. (`F10.create-validation`).\n- Known issue [OpenHands/software-agent-sdk#5838](https://github.com/OpenHands/software-agent-sdk/issues/5838): A live LLM switch during a running tool step can be acknowledged before the new LLM is persisted for GET/restart. Blocked: needs a genuine model-issued bounded tool step (for example `sleep 20`) and an approved starting model. Refused-loopback initialization never enters that step. (`F10.default-used`).\n- Known issue [#17806](https://github.com/OpenHands/OpenHands/issues/17806): A model-only Save must preserve an untouched key. Current SDK1.55 refused-loopback validation rejects Save before persistence; the stored key marker remains set. Completing Basic/Advanced preserved-key and runtime-switch proof needs an approved route that accepts preflight. Keep dummy traffic on refused loopback until an external route is authorized; a rejected Save does not establish either key loss or successful preservation through Save. (`F10.edit`).", "features": [ { "id": "F10.list", "description": "the \"Available Profiles\" list shows each row's name, model, a `Default` badge on the active profile and a `...` menu, sorted by name; reachable from the sidebar, the direct URL and the command menu." }, { "id": "F10.list-empty", "description": "with no profiles the list reads `No profiles saved yet. Add a profile to save your LLM configurations.` and both Add buttons stay available." }, { "id": "F10.list-states", "description": "a spinner shows while profiles load; a load failure shows `Failed to load profiles` in the theme's error colour." }, { "id": "F10.list-grouped", "description": "once any profile links to a provider connection, rows group under the connection's name, with unlinked rows last under `Not linked`; with no linked rows the list is flat again." }, { "id": "F10.broken-link", "description": "a profile whose linked connection no longer exists shows a warning `Broken link` badge whose tooltip says to edit the profile to re-link it; its editor offers the orphaned id in the Provider connection selector so it can be cleared." }, { "id": "F10.actions-menu", "description": "the row menu offers Edit, Rename, Duplicate, Set as default (disabled on the Default row) and Delete; it focuses Edit on open (mouse or Enter), arrows cycle over the enabled items (skipping a disabled Set as default), and Escape, Tab or an outside click close it." }, { "id": "F10.set-default", "description": "Set as default toasts `Switched to profile \"\"`, moves the badge, persists and becomes the agent's active LLM." }, { "id": "F10.duplicate", "description": "Duplicate saves `-copy`, then `-copy-1`, with the same model and stored key; the copy is not Default." }, { "id": "F10.rename", "description": "the Rename modal is prefilled and focused; invalid names disable Rename; an unchanged name or Cancel just closes; Enter submits; success toasts and persists, and the Default badge follows a renamed default." }, { "id": "F10.rename-conflict", "description": "renaming onto an existing name is refused with the server's message (`Profile '' already exists`) and the modal stays open." }, { "id": "F10.delete", "description": "Delete asks `Are you sure you want to delete the profile \"\"? This action cannot be undone.` with Cancel focused; Cancel keeps the row, Delete removes it with a toast." }, { "id": "F10.delete-default", "description": "deleting the Default profile promotes another remaining profile to Default." }, { "id": "F10.create", "description": "Add LLM Profile → provider, model and API key → Save shows `Validating...` then `Saving...`, toasts `Profile \"\" created` and returns to the list; the first profile on an empty list becomes Default." }, { "id": "F10.create-prefill", "description": "a new editor prefills the backend default model with a blank key, base URL and connection; the other LLM options it shows come from the agent's current LLM settings, i.e. the last applied Default profile (the editor opens on the tab they need), and Save stores what the form shows." }, { "id": "F10.api-key-help", "description": "under the API key field (Basic and Advanced), `Don't know your API key? Click here for instructions` opens the OpenHands docs on getting an API key in a new tab." }, { "id": "F10.name-autofill", "description": "the Profile Name is auto-filled from the chosen model and follows later model changes until the user types a name; clearing the field resumes it." }, { "id": "F10.name-validation", "description": "a name that breaks the rule (letter or digit first; letters, digits, `.`, `_`, `-`; max 64) disables Save and turns the rule text red; a name that already exists marks the field invalid, shows a taken-name message about LLM profiles and disables Save (Edit accepts the profile's own name)." }, { "id": "F10.create-validation", "description": "a Save whose validation call fails toasts the provider error and keeps the editor open with nothing saved." }, { "id": "F10.custom-model", "description": "the Advanced tab takes a free-text model and Base URL; an empty model is refused with `Model is required`." }, { "id": "F10.model-picker", "description": "the Basic provider and model comboboxes group options as Verified Models and Other Models; changing provider clears the model; the OpenHands provider shows account and key help." }, { "id": "F10.provider-list-complete", "description": "the Basic provider combobox offers every provider the backend reports or has verified models for (over a hundred, `xai` included); picking a late-sorting one fills the field and loads its models." }, { "id": "F10.cloud-provider-pagination", "description": "on a Cloud organization backend the provider list is fetched from `providers/search` page by page, so providers past the first page (`xai`, `openrouter`) are offered too (blocked here)." }, { "id": "F10.edit", "description": "Edit loads the profile (`Editing profile \"\" - save to apply changes`) with the stored key masked; Save stays disabled until something changes; a changed name renames before saving; the stored key is kept." }, { "id": "F10.basic-save-keeps-base-url", "description": "re-saving a profile from the Basic view without changing its model keeps the Base URL that view hides." }, { "id": "F10.edit-default-reapply", "description": "saving the Default profile re-applies it to the agent's LLM settings." }, { "id": "F10.editor-discard", "description": "Back and Cancel return to the list without saving and without an unsaved-changes prompt." }, { "id": "F10.schema-validation", "description": "bad values in schema fields (Temperature over 2, Top P over 1, invalid JSON) are refused with a toast naming the field." }, { "id": "F10.connection-link", "description": "with connections present the editor shows a Provider connection selector; linking hides API key and Base URL, skips validation on Save and regroups the row; None restores the key field." }, { "id": "F10.add-models", "description": "\"Add from provider connections\" (or a connection row's **Bulk add**) creates one linked or keyless profile per checked model; models that already have a profile of that name are hidden." }, { "id": "F10.subscription", "description": "Authentication → ChatGPT subscription swaps the model fields for a subscription model list and a device-login card (code, sign-in link, poll, cancel); switching back restores the API-key model." }, { "id": "F10.schema-unavailable", "description": "a backend without the settings-schema endpoints shows `SDK settings schema unavailable.` with an upgrade or session-key hint." }, { "id": "F10.cloud-readonly", "description": "Cloud org members without edit rights see the list read-only (no Add buttons, no row menus, no Provider connections section)." }, { "id": "F10.phone", "description": "list, row menu and editor fit a 390 px viewport without horizontal overflow." }, { "id": "F10.default-used", "description": "a conversation started after creating a profile in the UI runs on the Default profile's model." } ], "scenarios": [ { "name": "F10--empty-list", "title": "Empty list", "ids": [ "F10.list-empty" ], "body": "On a fresh run, navigate from the sidebar: `control-openhands browser click 'testid=backend-selector-settings-link' --expect-url '/settings'`, `control-openhands browser click 'testid=sidebar-settings-/settings/llm' --expect-url '/settings/llm'`, then `control-openhands browser text 'testid=profiles-empty'`. The text is `No profiles saved yet. Add a profile to save your LLM configurations.` and `control-openhands browser testids main` lists `add-llm-profile`, `add-models-from-provider` and `profiles-empty`.", "raw": "**Empty list (`F10.list-empty`).** On a fresh run, navigate from the sidebar: `control-openhands browser click 'testid=backend-selector-settings-link' --expect-url '/settings'`, `control-openhands browser click 'testid=sidebar-settings-/settings/llm' --expect-url '/settings/llm'`, then `control-openhands browser text 'testid=profiles-empty'`. The text is `No profiles saved yet. Add a profile to save your LLM configurations.` and `control-openhands browser testids main` lists `add-llm-profile`, `add-models-from-provider` and `profiles-empty`.", "source_line": 70, "expected": [ "with no profiles the list reads `No profiles saved yet. Add a profile to save your LLM configurations.` and both Add buttons stay available." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--open-the-editor", "title": "Open the editor", "ids": [ "F10.create" ], "body": "Run `control-openhands browser click 'testid=add-llm-profile'`. `control-openhands browser text 'testid=profile-editor-title'` is `Add LLM Profile`, `control-openhands browser text 'testid=profile-editor-description'` is `Configure your LLM settings below, then click Save to create this profile.`, and the form opens on the Basic tab with provider `OpenAI` and model `gpt-5.6-sol` (`browser value 'testid=llm-provider-input'`, `browser value 'testid=llm-model-input'`).", "raw": "**Open the editor (`F10.create`).** Run `control-openhands browser click 'testid=add-llm-profile'`. `control-openhands browser text 'testid=profile-editor-title'` is `Add LLM Profile`, `control-openhands browser text 'testid=profile-editor-description'` is `Configure your LLM settings below, then click Save to create this profile.`, and the form opens on the Basic tab with provider `OpenAI` and model `gpt-5.6-sol` (`browser value 'testid=llm-provider-input'`, `browser value 'testid=llm-model-input'`).", "source_line": 71, "expected": [ "Add LLM Profile → provider, model and API key → Save shows `Validating...` then `Saving...`, toasts `Profile \"\" created` and returns to the list; the first profile on an empty list becomes Default." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--api-key-help-link", "title": "API key help link", "ids": [ "F10.api-key-help" ], "body": "In that editor run `control-openhands browser click 'testid=sdk-section-basic-toggle'` (a new editor opens on All once the agent's LLM settings carry a non-default option; see `F10.create-prefill`), then `control-openhands browser text 'testid=llm-api-key-help-anchor'` is `Don't know your API key? Click here for instructions`; its link has `target` `_blank` (`browser eval \"document.querySelector('[data-testid=llm-api-key-help-anchor] a, a[data-testid=llm-api-key-help-anchor]').target\"`). Run `control-openhands browser click 'testid=llm-api-key-help-anchor >> role=link'`, then `control-openhands browser tabs` (rerun once if it still lists one page): page 1 is `https://docs.openhands.dev/.../local-setup#getting-an-api-key`. Return with `control-openhands browser close-tab 1` and `control-openhands browser tab 0`. The Advanced tab shows the same link as `testid=llm-api-key-help-anchor-advanced`.", "raw": "**API key help link (`F10.api-key-help`).** In that editor run `control-openhands browser click 'testid=sdk-section-basic-toggle'` (a new editor opens on All once the agent's LLM settings carry a non-default option; see `F10.create-prefill`), then `control-openhands browser text 'testid=llm-api-key-help-anchor'` is `Don't know your API key? Click here for instructions`; its link has `target` `_blank` (`browser eval \"document.querySelector('[data-testid=llm-api-key-help-anchor] a, a[data-testid=llm-api-key-help-anchor]').target\"`). Run `control-openhands browser click 'testid=llm-api-key-help-anchor >> role=link'`, then `control-openhands browser tabs` (rerun once if it still lists one page): page 1 is `https://docs.openhands.dev/.../local-setup#getting-an-api-key`. Return with `control-openhands browser close-tab 1` and `control-openhands browser tab 0`. The Advanced tab shows the same link as `testid=llm-api-key-help-anchor-advanced`.", "source_line": 72, "expected": [ "under the API key field (Basic and Advanced), `Don't know your API key? Click here for instructions` opens the OpenHands docs on getting an API key in a new tab." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--provider-and-model-lists", "title": "Provider and model lists", "ids": [ "F10.model-picker" ], "body": "Run `control-openhands browser click 'testid=llm-provider-input'` and `control-openhands browser snapshot 'role=listbox'`: groups `Verified Models` (OpenHands, Anthropic, OpenAI, Mistral AI, Gemini, deepseek, Moonshot, minimax, glm, nvidia, qwen) and `Other Models`. Pick DeepSeek with `control-openhands browser fill 'testid=llm-provider-input' deepseek` and `control-openhands browser click 'testid=provider-item-deepseek'`; `browser value 'testid=llm-model-input'` is now empty. `control-openhands browser click 'testid=llm-model-input'` and `browser snapshot 'role=listbox'` list `deepseek-chat`, `deepseek-v4-pro`, … as Verified and `deepseek-coder`, `deepseek-flash`, … under Other. For the OpenHands provider (`browser fill 'testid=llm-provider-input' openhands`, `browser click 'testid=provider-item-openhands'`), `control-openhands browser text 'testid=openhands-account-help'` is `Need an OpenHands Account? Click here`; after `control-openhands browser click 'role=option[name=\"claude-sonnet-5\"][exact]'` in the model list, `control-openhands browser text 'testid=openhands-api-key-help'` starts `You can find your OpenHands Provider LLM Key in the OpenHands LLM Key section of OpenHands Cloud`. Screenshot: `browser screenshot --feature F10.model-picker --name openhands-provider`.", "raw": "**Provider and model lists (`F10.model-picker`).** Run `control-openhands browser click 'testid=llm-provider-input'` and `control-openhands browser snapshot 'role=listbox'`: groups `Verified Models` (OpenHands, Anthropic, OpenAI, Mistral AI, Gemini, deepseek, Moonshot, minimax, glm, nvidia, qwen) and `Other Models`. Pick DeepSeek with `control-openhands browser fill 'testid=llm-provider-input' deepseek` and `control-openhands browser click 'testid=provider-item-deepseek'`; `browser value 'testid=llm-model-input'` is now empty. `control-openhands browser click 'testid=llm-model-input'` and `browser snapshot 'role=listbox'` list `deepseek-chat`, `deepseek-v4-pro`, … as Verified and `deepseek-coder`, `deepseek-flash`, … under Other. For the OpenHands provider (`browser fill 'testid=llm-provider-input' openhands`, `browser click 'testid=provider-item-openhands'`), `control-openhands browser text 'testid=openhands-account-help'` is `Need an OpenHands Account? Click here`; after `control-openhands browser click 'role=option[name=\"claude-sonnet-5\"][exact]'` in the model list, `control-openhands browser text 'testid=openhands-api-key-help'` starts `You can find your OpenHands Provider LLM Key in the OpenHands LLM Key section of OpenHands Cloud`. Screenshot: `browser screenshot --feature F10.model-picker --name openhands-provider`.", "source_line": 73, "expected": [ "the Basic provider and model comboboxes group options as Verified Models and Other Models; changing provider clears the model; the OpenHands provider shows account and key help." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--every-provider-is-offered", "title": "Every provider is offered", "ids": [ "F10.provider-list-complete" ], "body": "Run `control-openhands browser click 'testid=cancel-profile-btn'` (the Provider and model lists bullet leaves its editor open; skip this when the list is showing), `control-openhands browser click 'testid=add-llm-profile'`, `control-openhands browser click 'testid=sdk-section-basic-toggle'` and `control-openhands browser click 'testid=llm-provider-input'`. `control-openhands browser count 'role=listbox >> role=option'` is the size of the union of the two backend lists the page merges: the names in `control-openhands api GET /api/llm/providers --pick providers` (under `value`; `control-openhands api GET /api/llm/providers --pick providers | jq '.value | length'` is `160` today) and the provider keys of `control-openhands api GET /api/llm/models/verified --max-bytes 200000` (its `body` is `{\"models\": {: [, …]}}`; `control-openhands api GET /api/llm/models/verified --max-bytes 200000 | jq '.body.models | keys | length'` is `12` today, four of which, `glm`, `nvidia`, `openhands` and `qwen`, are missing from the first list). The union is `jq -s '([.[0].value[]] + (.[1].body.models | keys)) | unique | length' <(control-openhands api GET /api/llm/providers --pick providers) <(control-openhands api GET /api/llm/models/verified --max-bytes 200000)` (bash process substitution): `164` today, and the listbox count is the same `164`, with every backend name present under its display label (`openhands` as `OpenHands`, `mistral` as `Mistral AI`, `azure_ai` as `Azure AI Studio`). `control-openhands browser count 'role=option[name=\"xai\"][exact]'` is `1` (it sorts last but three, before `xiaomi_mimo`, `xinference` and `zai`). Pick it with `control-openhands browser click 'role=option[name=\"xai\"][exact]'`: `control-openhands browser value 'testid=llm-provider-input'` is `xai`, `control-openhands browser value 'testid=llm-model-input'` is empty, and after `control-openhands browser click 'testid=llm-model-input'` the listbox offers `grok-4.20`, `grok-4.20-0309`, … under Other Models (57 today; `control-openhands browser screenshot --feature F10.provider-list-complete --name xai-picked`). Close with `control-openhands browser press Escape` (the open listbox would swallow the next click) and `control-openhands browser click 'testid=cancel-profile-btn'`; the list is back.", "raw": "**Every provider is offered (`F10.provider-list-complete`).** Run `control-openhands browser click 'testid=cancel-profile-btn'` (the Provider and model lists bullet leaves its editor open; skip this when the list is showing), `control-openhands browser click 'testid=add-llm-profile'`, `control-openhands browser click 'testid=sdk-section-basic-toggle'` and `control-openhands browser click 'testid=llm-provider-input'`. `control-openhands browser count 'role=listbox >> role=option'` is the size of the union of the two backend lists the page merges: the names in `control-openhands api GET /api/llm/providers --pick providers` (under `value`; `control-openhands api GET /api/llm/providers --pick providers | jq '.value | length'` is `160` today) and the provider keys of `control-openhands api GET /api/llm/models/verified --max-bytes 200000` (its `body` is `{\"models\": {: [, …]}}`; `control-openhands api GET /api/llm/models/verified --max-bytes 200000 | jq '.body.models | keys | length'` is `12` today, four of which, `glm`, `nvidia`, `openhands` and `qwen`, are missing from the first list). The union is `jq -s '([.[0].value[]] + (.[1].body.models | keys)) | unique | length' <(control-openhands api GET /api/llm/providers --pick providers) <(control-openhands api GET /api/llm/models/verified --max-bytes 200000)` (bash process substitution): `164` today, and the listbox count is the same `164`, with every backend name present under its display label (`openhands` as `OpenHands`, `mistral` as `Mistral AI`, `azure_ai` as `Azure AI Studio`). `control-openhands browser count 'role=option[name=\"xai\"][exact]'` is `1` (it sorts last but three, before `xiaomi_mimo`, `xinference` and `zai`). Pick it with `control-openhands browser click 'role=option[name=\"xai\"][exact]'`: `control-openhands browser value 'testid=llm-provider-input'` is `xai`, `control-openhands browser value 'testid=llm-model-input'` is empty, and after `control-openhands browser click 'testid=llm-model-input'` the listbox offers `grok-4.20`, `grok-4.20-0309`, … under Other Models (57 today; `control-openhands browser screenshot --feature F10.provider-list-complete --name xai-picked`). Close with `control-openhands browser press Escape` (the open listbox would swallow the next click) and `control-openhands browser click 'testid=cancel-profile-btn'`; the list is back.", "source_line": 74, "expected": [ "the Basic provider combobox offers every provider the backend reports or has verified models for (over a hundred, `xai` included); picking a late-sorting one fills the field and loads its models." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--cloud-provider-pagination", "title": "Cloud provider pagination", "ids": [ "F10.cloud-provider-pagination" ], "body": "Blocked: needs a Cloud organization backend where the user can manage profiles (local backends serve the whole list in one `GET /api/llm/providers`). There, the Basic provider list comes from `providers/search` one page at a time (`control-openhands browser network --filter providers/search` lists one request per page, the later ones with a `page_id`), and `xai` and `openrouter`, which sort past the first page, are offered and pick like any other provider. Record `control-openhands evidence add --feature F10.cloud-provider-pagination --result blocked ...` naming the Cloud account.", "raw": "**Cloud provider pagination (`F10.cloud-provider-pagination`).** Blocked: needs a Cloud organization backend where the user can manage profiles (local backends serve the whole list in one `GET /api/llm/providers`). There, the Basic provider list comes from `providers/search` one page at a time (`control-openhands browser network --filter providers/search` lists one request per page, the later ones with a `page_id`), and `xai` and `openrouter`, which sort past the first page, are offered and pick like any other provider. Record `control-openhands evidence add --feature F10.cloud-provider-pagination --result blocked ...` naming the Cloud account.", "source_line": 75, "expected": [ "on a Cloud organization backend the provider list is fetched from `providers/search` page by page, so providers past the first page (`xai`, `openrouter`) are offered too (blocked here)." ], "notes": [], "annotations": [], "states": [ "blocked" ] }, { "name": "F10--create-the-first-profile", "title": "Create the first profile", "ids": [ "F10.create", "F10.name-autofill" ], "body": "In a fresh editor pick deepseek as above, then `control-openhands browser click 'testid=llm-model-input'` and `control-openhands browser click 'testid=model-item-deepseek-flash'`. `control-openhands browser value 'testid=profile-name-input'` now follows the model: `deepseek-flash` (right after the provider pick, while the model is still empty, it still reads `gpt-5.6-sol`). Run `control-openhands browser fill 'testid=profile-name-input' QA_flash`, `control-openhands browser fill 'testid=llm-api-key-input' --value-file ` (or `--value-env DEEPSEEK_API_KEY`), `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 4000` and `control-openhands browser wait-text 'Profile \"QA_flash\" created' --timeout 60000`. The observed button states are `Save`, `Validating...`, `Saving...`, then the editor closes. After `control-openhands browser reload`, `control-openhands browser text '[data-testid=profile-row]:has([title=\"QA_flash\"])'` reads `QA_flash`, `deepseek/deepseek-flash`, `Default`, and `control-openhands api GET /api/profiles` has `active_profile` `QA_flash`.", "raw": "**Create the first profile (`F10.create`, `F10.name-autofill`).** In a fresh editor pick deepseek as above, then `control-openhands browser click 'testid=llm-model-input'` and `control-openhands browser click 'testid=model-item-deepseek-flash'`. `control-openhands browser value 'testid=profile-name-input'` now follows the model: `deepseek-flash` (right after the provider pick, while the model is still empty, it still reads `gpt-5.6-sol`). Run `control-openhands browser fill 'testid=profile-name-input' QA_flash`, `control-openhands browser fill 'testid=llm-api-key-input' --value-file ` (or `--value-env DEEPSEEK_API_KEY`), `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 4000` and `control-openhands browser wait-text 'Profile \"QA_flash\" created' --timeout 60000`. The observed button states are `Save`, `Validating...`, `Saving...`, then the editor closes. After `control-openhands browser reload`, `control-openhands browser text '[data-testid=profile-row]:has([title=\"QA_flash\"])'` reads `QA_flash`, `deepseek/deepseek-flash`, `Default`, and `control-openhands api GET /api/profiles` has `active_profile` `QA_flash`.", "source_line": 76, "expected": [ "Add LLM Profile → provider, model and API key → Save shows `Validating...` then `Saving...`, toasts `Profile \"\" created` and returns to the list; the first profile on an empty list becomes Default.", "the Profile Name is auto-filled from the chosen model and follows later model changes until the user types a name; clearing the field resumes it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--a-typed-name-sticks-a-cleared-one-follows-again", "title": "A typed name sticks; a cleared one follows again", "ids": [ "F10.name-autofill" ], "body": "In a fresh editor (`control-openhands browser click 'testid=add-llm-profile'`) run `control-openhands browser fill 'testid=profile-name-input' QA_typed`, then pick deepseek as above, `control-openhands browser click 'testid=llm-model-input'` and `control-openhands browser click 'role=option[name=\"deepseek-v4-pro\"][exact]'`; `browser value 'testid=profile-name-input'` stays `QA_typed`. Run `control-openhands browser fill 'testid=profile-name-input' ''`, then `browser click 'testid=llm-model-input'` and `browser click 'testid=model-item-deepseek-flash'`: the name is `deepseek-flash`; pick `role=option[name=\"deepseek-v4-pro\"][exact]` again and it is `deepseek-v4-pro`.", "raw": "**A typed name sticks; a cleared one follows again (`F10.name-autofill`).** In a fresh editor (`control-openhands browser click 'testid=add-llm-profile'`) run `control-openhands browser fill 'testid=profile-name-input' QA_typed`, then pick deepseek as above, `control-openhands browser click 'testid=llm-model-input'` and `control-openhands browser click 'role=option[name=\"deepseek-v4-pro\"][exact]'`; `browser value 'testid=profile-name-input'` stays `QA_typed`. Run `control-openhands browser fill 'testid=profile-name-input' ''`, then `browser click 'testid=llm-model-input'` and `browser click 'testid=model-item-deepseek-flash'`: the name is `deepseek-flash`; pick `role=option[name=\"deepseek-v4-pro\"][exact]` again and it is `deepseek-v4-pro`.", "source_line": 77, "expected": [ "the Profile Name is auto-filled from the chosen model and follows later model changes until the user types a name; clearing the field resumes it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--name-rules", "title": "Name rules", "ids": [ "F10.name-validation" ], "body": "In the editor run `control-openhands browser fill 'testid=profile-name-input' 'QA bad'`; `control-openhands browser enabled 'testid=save-profile-btn'` is `false` and `control-openhands browser attr 'testid=profile-name-input' aria-invalid` is `true` (the same for `-QA` and a 65-character name), and the rule text under the field turns to the theme's danger colour: `control-openhands browser eval \"(()=>{const i=document.querySelector('[data-testid=profile-name-input]');return getComputedStyle(document.getElementById(i.getAttribute('aria-describedby'))).color+' '+getComputedStyle(document.body).getPropertyValue('--oh-color-danger')})()\"` reads `rgb(231, 106, 94) #e76a5e` on OpenHands-Neutral and `rgb(209, 52, 56) #D13438` on Light+ (`F16.color-theme`) (#18199); a valid name turns it back to the muted grey. An existing name (`QA_flash`) also gives `enabled` `false` and `aria-invalid` `true`, with a role=alert message under the field in the danger colour while the rule text stays grey. The message reads `An LLM profile with this name already exists. Choose a different name, or edit the existing one.` (`browser eval \"[...document.querySelectorAll('[role=alert]')].map(a=>a.innerText+' | '+getComputedStyle(a).color)\"`; screenshot `--feature F10.name-validation --name duplicate-neutral`). `QA_flash2` gives `true` and no message.", "raw": "**Name rules (`F10.name-validation`).** In the editor run `control-openhands browser fill 'testid=profile-name-input' 'QA bad'`; `control-openhands browser enabled 'testid=save-profile-btn'` is `false` and `control-openhands browser attr 'testid=profile-name-input' aria-invalid` is `true` (the same for `-QA` and a 65-character name), and the rule text under the field turns to the theme's danger colour: `control-openhands browser eval \"(()=>{const i=document.querySelector('[data-testid=profile-name-input]');return getComputedStyle(document.getElementById(i.getAttribute('aria-describedby'))).color+' '+getComputedStyle(document.body).getPropertyValue('--oh-color-danger')})()\"` reads `rgb(231, 106, 94) #e76a5e` on OpenHands-Neutral and `rgb(209, 52, 56) #D13438` on Light+ (`F16.color-theme`) (#18199); a valid name turns it back to the muted grey. An existing name (`QA_flash`) also gives `enabled` `false` and `aria-invalid` `true`, with a role=alert message under the field in the danger colour while the rule text stays grey. The message reads `An LLM profile with this name already exists. Choose a different name, or edit the existing one.` (`browser eval \"[...document.querySelectorAll('[role=alert]')].map(a=>a.innerText+' | '+getComputedStyle(a).color)\"`; screenshot `--feature F10.name-validation --name duplicate-neutral`). `QA_flash2` gives `true` and no message.", "source_line": 78, "expected": [ "a name that breaks the rule (letter or digit first; letters, digits, `.`, `_`, `-`; max 64) disables Save and turns the rule text red; a name that already exists marks the field invalid, shows a taken-name message about LLM profiles and disables Save (Edit accepts the profile's own name)." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--validation-failure", "title": "Validation failure", "ids": [ "F10.create-validation" ], "body": "In a fresh editor choose deepseek / `model-item-deepseek-flash`, fill `profile-name-input` with `QA_nokey`, leave the API key empty and run `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 3000` (states `Save`, `Validating...`, `Save`). `control-openhands browser wait-text 'Authentication Fails'` finds the toast `litellm.AuthenticationError: AuthenticationError: DeepseekException - Authentication Fails (...)`; `browser text 'testid=profile-editor-title'` is still `Add LLM Profile` and `control-openhands api GET /api/profiles` has no `QA_nokey`.", "raw": "**Validation failure (`F10.create-validation`).** In a fresh editor choose deepseek / `model-item-deepseek-flash`, fill `profile-name-input` with `QA_nokey`, leave the API key empty and run `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 3000` (states `Save`, `Validating...`, `Save`). `control-openhands browser wait-text 'Authentication Fails'` finds the toast `litellm.AuthenticationError: AuthenticationError: DeepseekException - Authentication Fails (...)`; `browser text 'testid=profile-editor-title'` is still `Add LLM Profile` and `control-openhands api GET /api/profiles` has no `QA_nokey`.", "source_line": 79, "expected": [ "a Save whose validation call fails toasts the provider error and keeps the editor open with nothing saved." ], "notes": [ "Known issue [#17785](https://github.com/OpenHands/OpenHands/issues/17785): Saving DeepSeek-v4-flash on Windows Desktop can fail with missing cache_creation_tokens instead of saving. The reported path needs a Windows Desktop 1.24.0 environment with its SDK dependency resolution and an approved DeepSeek key; this macOS source run cannot establish that result. (`F10.create-validation`).", "Known issue [OpenHands/software-agent-sdk#5862](https://github.com/OpenHands/software-agent-sdk/issues/5862): DeepSeek invalid-key401 responses can be classified as bad-request errors. Blocked by this pass's loopback-only provider routing: a real invalid-key validation request to DeepSeek must be authorized to capture its actual401 shape. No usable key or successful inference is needed, but a loopback connection error does not prove this classification. (`F10.create-validation`)." ], "annotations": [], "states": [ "issue" ] }, { "name": "F10--custom-model", "title": "Custom model", "ids": [ "F10.custom-model" ], "body": "In that editor run `control-openhands browser click 'testid=sdk-section-advanced-toggle'`. `browser value 'testid=llm-custom-model-input'` carries the Basic choice (`deepseek/deepseek-flash`; placeholder `openai/gpt-5.6-sol`), and `browser attr 'testid=base-url-input' placeholder` is `https://api.openai.com`. Run `control-openhands browser fill 'testid=llm-custom-model-input' ''`, `control-openhands browser fill 'testid=profile-name-input' QA_adv`, `control-openhands browser click 'testid=save-profile-btn'` and `control-openhands browser wait-text 'Model is required'`. Then `control-openhands browser fill 'testid=llm-custom-model-input' deepseek/deepseek-flash`, `control-openhands browser fill 'testid=base-url-input' https://api.deepseek.com`, fill the key as in Create, Save, and `control-openhands browser wait-text 'Profile \"QA_adv\" created' --timeout 60000`. `api GET /api/profiles` shows `QA_adv` with `base_url` `https://api.deepseek.com`.", "raw": "**Custom model (`F10.custom-model`).** In that editor run `control-openhands browser click 'testid=sdk-section-advanced-toggle'`. `browser value 'testid=llm-custom-model-input'` carries the Basic choice (`deepseek/deepseek-flash`; placeholder `openai/gpt-5.6-sol`), and `browser attr 'testid=base-url-input' placeholder` is `https://api.openai.com`. Run `control-openhands browser fill 'testid=llm-custom-model-input' ''`, `control-openhands browser fill 'testid=profile-name-input' QA_adv`, `control-openhands browser click 'testid=save-profile-btn'` and `control-openhands browser wait-text 'Model is required'`. Then `control-openhands browser fill 'testid=llm-custom-model-input' deepseek/deepseek-flash`, `control-openhands browser fill 'testid=base-url-input' https://api.deepseek.com`, fill the key as in Create, Save, and `control-openhands browser wait-text 'Profile \"QA_adv\" created' --timeout 60000`. `api GET /api/profiles` shows `QA_adv` with `base_url` `https://api.deepseek.com`.", "source_line": 80, "expected": [ "the Advanced tab takes a free-text model and Base URL; an empty model is refused with `Model is required`." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#4613](https://github.com/OpenHands/software-agent-sdk/issues/4613): A namespaced model such as openai/openai/example-model can have its inner provider prefix stripped again in Responses API and reach the provider as example-model. Reproduction needs an approved OpenAI-compatible endpoint serving the namespaced model and its exact stored API-mode settings; a loopback connection refusal cannot prove the outbound identifier. (`F10.custom-model`).", "Known issue [OpenHands/software-agent-sdk#4250](https://github.com/OpenHands/software-agent-sdk/issues/4250): A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (`F10.custom-model`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#4247](https://github.com/OpenHands/software-agent-sdk/issues/4247): An LM Studio configuration can fail with LLM Provider NOT provided although its models endpoint works. Reproduction needs the reachable LM Studio service and the exact model ID/provider-qualified config, which the report does not supply as text; the reported CLI/Docker SDK 1.16.1 installation is not this source stack. (`F10.custom-model`)." ], "annotations": [], "states": [ "issue" ] }, { "name": "F10--second-profile-from-a-verified-model", "title": "Second profile from a verified model", "ids": [], "body": "Repeat Create choosing `control-openhands browser click 'role=option[name=\"deepseek-v4-pro\"][exact]'` (verified models have no `model-item-*` test ID) and the name `QA_pro`. The toast reads `Profile \"QA_pro\" created`; the row `QA_pro deepseek/deepseek-v4-pro` has no badge.", "raw": "**Second profile from a verified model.** Repeat Create choosing `control-openhands browser click 'role=option[name=\"deepseek-v4-pro\"][exact]'` (verified models have no `model-item-*` test ID) and the name `QA_pro`. The toast reads `Profile \"QA_pro\" created`; the row `QA_pro deepseek/deepseek-v4-pro` has no badge.", "source_line": 81, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F10--row-menu", "title": "Row menu", "ids": [ "F10.actions-menu" ], "body": "Run `control-openhands browser click '[data-testid=profile-row]:has([title=\"QA_flash\"]) >> testid=profile-menu-trigger'` and `control-openhands browser snapshot 'testid=profile-actions-menu'`: menuitems Edit, Rename, Duplicate, `Set as default [disabled]`, Delete on the Default row (all enabled on other rows). `control-openhands browser eval \"document.activeElement.dataset.testid\"` is `profile-edit`: focus moves into the menu on open. Run `control-openhands browser press ArrowDown` and that eval four times: `profile-rename`, `profile-duplicate`, `profile-delete` (the disabled Set as default is skipped), `profile-edit` (wraps); `control-openhands browser press ArrowUp` twice gives `profile-delete`, `profile-duplicate`. `control-openhands browser press Tab` closes the menu (`browser count 'testid=profile-actions-menu'` is `0`) and moves focus on to the next row's trigger: the row-aware eval `control-openhands browser eval \"(()=>{const a=document.activeElement;const r=a.closest('[data-testid=profile-row]');return a.dataset.testid+'@'+(r?r.querySelector('[title]').title:'-')})()\"` is `profile-menu-trigger@QA_pro`. Reopen the `QA_flash` menu and `control-openhands browser press Escape`: count `0`, and the row-aware eval is `profile-menu-trigger@QA_flash` (focus is back on its own trigger). Reopen it and `control-openhands browser click 'role=heading[name=\"Available Profiles\"]'` closes it too. Keyboard open: `control-openhands browser focus '[data-testid=profile-row]:has([title=\"QA_pro\"]) >> testid=profile-menu-trigger'` and `control-openhands browser press Enter` put focus on `profile-edit`; on this non-default row `control-openhands browser snapshot 'testid=profile-actions-menu'` lists the same five menuitems with none `[disabled]`, and `control-openhands browser screenshot --feature F10.actions-menu --name menu-nondefault` shows them with the focus ring on Edit. Five `ArrowDown`s walk `profile-rename`, `profile-duplicate`, `profile-set-active`, `profile-delete`, `profile-edit`. `control-openhands browser press Escape` closes it, and the row-aware eval is `profile-menu-trigger@QA_pro`.", "raw": "**Row menu (`F10.actions-menu`).** Run `control-openhands browser click '[data-testid=profile-row]:has([title=\"QA_flash\"]) >> testid=profile-menu-trigger'` and `control-openhands browser snapshot 'testid=profile-actions-menu'`: menuitems Edit, Rename, Duplicate, `Set as default [disabled]`, Delete on the Default row (all enabled on other rows). `control-openhands browser eval \"document.activeElement.dataset.testid\"` is `profile-edit`: focus moves into the menu on open. Run `control-openhands browser press ArrowDown` and that eval four times: `profile-rename`, `profile-duplicate`, `profile-delete` (the disabled Set as default is skipped), `profile-edit` (wraps); `control-openhands browser press ArrowUp` twice gives `profile-delete`, `profile-duplicate`. `control-openhands browser press Tab` closes the menu (`browser count 'testid=profile-actions-menu'` is `0`) and moves focus on to the next row's trigger: the row-aware eval `control-openhands browser eval \"(()=>{const a=document.activeElement;const r=a.closest('[data-testid=profile-row]');return a.dataset.testid+'@'+(r?r.querySelector('[title]').title:'-')})()\"` is `profile-menu-trigger@QA_pro`. Reopen the `QA_flash` menu and `control-openhands browser press Escape`: count `0`, and the row-aware eval is `profile-menu-trigger@QA_flash` (focus is back on its own trigger). Reopen it and `control-openhands browser click 'role=heading[name=\"Available Profiles\"]'` closes it too. Keyboard open: `control-openhands browser focus '[data-testid=profile-row]:has([title=\"QA_pro\"]) >> testid=profile-menu-trigger'` and `control-openhands browser press Enter` put focus on `profile-edit`; on this non-default row `control-openhands browser snapshot 'testid=profile-actions-menu'` lists the same five menuitems with none `[disabled]`, and `control-openhands browser screenshot --feature F10.actions-menu --name menu-nondefault` shows them with the focus ring on Edit. Five `ArrowDown`s walk `profile-rename`, `profile-duplicate`, `profile-set-active`, `profile-delete`, `profile-edit`. `control-openhands browser press Escape` closes it, and the row-aware eval is `profile-menu-trigger@QA_pro`.", "source_line": 82, "expected": [ "the row menu offers Edit, Rename, Duplicate, Set as default (disabled on the Default row) and Delete; it focuses Edit on open (mouse or Enter), arrows cycle over the enabled items (skipping a disabled Set as default), and Escape, Tab or an outside click close it." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--set-as-default", "title": "Set as default", "ids": [ "F10.set-default" ], "body": "Run `control-openhands browser click '[data-testid=profile-row]:has([title=\"QA_pro\"]) >> testid=profile-menu-trigger'`, `control-openhands browser click 'testid=profile-actions-menu >> testid=profile-set-active'` and `control-openhands browser wait-text 'Switched to profile \"QA_pro\"'`. After `control-openhands browser reload`, `control-openhands browser count 'testid=profile-active-badge'` is `1` and `control-openhands browser count '[data-testid=profile-row]:has([title=\"QA_pro\"]) >> testid=profile-active-badge'` is `1`; `control-openhands api GET /api/profiles` has `active_profile` `QA_pro` and `control-openhands llm show` reports `active.model` `deepseek/deepseek-v4-pro`. A profile's Base URL follows it into the agent settings: `QA_adv` (Custom model above) carries one: Set as active on its row makes `control-openhands llm show` report `active.base_url` `https://api.deepseek.com`, also under `agent_settings.llm.base_url` in `control-openhands api GET /api/settings`. With a key, `control-openhands conversation start --prompt \"Reply with only: adv-ok. Do not run any tools.\" --wait --timeout 240` then shows the pill `QA_adv` and the reply `adv-ok` (driven 2026-10-08; `QA_pro` likewise replied `pro-ok` on `deepseek/deepseek-v4-pro`). Make `QA_pro` the Default again before Duplicate.", "raw": "**Set as default (`F10.set-default`).** Run `control-openhands browser click '[data-testid=profile-row]:has([title=\"QA_pro\"]) >> testid=profile-menu-trigger'`, `control-openhands browser click 'testid=profile-actions-menu >> testid=profile-set-active'` and `control-openhands browser wait-text 'Switched to profile \"QA_pro\"'`. After `control-openhands browser reload`, `control-openhands browser count 'testid=profile-active-badge'` is `1` and `control-openhands browser count '[data-testid=profile-row]:has([title=\"QA_pro\"]) >> testid=profile-active-badge'` is `1`; `control-openhands api GET /api/profiles` has `active_profile` `QA_pro` and `control-openhands llm show` reports `active.model` `deepseek/deepseek-v4-pro`. A profile's Base URL follows it into the agent settings: `QA_adv` (Custom model above) carries one: Set as active on its row makes `control-openhands llm show` report `active.base_url` `https://api.deepseek.com`, also under `agent_settings.llm.base_url` in `control-openhands api GET /api/settings`. With a key, `control-openhands conversation start --prompt \"Reply with only: adv-ok. Do not run any tools.\" --wait --timeout 240` then shows the pill `QA_adv` and the reply `adv-ok` (driven 2026-10-08; `QA_pro` likewise replied `pro-ok` on `deepseek/deepseek-v4-pro`). Make `QA_pro` the Default again before Duplicate.", "source_line": 83, "expected": [ "Set as default toasts `Switched to profile \"\"`, moves the badge, persists and becomes the agent's active LLM." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--duplicate", "title": "Duplicate", "ids": [ "F10.duplicate" ], "body": "From the `QA_pro` row menu click `testid=profile-actions-menu >> testid=profile-duplicate` and `control-openhands browser wait-text 'Profile duplicated as \"QA_pro-copy\"'`; repeat on `QA_pro` for `Profile duplicated as \"QA_pro-copy-1\"`, and on `QA_pro-copy` for `Profile duplicated as \"QA_pro-copy-copy\"`. After a reload `control-openhands browser text 'testid=profile-row >> has-text=QA_pro-copy-1'` reads `QA_pro-copy-1 deepseek/deepseek-v4-pro`, the badge stays on `QA_pro`, and `api GET /api/profiles` has `api_key_set: true` on every copy.", "raw": "**Duplicate (`F10.duplicate`).** From the `QA_pro` row menu click `testid=profile-actions-menu >> testid=profile-duplicate` and `control-openhands browser wait-text 'Profile duplicated as \"QA_pro-copy\"'`; repeat on `QA_pro` for `Profile duplicated as \"QA_pro-copy-1\"`, and on `QA_pro-copy` for `Profile duplicated as \"QA_pro-copy-copy\"`. After a reload `control-openhands browser text 'testid=profile-row >> has-text=QA_pro-copy-1'` reads `QA_pro-copy-1 deepseek/deepseek-v4-pro`, the badge stays on `QA_pro`, and `api GET /api/profiles` has `api_key_set: true` on every copy.", "source_line": 84, "expected": [ "Duplicate saves `-copy`, then `-copy-1`, with the same model and stored key; the copy is not Default." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--rename", "title": "Rename", "ids": [ "F10.rename" ], "body": "Open the `QA_pro-copy-copy` row menu and click `testid=profile-actions-menu >> testid=profile-rename`. `control-openhands browser value 'testid=rename-profile-input'` is `QA_pro-copy-copy` and `browser eval \"document.activeElement.dataset.testid\"` is `rename-profile-input`. `control-openhands browser fill 'testid=rename-profile-input' '-bad'` makes `control-openhands browser enabled 'role=dialog[name=\"Rename Profile\"] >> role=button[name=\"Rename\"]'` `false` and `control-openhands browser attr 'testid=rename-profile-rule' class` contains `text-danger` (the theme danger colour, #18199). Filling the old name and clicking Rename closes the modal with no toast (`browser count 'testid=rename-profile-modal'` is `0`); `control-openhands browser click 'role=dialog[name=\"Rename Profile\"] >> role=button[name=\"Cancel\"]'` also closes it. To rename, reopen, fill `QA_renamed`, `control-openhands browser press Enter`, `control-openhands browser wait-text 'Profile renamed to \"QA_renamed\"'`, reload, and the count of `'[data-testid=profile-row]:has([title=\"QA_renamed\"])'` is `1` while the old name's is `0`. Renaming the Default row (`QA_pro` → `QA_pro2`, submit with `testid=rename-profile-submit`) keeps its badge (`browser text '[data-testid=profile-row]:has([data-testid=profile-active-badge])'` reads `QA_pro2 … Default`) and `active_profile` follows.", "raw": "**Rename (`F10.rename`).** Open the `QA_pro-copy-copy` row menu and click `testid=profile-actions-menu >> testid=profile-rename`. `control-openhands browser value 'testid=rename-profile-input'` is `QA_pro-copy-copy` and `browser eval \"document.activeElement.dataset.testid\"` is `rename-profile-input`. `control-openhands browser fill 'testid=rename-profile-input' '-bad'` makes `control-openhands browser enabled 'role=dialog[name=\"Rename Profile\"] >> role=button[name=\"Rename\"]'` `false` and `control-openhands browser attr 'testid=rename-profile-rule' class` contains `text-danger` (the theme danger colour, #18199). Filling the old name and clicking Rename closes the modal with no toast (`browser count 'testid=rename-profile-modal'` is `0`); `control-openhands browser click 'role=dialog[name=\"Rename Profile\"] >> role=button[name=\"Cancel\"]'` also closes it. To rename, reopen, fill `QA_renamed`, `control-openhands browser press Enter`, `control-openhands browser wait-text 'Profile renamed to \"QA_renamed\"'`, reload, and the count of `'[data-testid=profile-row]:has([title=\"QA_renamed\"])'` is `1` while the old name's is `0`. Renaming the Default row (`QA_pro` → `QA_pro2`, submit with `testid=rename-profile-submit`) keeps its badge (`browser text '[data-testid=profile-row]:has([data-testid=profile-active-badge])'` reads `QA_pro2 … Default`) and `active_profile` follows.", "source_line": 85, "expected": [ "the Rename modal is prefilled and focused; invalid names disable Rename; an unchanged name or Cancel just closes; Enter submits; success toasts and persists, and the Default badge follows a renamed default." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--rename-onto-an-existing-name", "title": "Rename onto an existing name", "ids": [ "F10.rename-conflict" ], "body": "The Rename step above already renamed `QA_pro-copy-copy` and `QA_pro`, so use the remaining fixtures: open the `QA_pro-copy` row menu, click `testid=profile-actions-menu >> testid=profile-rename`, fill `testid=rename-profile-input` with `QA_pro2` and click `testid=rename-profile-submit`. The modal stays open (`count 'testid=rename-profile-modal'` is `1`), `control-openhands browser toasts` has the server's message `Profile 'QA_pro2' already exists` (no `HTTP request failed` prefix, no JSON), and `browser screenshot --feature F10.rename-conflict --name toast` shows that toast above the open modal. Close it with `control-openhands browser click 'role=dialog[name=\"Rename Profile\"] >> role=button[name=\"Cancel\"]'`.", "raw": "**Rename onto an existing name (`F10.rename-conflict`).** The Rename step above already renamed `QA_pro-copy-copy` and `QA_pro`, so use the remaining fixtures: open the `QA_pro-copy` row menu, click `testid=profile-actions-menu >> testid=profile-rename`, fill `testid=rename-profile-input` with `QA_pro2` and click `testid=rename-profile-submit`. The modal stays open (`count 'testid=rename-profile-modal'` is `1`), `control-openhands browser toasts` has the server's message `Profile 'QA_pro2' already exists` (no `HTTP request failed` prefix, no JSON), and `browser screenshot --feature F10.rename-conflict --name toast` shows that toast above the open modal. Close it with `control-openhands browser click 'role=dialog[name=\"Rename Profile\"] >> role=button[name=\"Cancel\"]'`.", "source_line": 86, "expected": [ "renaming onto an existing name is refused with the server's message (`Profile '' already exists`) and the modal stays open." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--edit", "title": "Edit", "ids": [ "F10.edit" ], "body": "Open the `QA_renamed` row menu and click `testid=profile-actions-menu >> testid=profile-edit`. `browser text 'testid=profile-editor-title'` is `Edit LLM Profile`, `browser text 'testid=profile-editor-description'` is `Editing profile \"QA_renamed\" - save to apply changes`, `browser attr 'testid=llm-api-key-input' placeholder` is ``, `browser count 'testid=set-indicator'` is `1` and `browser enabled 'testid=save-profile-btn'` is `false` (screenshot `--feature F10.edit --name loaded`). Filling another profile's name (`QA_flash`) sets `aria-invalid` `true` with the taken-name message of `F10.name-validation` and keeps Save disabled; refilling `QA_renamed` clears it. Run `control-openhands browser click 'testid=llm-model-input'`, `control-openhands browser click 'testid=model-item-deepseek-flash'` (Save turns `true`), `control-openhands browser fill 'testid=profile-name-input' QA_edited`, Save, and `control-openhands browser wait-text 'Profile \"QA_edited\" updated' --timeout 60000`. After a reload `browser text '[data-testid=profile-row]:has([title=\"QA_edited\"])'` reads `QA_edited deepseek/deepseek-flash` and `QA_renamed` is gone. The save passed validation without retyping the key, which proves the stored key was kept.", "raw": "**Edit (`F10.edit`).** Open the `QA_renamed` row menu and click `testid=profile-actions-menu >> testid=profile-edit`. `browser text 'testid=profile-editor-title'` is `Edit LLM Profile`, `browser text 'testid=profile-editor-description'` is `Editing profile \"QA_renamed\" - save to apply changes`, `browser attr 'testid=llm-api-key-input' placeholder` is ``, `browser count 'testid=set-indicator'` is `1` and `browser enabled 'testid=save-profile-btn'` is `false` (screenshot `--feature F10.edit --name loaded`). Filling another profile's name (`QA_flash`) sets `aria-invalid` `true` with the taken-name message of `F10.name-validation` and keeps Save disabled; refilling `QA_renamed` clears it. Run `control-openhands browser click 'testid=llm-model-input'`, `control-openhands browser click 'testid=model-item-deepseek-flash'` (Save turns `true`), `control-openhands browser fill 'testid=profile-name-input' QA_edited`, Save, and `control-openhands browser wait-text 'Profile \"QA_edited\" updated' --timeout 60000`. After a reload `browser text '[data-testid=profile-row]:has([title=\"QA_edited\"])'` reads `QA_edited deepseek/deepseek-flash` and `QA_renamed` is gone. The save passed validation without retyping the key, which proves the stored key was kept.", "source_line": 87, "expected": [ "Edit loads the profile (`Editing profile \"\" - save to apply changes`) with the stored key masked; Save stays disabled until something changes; a changed name renames before saving; the stored key is kept." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5890](https://github.com/OpenHands/software-agent-sdk/issues/5890): Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (`F10.edit`, `F10.default-used`).", "Known issue [#17806](https://github.com/OpenHands/OpenHands/issues/17806): A model-only Save must preserve an untouched key. Current SDK1.55 refused-loopback validation rejects Save before persistence; the stored key marker remains set. Completing Basic/Advanced preserved-key and runtime-switch proof needs an approved route that accepts preflight. Keep dummy traffic on refused loopback until an external route is authorized; a rejected Save does not establish either key loss or successful preservation through Save. (`F10.edit`)." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F10--basic-re-save-keeps-a-hidden-base-url", "title": "Basic re-save keeps a hidden Base URL", "ids": [ "F10.basic-save-keeps-base-url" ], "body": "`QA_adv` carries `base_url` `https://api.deepseek.com` (Custom model above). Open its row menu, click `testid=profile-actions-menu >> testid=profile-edit`, then `control-openhands browser click 'testid=sdk-section-advanced-toggle'`: `control-openhands browser value 'testid=base-url-input'` is `https://api.deepseek.com`. Run `control-openhands browser click 'testid=sdk-section-basic-toggle'`: `control-openhands browser count 'testid=base-url-input'` is `0` (the view hides it), `control-openhands browser value 'testid=llm-model-input'` is `deepseek-flash` and `control-openhands browser enabled 'testid=save-profile-btn'` is `false` (`control-openhands browser screenshot --feature F10.basic-save-keeps-base-url --name basic-view`). Without touching the model, run `control-openhands browser fill 'testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY` (Save turns `true`), `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 4000` (`Save`, `Validating...`, `Saving...`) and `control-openhands browser wait-text 'Profile \"QA_adv\" updated' --timeout 60000`. Then `control-openhands api GET /api/profiles/QA_adv --pick config.base_url` is still `https://api.deepseek.com` and `control-openhands api GET /api/profiles/QA_adv --pick config.model` is `deepseek/deepseek-flash`; after `control-openhands browser reload` the same GETs agree. Without a key, arrange `QA_DUMMY_KEY=qa control-openhands llm set --profile QA_base --model deepseek/deepseek-chat --api-key-env QA_DUMMY_KEY --base-url https://api.deepseek.com --no-validate --no-activate` (it prints `activated` `false` and `control-openhands api GET /api/profiles --pick active_profile` is unchanged: the Default stays where it is and the editors below inherit it, as long as some profile is Default; on a run where `QA_base` is the only profile the page makes it Default at the next load, see Gotchas) and `control-openhands browser reload`, then drive the same steps on `QA_base`'s row with its own readbacks: its editor opens on Advanced (the Base URL is a non-default value) with `control-openhands browser value 'testid=base-url-input'` `https://api.deepseek.com`, Basic hides it (`count` `0`), `control-openhands browser value 'testid=llm-model-input'` is `deepseek-chat` and `control-openhands browser enabled 'testid=save-profile-btn'` is `false`; fill the key with `QA_DUMMY_KEY=qa-dummy-key control-openhands browser fill 'testid=llm-api-key-input' --value-env QA_DUMMY_KEY` (Save turns `true`) and Save: the observed states are `Save`, `Validating...` (`[disabled] [busy]`), `Save` within a second, `control-openhands browser toasts` has the provider's message `litellm.BadRequestError: DeepseekException - {\"error\":{\"message\":\"Authentication Fails, Your api key: ****-key is invalid …`, `browser text 'testid=profile-editor-title'` is still `Edit LLM Profile`, `control-openhands api GET /api/profiles/QA_base --pick config.base_url` is still `https://api.deepseek.com` and `--pick config.model` is `deepseek/deepseek-chat`; record `blocked` with `DEEPSEEK_API_KEY` as the prerequisite, leave with `control-openhands browser click 'testid=cancel-profile-btn'` and remove the arranged profile with `control-openhands api DELETE /api/profiles/QA_base --write` (arrange, not proof).", "raw": "**Basic re-save keeps a hidden Base URL (`F10.basic-save-keeps-base-url`).** `QA_adv` carries `base_url` `https://api.deepseek.com` (Custom model above). Open its row menu, click `testid=profile-actions-menu >> testid=profile-edit`, then `control-openhands browser click 'testid=sdk-section-advanced-toggle'`: `control-openhands browser value 'testid=base-url-input'` is `https://api.deepseek.com`. Run `control-openhands browser click 'testid=sdk-section-basic-toggle'`: `control-openhands browser count 'testid=base-url-input'` is `0` (the view hides it), `control-openhands browser value 'testid=llm-model-input'` is `deepseek-flash` and `control-openhands browser enabled 'testid=save-profile-btn'` is `false` (`control-openhands browser screenshot --feature F10.basic-save-keeps-base-url --name basic-view`). Without touching the model, run `control-openhands browser fill 'testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY` (Save turns `true`), `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 4000` (`Save`, `Validating...`, `Saving...`) and `control-openhands browser wait-text 'Profile \"QA_adv\" updated' --timeout 60000`. Then `control-openhands api GET /api/profiles/QA_adv --pick config.base_url` is still `https://api.deepseek.com` and `control-openhands api GET /api/profiles/QA_adv --pick config.model` is `deepseek/deepseek-flash`; after `control-openhands browser reload` the same GETs agree. Without a key, arrange `QA_DUMMY_KEY=qa control-openhands llm set --profile QA_base --model deepseek/deepseek-chat --api-key-env QA_DUMMY_KEY --base-url https://api.deepseek.com --no-validate --no-activate` (it prints `activated` `false` and `control-openhands api GET /api/profiles --pick active_profile` is unchanged: the Default stays where it is and the editors below inherit it, as long as some profile is Default; on a run where `QA_base` is the only profile the page makes it Default at the next load, see Gotchas) and `control-openhands browser reload`, then drive the same steps on `QA_base`'s row with its own readbacks: its editor opens on Advanced (the Base URL is a non-default value) with `control-openhands browser value 'testid=base-url-input'` `https://api.deepseek.com`, Basic hides it (`count` `0`), `control-openhands browser value 'testid=llm-model-input'` is `deepseek-chat` and `control-openhands browser enabled 'testid=save-profile-btn'` is `false`; fill the key with `QA_DUMMY_KEY=qa-dummy-key control-openhands browser fill 'testid=llm-api-key-input' --value-env QA_DUMMY_KEY` (Save turns `true`) and Save: the observed states are `Save`, `Validating...` (`[disabled] [busy]`), `Save` within a second, `control-openhands browser toasts` has the provider's message `litellm.BadRequestError: DeepseekException - {\"error\":{\"message\":\"Authentication Fails, Your api key: ****-key is invalid …`, `browser text 'testid=profile-editor-title'` is still `Edit LLM Profile`, `control-openhands api GET /api/profiles/QA_base --pick config.base_url` is still `https://api.deepseek.com` and `--pick config.model` is `deepseek/deepseek-chat`; record `blocked` with `DEEPSEEK_API_KEY` as the prerequisite, leave with `control-openhands browser click 'testid=cancel-profile-btn'` and remove the arranged profile with `control-openhands api DELETE /api/profiles/QA_base --write` (arrange, not proof).", "source_line": 88, "expected": [ "re-saving a profile from the Basic view without changing its model keeps the Base URL that view hides." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--leave-without-saving", "title": "Leave without saving", "ids": [ "F10.editor-discard" ], "body": "Edit `QA_adv`, `control-openhands browser click 'testid=sdk-section-all-toggle'`, `control-openhands browser fill 'testid=sdk-settings-llm.top_p' 0.5` (Save turns enabled), then `control-openhands browser click 'testid=back-to-profiles'`. The list returns at once (`browser count 'testid=add-llm-profile'` is `1`) with no prompt (`control-openhands browser events --kinds dialog` is empty), and reopening Edit → All shows `top_p` empty. `testid=cancel-profile-btn` behaves the same.", "raw": "**Leave without saving (`F10.editor-discard`).** Edit `QA_adv`, `control-openhands browser click 'testid=sdk-section-all-toggle'`, `control-openhands browser fill 'testid=sdk-settings-llm.top_p' 0.5` (Save turns enabled), then `control-openhands browser click 'testid=back-to-profiles'`. The list returns at once (`browser count 'testid=add-llm-profile'` is `1`) with no prompt (`control-openhands browser events --kinds dialog` is empty), and reopening Edit → All shows `top_p` empty. `testid=cancel-profile-btn` behaves the same.", "source_line": 89, "expected": [ "Back and Cancel return to the list without saving and without an unsaved-changes prompt." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--schema-field-validation-and-re-apply", "title": "Schema field validation and re-apply", "ids": [ "F10.schema-validation", "F10.edit-default-reapply" ], "body": "Edit the Default profile and run `control-openhands browser click 'testid=sdk-section-all-toggle'`. `control-openhands browser fill 'testid=sdk-settings-llm.temperature' 5`, Save, then `control-openhands browser wait-text 'must be at most 2'` (toast `Temperature must be at most 2`). Clear it, fill `testid=sdk-settings-llm.top_p` with `3`: `Top P must be at most 1`. Clear it, fill `testid=sdk-settings-llm.extra_headers` with `{bad`: `Invalid JSON for Extra Headers`. Clear that, fill temperature `0.3`, Save, and `wait-text 'Profile \"\" updated'`. `control-openhands api GET /api/settings` then has `agent_settings.llm.temperature` `0.3`, and reopening Edit → All shows `0.3`.", "raw": "**Schema field validation and re-apply (`F10.schema-validation`, `F10.edit-default-reapply`).** Edit the Default profile and run `control-openhands browser click 'testid=sdk-section-all-toggle'`. `control-openhands browser fill 'testid=sdk-settings-llm.temperature' 5`, Save, then `control-openhands browser wait-text 'must be at most 2'` (toast `Temperature must be at most 2`). Clear it, fill `testid=sdk-settings-llm.top_p` with `3`: `Top P must be at most 1`. Clear it, fill `testid=sdk-settings-llm.extra_headers` with `{bad`: `Invalid JSON for Extra Headers`. Clear that, fill temperature `0.3`, Save, and `wait-text 'Profile \"\" updated'`. `control-openhands api GET /api/settings` then has `agent_settings.llm.temperature` `0.3`, and reopening Edit → All shows `0.3`.", "source_line": 90, "expected": [ "bad values in schema fields (Temperature over 2, Top P over 1, invalid JSON) are refused with a toast naming the field.", "saving the Default profile re-applies it to the agent's LLM settings." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5181](https://github.com/OpenHands/software-agent-sdk/issues/5181): The SDK can turn an advertised output limit into a requested default cap and silently drop an explicitly enabled reasoning_effort. The narrowed maintainer scope needs an authenticated OpenRouter/proxy route exposing lower-limit endpoints and capability metadata, or a real endpoint accepting the explicit override; a synthetic empty-metadata response would not establish production behavior. (`F10.schema-validation`).", "Known issue [OpenHands/software-agent-sdk#4934](https://github.com/OpenHands/software-agent-sdk/issues/4934): Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (`F10.schema-validation`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#4511](https://github.com/OpenHands/software-agent-sdk/issues/4511): Disabling prompt caching can still emit cache_control on preexisting cache-marked tool messages. The report only supplies a Python serializer reproduction; a Canvas check needs the exact persisted/tool-message precondition and a provider request that exposes the cache markers, with approved model access. (`F10.schema-validation`)." ], "annotations": [], "states": [ "issue" ] }, { "name": "F10--new-profile-prefill", "title": "New profile prefill", "ids": [ "F10.create-prefill" ], "body": "With that Default (temperature `0.3`) run `control-openhands browser click 'testid=add-llm-profile'`. The editor opens on All (`browser eval \"[...document.querySelectorAll('[role=tab]')].map(t=>t.innerText+':'+t.getAttribute('aria-selected')).join(',')\"` is `Basic:false,Advanced:false,All:true`), `browser value 'testid=llm-custom-model-input'` is the backend default `openai/gpt-5.6-sol`, and `control-openhands assert --field value --equals 0.3 -- browser value 'testid=sdk-settings-llm.temperature'` passes: the temperature is inherited from the Default. Run `control-openhands browser fill 'testid=llm-custom-model-input' deepseek/deepseek-flash`, `browser fill 'testid=profile-name-input' QA_inherit`, fill the key as in Create, `control-openhands browser scroll 'testid=sdk-settings-llm.temperature'`, `control-openhands browser screenshot --feature F10.create-prefill --name inherited-before-save`, Save and `wait-text 'Profile \"QA_inherit\" created' --timeout 60000`. The saved profile has the temperature the form showed: `control-openhands api GET /api/profiles/QA_inherit --pick config.temperature` is `0.3`. After `browser reload`, open `QA_inherit`'s row menu (`'[data-testid=profile-row]:has([title=\"QA_inherit\"]) >> testid=profile-menu-trigger'`), click `testid=profile-actions-menu >> testid=profile-edit` and `testid=sdk-section-all-toggle`, `control-openhands browser scroll 'testid=sdk-settings-llm.temperature'` and `control-openhands assert --field value --equals 0.3 -- browser value 'testid=sdk-settings-llm.temperature'` passes (screenshot `--feature F10.create-prefill --name edit-after-create`); leave with `testid=cancel-profile-btn`. Repeat at `browser viewport phone` after `control-openhands api DELETE /api/profiles/QA_inherit --write` (the desktop pass already took the name); the same holds.", "raw": "**New profile prefill (`F10.create-prefill`).** With that Default (temperature `0.3`) run `control-openhands browser click 'testid=add-llm-profile'`. The editor opens on All (`browser eval \"[...document.querySelectorAll('[role=tab]')].map(t=>t.innerText+':'+t.getAttribute('aria-selected')).join(',')\"` is `Basic:false,Advanced:false,All:true`), `browser value 'testid=llm-custom-model-input'` is the backend default `openai/gpt-5.6-sol`, and `control-openhands assert --field value --equals 0.3 -- browser value 'testid=sdk-settings-llm.temperature'` passes: the temperature is inherited from the Default. Run `control-openhands browser fill 'testid=llm-custom-model-input' deepseek/deepseek-flash`, `browser fill 'testid=profile-name-input' QA_inherit`, fill the key as in Create, `control-openhands browser scroll 'testid=sdk-settings-llm.temperature'`, `control-openhands browser screenshot --feature F10.create-prefill --name inherited-before-save`, Save and `wait-text 'Profile \"QA_inherit\" created' --timeout 60000`. The saved profile has the temperature the form showed: `control-openhands api GET /api/profiles/QA_inherit --pick config.temperature` is `0.3`. After `browser reload`, open `QA_inherit`'s row menu (`'[data-testid=profile-row]:has([title=\"QA_inherit\"]) >> testid=profile-menu-trigger'`), click `testid=profile-actions-menu >> testid=profile-edit` and `testid=sdk-section-all-toggle`, `control-openhands browser scroll 'testid=sdk-settings-llm.temperature'` and `control-openhands assert --field value --equals 0.3 -- browser value 'testid=sdk-settings-llm.temperature'` passes (screenshot `--feature F10.create-prefill --name edit-after-create`); leave with `testid=cancel-profile-btn`. Repeat at `browser viewport phone` after `control-openhands api DELETE /api/profiles/QA_inherit --write` (the desktop pass already took the name); the same holds.", "source_line": 91, "expected": [ "a new editor prefills the backend default model with a blank key, base URL and connection; the other LLM options it shows come from the agent's current LLM settings, i.e. the last applied Default profile (the editor opens on the tab they need), and Save stores what the form shows." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--link-to-a-provider-connection", "title": "Link to a provider connection", "ids": [ "F10.connection-link", "F10.list-grouped" ], "body": "Arrange a connection through the F11 form: `control-openhands browser click 'testid=add-provider-connection'`, `control-openhands browser fill 'testid=provider-connection-name-input' QA_ds_conn`, `control-openhands browser click 'testid=provider-connection-provider-input'`, `control-openhands browser fill 'testid=provider-connection-provider-input' deepseek`, `control-openhands browser click 'testid=provider-item-deepseek'`, `control-openhands browser fill 'testid=provider-connection-api-key-input' --value-file `, `control-openhands browser click 'testid=provider-connection-submit'` and `control-openhands browser wait-text 'Connection \"QA_ds_conn\" created'`. Edit `QA_adv`, click `testid=sdk-section-basic-toggle`, run `control-openhands browser click 'testid=llm-provider-connection-input'` (options `None`, `QA_ds_conn`) and `control-openhands browser click 'role=option[name=\"QA_ds_conn\"]'`; `browser count 'testid=llm-api-key-input'` is `0`, and on the Advanced tab `browser count 'testid=base-url-input'` is `0`. Picking `role=option[name=\"None\"]` brings the key field back (`1`). Re-select `QA_ds_conn`, then `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 3000`: states `Save`, `Saving...` (no `Validating...`), and `wait-text 'Profile \"QA_adv\" updated'`. After a reload `control-openhands browser eval \"[...document.querySelectorAll('[data-testid=profile-group-header]')].map(e=>e.innerText).join(' | ')\"` is `QA_DS_CONN | NOT LINKED` (CSS upper-case) and `control-openhands browser count 'div:has(> [data-testid=profile-group-header]:text-is(\"QA_ds_conn\")) [data-testid=profile-row]:has([title=\"QA_adv\"])'` is `1` (screenshot `--feature F10.list-grouped --name grouped`).", "raw": "**Link to a provider connection (`F10.connection-link`, `F10.list-grouped`).** Arrange a connection through the F11 form: `control-openhands browser click 'testid=add-provider-connection'`, `control-openhands browser fill 'testid=provider-connection-name-input' QA_ds_conn`, `control-openhands browser click 'testid=provider-connection-provider-input'`, `control-openhands browser fill 'testid=provider-connection-provider-input' deepseek`, `control-openhands browser click 'testid=provider-item-deepseek'`, `control-openhands browser fill 'testid=provider-connection-api-key-input' --value-file `, `control-openhands browser click 'testid=provider-connection-submit'` and `control-openhands browser wait-text 'Connection \"QA_ds_conn\" created'`. Edit `QA_adv`, click `testid=sdk-section-basic-toggle`, run `control-openhands browser click 'testid=llm-provider-connection-input'` (options `None`, `QA_ds_conn`) and `control-openhands browser click 'role=option[name=\"QA_ds_conn\"]'`; `browser count 'testid=llm-api-key-input'` is `0`, and on the Advanced tab `browser count 'testid=base-url-input'` is `0`. Picking `role=option[name=\"None\"]` brings the key field back (`1`). Re-select `QA_ds_conn`, then `control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 3000`: states `Save`, `Saving...` (no `Validating...`), and `wait-text 'Profile \"QA_adv\" updated'`. After a reload `control-openhands browser eval \"[...document.querySelectorAll('[data-testid=profile-group-header]')].map(e=>e.innerText).join(' | ')\"` is `QA_DS_CONN | NOT LINKED` (CSS upper-case) and `control-openhands browser count 'div:has(> [data-testid=profile-group-header]:text-is(\"QA_ds_conn\")) [data-testid=profile-row]:has([title=\"QA_adv\"])'` is `1` (screenshot `--feature F10.list-grouped --name grouped`).", "source_line": 92, "expected": [ "with connections present the editor shows a Provider connection selector; linking hides API key and Base URL, skips validation on Save and regroups the row; None restores the key field.", "once any profile links to a provider connection, rows group under the connection's name, with unlinked rows last under `Not linked`; with no linked rows the list is flat again." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--add-models-from-a-provider", "title": "Add models from a provider", "ids": [ "F10.add-models" ], "body": "Run `control-openhands browser click 'testid=add-models-from-provider'`; `browser enabled 'testid=add-models-submit'` is `false`. Keyless: `control-openhands browser select 'testid=add-models-provider' moonshot` and `control-openhands browser wait 'testid=add-models-select-all'`; `browser text 'testid=add-models-keyless-note'` is `No connection — profiles will need a key added later`, rows start unchecked (`Add 0 profiles`), and `control-openhands browser check 'testid=add-models-select-all'` makes `browser text 'testid=add-models-submit'` read `Add 12 profiles`; uncheck it again. Linked: `control-openhands browser select 'testid=add-models-provider' deepseek`; the connection field (`testid=add-models-connection-field`, options `No connection (keyless)` and `QA_ds_conn`) is pre-bound to `QA_ds_conn`. Run `control-openhands browser check 'testid=add-models-check-deepseek/deepseek-chat'`, `control-openhands browser check 'testid=add-models-check-deepseek/deepseek-v3'`, `control-openhands browser click 'testid=add-models-submit'` and `control-openhands browser wait-text 'Added 2'` (toast `Added 2 profiles`). The modal closes; after a reload `deepseek-chat` and `deepseek-v3` sit under the `QA_DS_CONN` header and `api GET /api/profiles` shows both with a `provider_connection_id`. Preselect entry: `control-openhands browser click 'testid=provider-connection-row >> has-text=QA_ds_conn >> testid=provider-connection-menu-trigger'` (menu: Bulk add, Edit, Delete), `control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-add-models'`; provider `deepseek` and the connection are preset, every remaining row is checked (`Add 10 profiles`), and `browser count 'testid=add-models-row-deepseek/deepseek-chat'` is `0` (already added). Close with `control-openhands browser click 'role=dialog[name=\"Add models as profiles\"] >> role=button[name=\"Cancel\"]'`.", "raw": "**Add models from a provider (`F10.add-models`).** Run `control-openhands browser click 'testid=add-models-from-provider'`; `browser enabled 'testid=add-models-submit'` is `false`. Keyless: `control-openhands browser select 'testid=add-models-provider' moonshot` and `control-openhands browser wait 'testid=add-models-select-all'`; `browser text 'testid=add-models-keyless-note'` is `No connection — profiles will need a key added later`, rows start unchecked (`Add 0 profiles`), and `control-openhands browser check 'testid=add-models-select-all'` makes `browser text 'testid=add-models-submit'` read `Add 12 profiles`; uncheck it again. Linked: `control-openhands browser select 'testid=add-models-provider' deepseek`; the connection field (`testid=add-models-connection-field`, options `No connection (keyless)` and `QA_ds_conn`) is pre-bound to `QA_ds_conn`. Run `control-openhands browser check 'testid=add-models-check-deepseek/deepseek-chat'`, `control-openhands browser check 'testid=add-models-check-deepseek/deepseek-v3'`, `control-openhands browser click 'testid=add-models-submit'` and `control-openhands browser wait-text 'Added 2'` (toast `Added 2 profiles`). The modal closes; after a reload `deepseek-chat` and `deepseek-v3` sit under the `QA_DS_CONN` header and `api GET /api/profiles` shows both with a `provider_connection_id`. Preselect entry: `control-openhands browser click 'testid=provider-connection-row >> has-text=QA_ds_conn >> testid=provider-connection-menu-trigger'` (menu: Bulk add, Edit, Delete), `control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-add-models'`; provider `deepseek` and the connection are preset, every remaining row is checked (`Add 10 profiles`), and `browser count 'testid=add-models-row-deepseek/deepseek-chat'` is `0` (already added). Close with `control-openhands browser click 'role=dialog[name=\"Add models as profiles\"] >> role=button[name=\"Cancel\"]'`.", "source_line": 93, "expected": [ "\"Add from provider connections\" (or a connection row's **Bulk add**) creates one linked or keyless profile per checked model; models that already have a profile of that name are hidden." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--broken-link", "title": "Broken link", "ids": [ "F10.broken-link" ], "body": "The local server refuses to delete a referenced connection (row menu → `testid=provider-connection-delete` → `testid=delete-provider-connection-confirm` toasts `Provider connection cannot be deleted while it is referenced by LLM profile(s): ...`), so arrange the state directly: `control-openhands api POST /api/profiles/QA_broken --write --data '{\"llm\":{\"model\":\"deepseek/deepseek-chat\",\"provider_connection_id\":\"00000000000000000000000000000000\"},\"include_secrets\":false}'`. Then `control-openhands browser reload` and `control-openhands browser text '[data-testid=profile-row]:has([title=\"QA_broken\"])'` reads `QA_broken deepseek/deepseek-chat Broken link`; `control-openhands browser attr '[data-testid=profile-row]:has([title=\"QA_broken\"]) >> testid=profile-broken-connection-badge' title` is `The linked provider connection was deleted. Edit this profile to re-link it.` (screenshot `--feature F10.broken-link --name badge`). Edit it: `browser value 'testid=llm-provider-connection-input'` is the orphaned id and the listbox offers `None`, `QA_ds_conn` and that id. The row's group header shows the raw id, not a name (see Gotchas).", "raw": "**Broken link (`F10.broken-link`).** The local server refuses to delete a referenced connection (row menu → `testid=provider-connection-delete` → `testid=delete-provider-connection-confirm` toasts `Provider connection cannot be deleted while it is referenced by LLM profile(s): ...`), so arrange the state directly: `control-openhands api POST /api/profiles/QA_broken --write --data '{\"llm\":{\"model\":\"deepseek/deepseek-chat\",\"provider_connection_id\":\"00000000000000000000000000000000\"},\"include_secrets\":false}'`. Then `control-openhands browser reload` and `control-openhands browser text '[data-testid=profile-row]:has([title=\"QA_broken\"])'` reads `QA_broken deepseek/deepseek-chat Broken link`; `control-openhands browser attr '[data-testid=profile-row]:has([title=\"QA_broken\"]) >> testid=profile-broken-connection-badge' title` is `The linked provider connection was deleted. Edit this profile to re-link it.` (screenshot `--feature F10.broken-link --name badge`). Edit it: `browser value 'testid=llm-provider-connection-input'` is the orphaned id and the listbox offers `None`, `QA_ds_conn` and that id. The row's group header shows the raw id, not a name (see Gotchas).", "source_line": 94, "expected": [ "a profile whose linked connection no longer exists shows a warning `Broken link` badge whose tooltip says to edit the profile to re-link it; its editor offers the orphaned id in the Provider connection selector so it can be cleared." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--chatgpt-subscription-pre-auth", "title": "ChatGPT subscription, pre-auth", "ids": [ "F10.subscription" ], "body": "In a fresh editor run `control-openhands browser click 'testid=sdk-section-basic-toggle'` first (after the schema step the Default profile has a temperature, so a new editor opens on All; see `F10.create-prefill`), then `control-openhands browser click 'testid=llm-auth-type-input'` (options `API key`, `ChatGPT subscription`) and `control-openhands browser click 'role=option[name=\"ChatGPT subscription\"]'`. `browser text 'testid=subscription-status'` is `ChatGPT subscription not connected`, `browser value 'testid=llm-subscription-model-input'` is `gpt-5.5` (list: gpt-5.5, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-6-astra), and `llm-provider-input` and `llm-api-key-input` are gone (count `0`). Known failure (reproduced 2026-10-13, #17135): run `control-openhands browser fill 'testid=profile-name-input' QA_sub`, `control-openhands browser network --clear`, `control-openhands browser click 'testid=save-profile-btn'` and `control-openhands browser network --bodies --filter '/api/profiles'`. Expected: Save validates and saves the shown `gpt-5.5`. Actual: one `POST /api/profiles/QA_sub/validate` whose body has `\"model\":\"openai/gpt-5.6-sol\"`, the API-key form's hidden default, although the dropdown shows `gpt-5.5` (it shows the first subscription model without writing it to the form); `browser toasts --history` has `OpenAI subscription login is required`, because no subscription is connected. Picking a model in `testid=llm-subscription-model-input`, even `gpt-5.5` again, makes the next Save send `POST /api/profiles/QA_sub/validate`, which answers `OpenAI subscription login is required` here (`browser toasts --history`), because no subscription is connected. `control-openhands browser click 'testid=subscription-connect'` and `control-openhands browser wait 'testid=subscription-user-code'`: the card shows the code, `Open sign-in page`, `I've finished signing in` (`subscription-poll`) and Cancel, and `control-openhands browser tabs` lists a second page on `auth.openai.com` (it opens a second or two after the code appears; rerun `browser tabs` if it lists one page). The card polls on its own, so `Sign-in is not complete yet. ...` can already show before you click poll. Close it with `control-openhands browser close-tab 1` and `control-openhands browser tab 0`. `control-openhands browser click 'testid=subscription-poll'` leaves `Sign-in is not complete yet. Finish the browser flow and try again.` on the card, and `control-openhands browser click 'testid=subscription-cancel'` removes the code (`count 'testid=subscription-user-code'` `0`). Switching back with `role=option[name=\"API key\"]` restores provider `OpenAI` and model `gpt-5.6-sol`. Completing sign-in is blocked (needs a ChatGPT Plus/Pro account).", "raw": "**ChatGPT subscription, pre-auth (`F10.subscription`).** In a fresh editor run `control-openhands browser click 'testid=sdk-section-basic-toggle'` first (after the schema step the Default profile has a temperature, so a new editor opens on All; see `F10.create-prefill`), then `control-openhands browser click 'testid=llm-auth-type-input'` (options `API key`, `ChatGPT subscription`) and `control-openhands browser click 'role=option[name=\"ChatGPT subscription\"]'`. `browser text 'testid=subscription-status'` is `ChatGPT subscription not connected`, `browser value 'testid=llm-subscription-model-input'` is `gpt-5.5` (list: gpt-5.5, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-6-astra), and `llm-provider-input` and `llm-api-key-input` are gone (count `0`). Known failure (reproduced 2026-10-13, #17135): run `control-openhands browser fill 'testid=profile-name-input' QA_sub`, `control-openhands browser network --clear`, `control-openhands browser click 'testid=save-profile-btn'` and `control-openhands browser network --bodies --filter '/api/profiles'`. Expected: Save validates and saves the shown `gpt-5.5`. Actual: one `POST /api/profiles/QA_sub/validate` whose body has `\"model\":\"openai/gpt-5.6-sol\"`, the API-key form's hidden default, although the dropdown shows `gpt-5.5` (it shows the first subscription model without writing it to the form); `browser toasts --history` has `OpenAI subscription login is required`, because no subscription is connected. Picking a model in `testid=llm-subscription-model-input`, even `gpt-5.5` again, makes the next Save send `POST /api/profiles/QA_sub/validate`, which answers `OpenAI subscription login is required` here (`browser toasts --history`), because no subscription is connected. `control-openhands browser click 'testid=subscription-connect'` and `control-openhands browser wait 'testid=subscription-user-code'`: the card shows the code, `Open sign-in page`, `I've finished signing in` (`subscription-poll`) and Cancel, and `control-openhands browser tabs` lists a second page on `auth.openai.com` (it opens a second or two after the code appears; rerun `browser tabs` if it lists one page). The card polls on its own, so `Sign-in is not complete yet. ...` can already show before you click poll. Close it with `control-openhands browser close-tab 1` and `control-openhands browser tab 0`. `control-openhands browser click 'testid=subscription-poll'` leaves `Sign-in is not complete yet. Finish the browser flow and try again.` on the card, and `control-openhands browser click 'testid=subscription-cancel'` removes the code (`count 'testid=subscription-user-code'` `0`). Switching back with `role=option[name=\"API key\"]` restores provider `OpenAI` and model `gpt-5.6-sol`. Completing sign-in is blocked (needs a ChatGPT Plus/Pro account).", "source_line": 95, "expected": [ "Authentication → ChatGPT subscription swaps the model fields for a subscription model list and a device-login card (code, sign-in link, poll, cancel); switching back restores the API-key model." ], "notes": [ "Known issue [#17391](https://github.com/OpenHands/OpenHands/issues/17391): ChatGPT Business subscription login can be refused or use a fallback model with a 404. Completing the entitlement and selected-model checks needs an approved ChatGPT Business workspace login and the reported Docker subscription runtime. (`F10.subscription`)." ], "annotations": [ "Known failure (reproduced 2026-10-13, #17135): run `control-openhands browser fill 'testid=profile-name-input' QA_sub`, `control-openhands browser network --clear`, `control-openhands browser click 'testid=save-profile-btn'` and `control-openhands browser network --bodies --filter '/api/profiles'`. Expected: Save validates and saves the shown `gpt-5.5`. Actual: one `POST /api/profiles/QA_sub/validate` whose body has `\"model\":\"openai/gpt-5.6-sol\"`, the API-key form's hidden default, although the dropdown shows `gpt-5.5` (it shows the first subscription model without writing it to the form); `browser toasts --history` has `OpenAI subscription login is required`, because no subscription is connected. Picking a model in `testid=llm-subscription-model-input`, even `gpt-5.5` again, makes the next Save send `POST /api/profiles/QA_sub/validate`, which answers `OpenAI subscription login is required` here (`browser toasts --history`), because no subscription is connected. `control-openhands browser click 'testid=subscription-connect'` and `control-openhands browser wait 'testid=subscription-user-code'`: the card shows the code, `Open sign-in page`, `I've finished signing in` (`subscription-poll`) and Cancel, and `control-openhands browser tabs` lists a second page on `auth.openai.com` (it opens a second or two after the code appears; rerun `browser tabs` if it lists one page). The card polls on its own, so `Sign-in is not complete yet. ...` can already show before you click poll. Close it with `control-openhands browser close-tab 1` and `control-openhands browser tab 0`. `control-openhands browser click 'testid=subscription-poll'` leaves `Sign-in is not complete yet. Finish the browser flow and try again.` on the card, and `control-openhands browser click 'testid=subscription-cancel'` removes the code (`count 'testid=subscription-user-code'` `0`). Switching back with `role=option[name=\"API key\"]` restores provider `OpenAI` and model `gpt-5.6-sol`. Completing sign-in is blocked (needs a ChatGPT Plus/Pro account)." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F10--delete", "title": "Delete", "ids": [ "F10.delete", "F10.delete-default" ], "body": "Run `control-openhands browser click '[data-testid=profile-row]:has([title=\"QA_pro-copy\"]) >> testid=profile-menu-trigger'` and `control-openhands browser click 'testid=profile-actions-menu >> testid=profile-delete'`. `control-openhands browser snapshot 'role=dialog'` shows `Delete Profile` with `Are you sure you want to delete the profile \"QA_pro-copy\"? This action cannot be undone.`, and `browser eval \"document.activeElement.innerText\"` is `Cancel`. `control-openhands browser click 'role=dialog[name=\"Delete Profile\"] >> role=button[name=\"Cancel\"]'` keeps the row (count `1`). Repeat, then `control-openhands browser click 'testid=delete-profile-confirm'`, `control-openhands browser wait-text 'Profile \"QA_pro-copy\" deleted'`, reload, and the count is `0`. Deleting the Default row the same way leaves exactly one `profile-active-badge` on another profile (`browser text '[data-testid=profile-row]:has([data-testid=profile-active-badge])'`), and `api GET /api/profiles` names it in `active_profile`.", "raw": "**Delete (`F10.delete`, `F10.delete-default`).** Run `control-openhands browser click '[data-testid=profile-row]:has([title=\"QA_pro-copy\"]) >> testid=profile-menu-trigger'` and `control-openhands browser click 'testid=profile-actions-menu >> testid=profile-delete'`. `control-openhands browser snapshot 'role=dialog'` shows `Delete Profile` with `Are you sure you want to delete the profile \"QA_pro-copy\"? This action cannot be undone.`, and `browser eval \"document.activeElement.innerText\"` is `Cancel`. `control-openhands browser click 'role=dialog[name=\"Delete Profile\"] >> role=button[name=\"Cancel\"]'` keeps the row (count `1`). Repeat, then `control-openhands browser click 'testid=delete-profile-confirm'`, `control-openhands browser wait-text 'Profile \"QA_pro-copy\" deleted'`, reload, and the count is `0`. Deleting the Default row the same way leaves exactly one `profile-active-badge` on another profile (`browser text '[data-testid=profile-row]:has([data-testid=profile-active-badge])'`), and `api GET /api/profiles` names it in `active_profile`.", "source_line": 96, "expected": [ "Delete asks `Are you sure you want to delete the profile \"\"? This action cannot be undone.` with Cancel focused; Cancel keeps the row, Delete removes it with a toast.", "deleting the Default profile promotes another remaining profile to Default." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--command-menu-and-direct-url", "title": "Command menu and direct URL", "ids": [ "F10.list" ], "body": "Run `control-openhands browser goto /` (the bullets above end on `/settings/llm`, where the URL wait would pass at once), `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' LLM` and `control-openhands browser snapshot 'testid=command-menu'`: two options, `LLM profiles` (\"Manage models, providers, and API keys.\") and `Model Router` (F12; its keywords include \"llm\"). Hovering an option selects it, so which one Enter runs depends on where the mouse rests: click the option instead, `control-openhands browser click 'testid=command-menu >> role=option[name=\"LLM profiles\"]' --expect-url '/settings/llm(\\?|$)'`. `browser count 'testid=add-llm-profile'` is `1` on arrival. `control-openhands browser goto /settings/llm` shows the same list (`browser count 'testid=profile-row'`, one `profile-active-badge`; screenshot `--feature F10.list --name list`).", "raw": "**Command menu and direct URL (`F10.list`).** Run `control-openhands browser goto /` (the bullets above end on `/settings/llm`, where the URL wait would pass at once), `control-openhands browser press Control+k`, `control-openhands browser type 'testid=command-menu >> role=combobox' LLM` and `control-openhands browser snapshot 'testid=command-menu'`: two options, `LLM profiles` (\"Manage models, providers, and API keys.\") and `Model Router` (F12; its keywords include \"llm\"). Hovering an option selects it, so which one Enter runs depends on where the mouse rests: click the option instead, `control-openhands browser click 'testid=command-menu >> role=option[name=\"LLM profiles\"]' --expect-url '/settings/llm(\\?|$)'`. `browser count 'testid=add-llm-profile'` is `1` on arrival. `control-openhands browser goto /settings/llm` shows the same list (`browser count 'testid=profile-row'`, one `profile-active-badge`; screenshot `--feature F10.list --name list`).", "source_line": 97, "expected": [ "the \"Available Profiles\" list shows each row's name, model, a `Default` badge on the active profile and a `...` menu, sorted by name; reachable from the sidebar, the direct URL and the command menu." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--phone-layout", "title": "Phone layout", "ids": [ "F10.phone" ], "body": "Run `control-openhands browser viewport phone`, `control-openhands browser goto /settings/llm`, `control-openhands browser bbox 'testid=add-llm-profile'`, `control-openhands browser bbox 'testid=profile-row >> nth=0'` and `control-openhands browser screenshot --feature F10.phone --name list`: `insideViewport` `true`, `pageHorizontalOverflow` `false`. Open a row menu: `browser bbox 'testid=profile-actions-menu'` is inside the viewport. Open Edit: `browser bbox 'testid=profile-name-input'` is 348 px wide inside the 390 px viewport and `pageHorizontalOverflow` is `false` (`llm-settings-screen` may report `insideViewport` `false` when the edited profile's form is taller than the screen; only the overflow flag matters). Return with `control-openhands browser click 'testid=cancel-profile-btn'` and `control-openhands browser viewport desktop`.", "raw": "**Phone layout (`F10.phone`).** Run `control-openhands browser viewport phone`, `control-openhands browser goto /settings/llm`, `control-openhands browser bbox 'testid=add-llm-profile'`, `control-openhands browser bbox 'testid=profile-row >> nth=0'` and `control-openhands browser screenshot --feature F10.phone --name list`: `insideViewport` `true`, `pageHorizontalOverflow` `false`. Open a row menu: `browser bbox 'testid=profile-actions-menu'` is inside the viewport. Open Edit: `browser bbox 'testid=profile-name-input'` is 348 px wide inside the 390 px viewport and `pageHorizontalOverflow` is `false` (`llm-settings-screen` may report `insideViewport` `false` when the edited profile's form is taller than the screen; only the overflow flag matters). Return with `control-openhands browser click 'testid=cancel-profile-btn'` and `control-openhands browser viewport desktop`.", "source_line": 98, "expected": [ "list, row menu and editor fit a 390 px viewport without horizontal overflow." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--default-profile-drives-a-conversation", "title": "Default profile drives a conversation", "ids": [ "F10.default-used" ], "body": "Make a profile with a distinct model Default through the row menu (for example `QA_pro-copy-1`, `deepseek/deepseek-v4-pro`, then `wait-text 'Switched to profile \"QA_pro-copy-1\"'`), and run `control-openhands conversation start --prompt \"Reply with exactly the word: pong\" --wait --timeout 180`. The JSON has `\"status\": \"finished\"` and `\"model\": \"deepseek/deepseek-v4-pro\"`. The browser is now on the conversation page: `control-openhands browser goto /settings/llm` before driving this family again.", "raw": "**Default profile drives a conversation (`F10.default-used`).** Make a profile with a distinct model Default through the row menu (for example `QA_pro-copy-1`, `deepseek/deepseek-v4-pro`, then `wait-text 'Switched to profile \"QA_pro-copy-1\"'`), and run `control-openhands conversation start --prompt \"Reply with exactly the word: pong\" --wait --timeout 180`. The JSON has `\"status\": \"finished\"` and `\"model\": \"deepseek/deepseek-v4-pro\"`. The browser is now on the conversation page: `control-openhands browser goto /settings/llm` before driving this family again.", "source_line": 99, "expected": [ "a conversation started after creating a profile in the UI runs on the Default profile's model." ], "notes": [ "Known issue [#17706](https://github.com/OpenHands/OpenHands/issues/17706): A freshly resolved npm-launcher environment can reject openhands/claude-* conversations because LiteLLM sends prompt_cache_key. Reproduction needs an approved OpenHands proxy key with an Anthropic model and the affected fresh transitive dependency resolution; a keyless initialization cannot prove provider acceptance. (`F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#4934](https://github.com/OpenHands/software-agent-sdk/issues/4934): Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (`F10.schema-validation`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#4250](https://github.com/OpenHands/software-agent-sdk/issues/4250): A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (`F10.custom-model`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#5890](https://github.com/OpenHands/software-agent-sdk/issues/5890): Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (`F10.edit`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#5838](https://github.com/OpenHands/software-agent-sdk/issues/5838): A live LLM switch during a running tool step can be acknowledged before the new LLM is persisted for GET/restart. Blocked: needs a genuine model-issued bounded tool step (for example `sleep 20`) and an approved starting model. Refused-loopback initialization never enters that step. (`F10.default-used`)." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F10--persistence-across-a-restart", "title": "Persistence across a restart", "ids": [ "F10.list" ], "body": "`control-openhands restart`, `control-openhands doctor`, `control-openhands browser goto /settings/llm`: the same rows and the same Default badge are back.", "raw": "**Persistence across a restart (`F10.list`).** `control-openhands restart`, `control-openhands doctor`, `control-openhands browser goto /settings/llm`: the same rows and the same Default badge are back.", "source_line": 100, "expected": [ "the \"Available Profiles\" list shows each row's name, model, a `Default` badge on the active profile and a `...` menu, sorted by name; reachable from the sidebar, the direct URL and the command menu." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--load-error", "title": "Load error", "ids": [ "F10.list-states" ], "body": "With any profile linked to a connection (here `QA_adv` → `QA_ds_conn`, from the Link bullet), make this run's connections store unreadable (shell arrange, this run only): `F=$OH_VERIFY_RUN/private/provider-connections/provider_connections.json; cp $F $F.bak; printf '{not json' > $F`. `control-openhands api GET /api/profiles` is now `400` (`Provider connections file is unreadable: …`). Run `control-openhands browser reload`, `control-openhands browser wait 'testid=profiles-load-error'` and `control-openhands browser wait 'testid=provider-connections-load-error'` (it can appear one retry after the profiles error): the first reads `Failed to load profiles` in the theme's error colour (`browser eval \"getComputedStyle(document.querySelector('[data-testid=profiles-load-error] p')).color\"` is `rgb(231, 106, 94)` on OpenHands-Neutral and `rgb(192, 40, 45)` on Light+, #18330), with both Add buttons still shown and `provider-connections-load-error` below it (`F11.load-error`); screenshot `--feature F10.list-states --name load-error-neutral`. Restore with `cp $F.bak $F && rm $F.bak` and `browser reload`. The backend-down route does not reach it: `control-openhands service stop agent-server` and `control-openhands browser reload` do not reach `testid=profiles-load-error`: the whole app is replaced by the disconnected-backend screen (`testid=agent-server-onboarding-screen`, toasts `An error occurred` (twice, one after the other) and `Could not connect to the configured agent server. Make sure it is running and reachable, then reload the page.` (#18051); the Manage backends row detail still reads `HTTP request failed (502 Bad Gateway): \"Bad Gateway: connect ECONNREFUSED 127.0.0.1:\"`). The list error needs `/api/profiles` alone to fail. Bring the stack back with `control-openhands restart`.", "raw": "**Load error (`F10.list-states`).** With any profile linked to a connection (here `QA_adv` → `QA_ds_conn`, from the Link bullet), make this run's connections store unreadable (shell arrange, this run only): `F=$OH_VERIFY_RUN/private/provider-connections/provider_connections.json; cp $F $F.bak; printf '{not json' > $F`. `control-openhands api GET /api/profiles` is now `400` (`Provider connections file is unreadable: …`). Run `control-openhands browser reload`, `control-openhands browser wait 'testid=profiles-load-error'` and `control-openhands browser wait 'testid=provider-connections-load-error'` (it can appear one retry after the profiles error): the first reads `Failed to load profiles` in the theme's error colour (`browser eval \"getComputedStyle(document.querySelector('[data-testid=profiles-load-error] p')).color\"` is `rgb(231, 106, 94)` on OpenHands-Neutral and `rgb(192, 40, 45)` on Light+, #18330), with both Add buttons still shown and `provider-connections-load-error` below it (`F11.load-error`); screenshot `--feature F10.list-states --name load-error-neutral`. Restore with `cp $F.bak $F && rm $F.bak` and `browser reload`. The backend-down route does not reach it: `control-openhands service stop agent-server` and `control-openhands browser reload` do not reach `testid=profiles-load-error`: the whole app is replaced by the disconnected-backend screen (`testid=agent-server-onboarding-screen`, toasts `An error occurred` (twice, one after the other) and `Could not connect to the configured agent server. Make sure it is running and reachable, then reload the page.` (#18051); the Manage backends row detail still reads `HTTP request failed (502 Bad Gateway): \"Bad Gateway: connect ECONNREFUSED 127.0.0.1:\"`). The list error needs `/api/profiles` alone to fail. Bring the stack back with `control-openhands restart`.", "source_line": 101, "expected": [ "a spinner shows while profiles load; a load failure shows `Failed to load profiles` in the theme's error colour." ], "notes": [], "annotations": [], "states": [ "mapped" ] }, { "name": "F10--cleanup", "title": "Cleanup", "ids": [], "body": "Run `control-openhands browser goto /settings/llm` (the Load error bullet left the disconnected-backend screen), then delete every `QA_*`, `deepseek-chat` and `deepseek-v3` profile through the row menu, linked ones first; `control-openhands browser count 'testid=profile-group-header'` drops to `0` once no linked row is left (`F10.list-grouped`). Then delete `QA_ds_conn` (`testid=provider-connection-row >> has-text=QA_ds_conn >> testid=provider-connection-menu-trigger`, `testid=provider-connection-actions-menu >> testid=provider-connection-delete`, `testid=delete-provider-connection-confirm`; toast `Connection \"QA_ds_conn\" deleted`). With no profiles left, `testid=profiles-empty` returns and `api GET /api/profiles` is `{\"profiles\":[],\"active_profile\":null}`.", "raw": "**Cleanup.** Run `control-openhands browser goto /settings/llm` (the Load error bullet left the disconnected-backend screen), then delete every `QA_*`, `deepseek-chat` and `deepseek-v3` profile through the row menu, linked ones first; `control-openhands browser count 'testid=profile-group-header'` drops to `0` once no linked row is left (`F10.list-grouped`). Then delete `QA_ds_conn` (`testid=provider-connection-row >> has-text=QA_ds_conn >> testid=provider-connection-menu-trigger`, `testid=provider-connection-actions-menu >> testid=provider-connection-delete`, `testid=delete-provider-connection-confirm`; toast `Connection \"QA_ds_conn\" deleted`). With no profiles left, `testid=profiles-empty` returns and `api GET /api/profiles` is `{\"profiles\":[],\"active_profile\":null}`.", "source_line": 102, "expected": [], "notes": [], "annotations": [], "states": [ "support" ] }, { "name": "F10--embedded-editor-starts-on-basic", "title": "Embedded editor starts on Basic", "ids": [ "F10.create-prefill", "F10.edit" ], "body": "Arrange an active disposable `QA_LOOPBACK` profile through the real profiles API with `llm.model` `openai/gpt-4o`, a dummy key, `llm.base_url` `http://127.0.0.1:1/v1`, `num_retries` `0` and temperature `0.4` (no completion). Run `control-openhands browser goto /settings/llm`, `control-openhands browser click 'testid=add-llm-profile'` and `control-openhands browser snapshot 'role=tablist'`. Expected by [#17801](https://github.com/OpenHands/OpenHands/issues/17801): Basic selected each time the embedded form opens. Known failure, reproduced 2026-10-12: All is selected. Back, open the exact `QA_LOOPBACK` row menu and `testid=profile-actions-menu >> testid=profile-edit`; the tablist also selects All. Repeat Add/Edit once, then Back without saving. This exercises the non-default settings condition; it does not claim that a default-only profile opens on All.", "raw": "**Embedded editor starts on Basic (`F10.create-prefill`, `F10.edit`).** Arrange an active disposable `QA_LOOPBACK` profile through the real profiles API with `llm.model` `openai/gpt-4o`, a dummy key, `llm.base_url` `http://127.0.0.1:1/v1`, `num_retries` `0` and temperature `0.4` (no completion). Run `control-openhands browser goto /settings/llm`, `control-openhands browser click 'testid=add-llm-profile'` and `control-openhands browser snapshot 'role=tablist'`. Expected by [#17801](https://github.com/OpenHands/OpenHands/issues/17801): Basic selected each time the embedded form opens. Known failure, reproduced 2026-10-12: All is selected. Back, open the exact `QA_LOOPBACK` row menu and `testid=profile-actions-menu >> testid=profile-edit`; the tablist also selects All. Repeat Add/Edit once, then Back without saving. This exercises the non-default settings condition; it does not claim that a default-only profile opens on All.", "source_line": 104, "expected": [ "a new editor prefills the backend default model with a blank key, base URL and connection; the other LLM options it shows come from the agent's current LLM settings, i.e. the last applied Default profile (the editor opens on the tab they need), and Save stores what the form shows.", "Edit loads the profile (`Editing profile \"\" - save to apply changes`) with the stored key masked; Save stays disabled until something changes; a changed name renames before saving; the stored key is kept." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#5890](https://github.com/OpenHands/software-agent-sdk/issues/5890): Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (`F10.edit`, `F10.default-used`).", "Known issue [#17806](https://github.com/OpenHands/OpenHands/issues/17806): A model-only Save must preserve an untouched key. Current SDK1.55 refused-loopback validation rejects Save before persistence; the stored key marker remains set. Completing Basic/Advanced preserved-key and runtime-switch proof needs an approved route that accepts preflight. Keep dummy traffic on refused loopback until an external route is authorized; a rejected Save does not establish either key loss or successful preservation through Save. (`F10.edit`)." ], "annotations": [ "Known failure, reproduced 2026-10-12: All is selected. Back, open the exact `QA_LOOPBACK` row menu and `testid=profile-actions-menu >> testid=profile-edit`; the tablist also selects All. Repeat Add/Edit once, then Back without saving. This exercises the non-default settings condition; it does not claim that a default-only profile opens on All." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F10--off-catalog-basic-model", "title": "Off-catalog Basic model", "ids": [ "F10.model-picker" ], "body": "From the list open Add, click `testid=sdk-section-basic-toggle`, then `control-openhands browser choose 'testid=llm-provider-input' OpenRouter`. Run `control-openhands browser fill 'testid=llm-model-input' 'thinkingmachines/inkling-small:free'`, `control-openhands browser press Tab` and `control-openhands browser value 'testid=llm-model-input'`; repeat fill/Tab once. Expected: keep the model entered under the selected provider. Known failure, reproduced 2026-10-12: the value is empty on both attempts ([#17577](https://github.com/OpenHands/OpenHands/issues/17577), concrete companion of [#17233](https://github.com/OpenHands/OpenHands/issues/17233)). Capture `browser screenshot --feature F10.model-picker --name off-catalog-discarded` and Back without saving. The Advanced provider-prefix/authenticated-save half needs a working OpenRouter route; this check proves the Basic entry dead end.", "raw": "**Off-catalog Basic model (`F10.model-picker`).** From the list open Add, click `testid=sdk-section-basic-toggle`, then `control-openhands browser choose 'testid=llm-provider-input' OpenRouter`. Run `control-openhands browser fill 'testid=llm-model-input' 'thinkingmachines/inkling-small:free'`, `control-openhands browser press Tab` and `control-openhands browser value 'testid=llm-model-input'`; repeat fill/Tab once. Expected: keep the model entered under the selected provider. Known failure, reproduced 2026-10-12: the value is empty on both attempts ([#17577](https://github.com/OpenHands/OpenHands/issues/17577), concrete companion of [#17233](https://github.com/OpenHands/OpenHands/issues/17233)). Capture `browser screenshot --feature F10.model-picker --name off-catalog-discarded` and Back without saving. The Advanced provider-prefix/authenticated-save half needs a working OpenRouter route; this check proves the Basic entry dead end.", "source_line": 106, "expected": [ "the Basic provider and model comboboxes group options as Verified Models and Other Models; changing provider clears the model; the OpenHands provider shows account and key help." ], "notes": [], "annotations": [ "Known failure, reproduced 2026-10-12: the value is empty on both attempts ([#17577](https://github.com/OpenHands/OpenHands/issues/17577), concrete companion of [#17233](https://github.com/OpenHands/OpenHands/issues/17233)). Capture `browser screenshot --feature F10.model-picker --name off-catalog-discarded` and Back without saving. The Advanced provider-prefix/authenticated-save half needs a working OpenRouter route; this check proves the Basic entry dead end." ], "states": [ "failure" ] }, { "name": "F10--azure-base-url-on-basic", "title": "Azure Base URL on Basic", "ids": [ "F10.model-picker" ], "body": "Open Add, explicitly select Basic and `control-openhands browser choose 'testid=llm-provider-input' Azure`. `control-openhands browser count 'testid=base-url-input'` should be `1` for Azure's required endpoint. Known failure, reproduced 2026-10-12 ([#16726](https://github.com/OpenHands/OpenHands/issues/16726)): Basic count `0`, after `browser click 'testid=sdk-section-advanced-toggle'` count `1`, and after `browser click 'testid=sdk-section-basic-toggle'` count `0` again. Screenshot Basic and Back without saving. No Azure account is needed for this form omission; its reported authenticated conversation error is outside this observation.", "raw": "**Azure Base URL on Basic (`F10.model-picker`).** Open Add, explicitly select Basic and `control-openhands browser choose 'testid=llm-provider-input' Azure`. `control-openhands browser count 'testid=base-url-input'` should be `1` for Azure's required endpoint. Known failure, reproduced 2026-10-12 ([#16726](https://github.com/OpenHands/OpenHands/issues/16726)): Basic count `0`, after `browser click 'testid=sdk-section-advanced-toggle'` count `1`, and after `browser click 'testid=sdk-section-basic-toggle'` count `0` again. Screenshot Basic and Back without saving. No Azure account is needed for this form omission; its reported authenticated conversation error is outside this observation.", "source_line": 108, "expected": [ "the Basic provider and model comboboxes group options as Verified Models and Other Models; changing provider clears the model; the OpenHands provider shows account and key help." ], "notes": [], "annotations": [ "Known failure, reproduced 2026-10-12 ([#16726](https://github.com/OpenHands/OpenHands/issues/16726)): Basic count `0`, after `browser click 'testid=sdk-section-advanced-toggle'` count `1`, and after `browser click 'testid=sdk-section-basic-toggle'` count `0` again. Screenshot Basic and Back without saving. No Azure account is needed for this form omission; its reported authenticated conversation error is outside this observation." ], "states": [ "failure" ] }, { "name": "F10--slow-invalid-preflight", "title": "Slow invalid preflight", "ids": [ "F10.create-validation" ], "body": "Arrange the disposable dummy-key `QA_LOOPBACK` profile from the embedded-editor bullet (all model traffic to `http://127.0.0.1:1/v1`). Open Add, explicitly select All and fill `testid=sdk-settings-llm.num_retries` with `5`; select Advanced, fill `llm-custom-model-input` with `openai/gpt-4o`, `base-url-input` with the refused loopback URL, the key via `--value-file ` and `profile-name-input` with `QA_SLOW`. Clear `browser network --filter QA_SLOW --clear`, click Save with `--observe 'testid=save-profile-btn' --observe-ms 1000`, then inspect toasts and network after the 30-second client deadline. Expected: refuse the unreachable endpoint and save nothing. Known failure, reproduced twice 2026-10-12 ([#15778](https://github.com/OpenHands/OpenHands/issues/15778), slow branch of [OpenHands/software-agent-sdk#5100](https://github.com/OpenHands/software-agent-sdk/issues/5100)): `Validating...` lasts until the browser aborts; 30 seconds after the validation POST, a profile POST succeeds with 201 and the profile appears in the list. The frontend deliberately returns no verdict for an abort, then saves. Repeating under `QA_SLOW_2` behaves the same; capture the list and `browser network --filter QA_SLOW_2 --bodies`. With retries `1`, the same refused endpoint instead failed in under a second, showed `Connection error` and left `GET /api/profiles/QA_INVALID` at 404 on both saves. Delete the two saved QA_SLOW profiles via the real API as fixture cleanup. This does not verify [OpenHands/software-agent-sdk#5100](https://github.com/OpenHands/software-agent-sdk/issues/5100)'s separate exhausted-budget 429 branch, which needs an approved real route returning that provider verdict.", "raw": "**Slow invalid preflight (`F10.create-validation`).** Arrange the disposable dummy-key `QA_LOOPBACK` profile from the embedded-editor bullet (all model traffic to `http://127.0.0.1:1/v1`). Open Add, explicitly select All and fill `testid=sdk-settings-llm.num_retries` with `5`; select Advanced, fill `llm-custom-model-input` with `openai/gpt-4o`, `base-url-input` with the refused loopback URL, the key via `--value-file ` and `profile-name-input` with `QA_SLOW`. Clear `browser network --filter QA_SLOW --clear`, click Save with `--observe 'testid=save-profile-btn' --observe-ms 1000`, then inspect toasts and network after the 30-second client deadline. Expected: refuse the unreachable endpoint and save nothing. Known failure, reproduced twice 2026-10-12 ([#15778](https://github.com/OpenHands/OpenHands/issues/15778), slow branch of [OpenHands/software-agent-sdk#5100](https://github.com/OpenHands/software-agent-sdk/issues/5100)): `Validating...` lasts until the browser aborts; 30 seconds after the validation POST, a profile POST succeeds with 201 and the profile appears in the list. The frontend deliberately returns no verdict for an abort, then saves. Repeating under `QA_SLOW_2` behaves the same; capture the list and `browser network --filter QA_SLOW_2 --bodies`. With retries `1`, the same refused endpoint instead failed in under a second, showed `Connection error` and left `GET /api/profiles/QA_INVALID` at 404 on both saves. Delete the two saved QA_SLOW profiles via the real API as fixture cleanup. This does not verify [OpenHands/software-agent-sdk#5100](https://github.com/OpenHands/software-agent-sdk/issues/5100)'s separate exhausted-budget 429 branch, which needs an approved real route returning that provider verdict.", "source_line": 110, "expected": [ "a Save whose validation call fails toasts the provider error and keeps the editor open with nothing saved." ], "notes": [ "Known issue [#17785](https://github.com/OpenHands/OpenHands/issues/17785): Saving DeepSeek-v4-flash on Windows Desktop can fail with missing cache_creation_tokens instead of saving. The reported path needs a Windows Desktop 1.24.0 environment with its SDK dependency resolution and an approved DeepSeek key; this macOS source run cannot establish that result. (`F10.create-validation`).", "Known issue [OpenHands/software-agent-sdk#5862](https://github.com/OpenHands/software-agent-sdk/issues/5862): DeepSeek invalid-key401 responses can be classified as bad-request errors. Blocked by this pass's loopback-only provider routing: a real invalid-key validation request to DeepSeek must be authorized to capture its actual401 shape. No usable key or successful inference is needed, but a loopback connection error does not prove this classification. (`F10.create-validation`)." ], "annotations": [ "Known failure, reproduced twice 2026-10-12 ([#15778](https://github.com/OpenHands/OpenHands/issues/15778), slow branch of [OpenHands/software-agent-sdk#5100](https://github.com/OpenHands/software-agent-sdk/issues/5100)): `Validating...` lasts until the browser aborts; 30 seconds after the validation POST, a profile POST succeeds with 201 and the profile appears in the list. The frontend deliberately returns no verdict for an abort, then saves. Repeating under `QA_SLOW_2` behaves the same; capture the list and `browser network --filter QA_SLOW_2 --bodies`. With retries `1`, the same refused endpoint instead failed in under a second, showed `Connection error` and left `GET /api/profiles/QA_INVALID` at 404 on both saves. Delete the two saved QA_SLOW profiles via the real API as fixture cleanup. This does not verify [OpenHands/software-agent-sdk#5100](https://github.com/OpenHands/software-agent-sdk/issues/5100)'s separate exhausted-budget 429 branch, which needs an approved real route returning that provider verdict." ], "states": [ "failure", "issue" ] }, { "name": "F10--deleted-title-profile-after-failed-cleanup", "title": "Deleted title profile after failed cleanup", "ids": [ "F10.delete" ], "body": "Arrange a non-default dummy loopback profile `QA_Titles3`, select it on Application's Title generation model dropdown and Save there (this also hydrates the frontend settings cache). Navigate back via `sidebar-settings-/settings/llm`, open the exact profile row menu and Delete. On macOS arrange a selective real write failure by recording `settings.json`'s original file flags, setting only `$OH_VERIFY_RUN/private/settings.json` to `stat.UF_IMMUTABLE` with `os.chflags`, and restoring its flags in a `finally` block immediately after the delete observation. Child `profiles/` stays writable and settings stay readable; ordinary chmod is insufficient because SDK Save repairs directory permissions. Clear the network log, confirm Delete, read toasts, the exact row count and `api GET /api/profiles/QA_Titles3`. Expected: remove the row despite the cleanup error. Known failure, reproduced twice 2026-10-12 ([#17520](https://github.com/OpenHands/OpenHands/issues/17520)): DELETE returns 200, the following title-cleanup PATCH returns 500 on three retries, the error surfaces, but the row remains (`1`) while the API returns 404. Repeat with `QA_Titles4`. Restore flags before `doctor`; clear the stale title preference through the real settings API. No provider validation or network interception is needed.", "raw": "**Deleted title profile after failed cleanup (`F10.delete`).** Arrange a non-default dummy loopback profile `QA_Titles3`, select it on Application's Title generation model dropdown and Save there (this also hydrates the frontend settings cache). Navigate back via `sidebar-settings-/settings/llm`, open the exact profile row menu and Delete. On macOS arrange a selective real write failure by recording `settings.json`'s original file flags, setting only `$OH_VERIFY_RUN/private/settings.json` to `stat.UF_IMMUTABLE` with `os.chflags`, and restoring its flags in a `finally` block immediately after the delete observation. Child `profiles/` stays writable and settings stay readable; ordinary chmod is insufficient because SDK Save repairs directory permissions. Clear the network log, confirm Delete, read toasts, the exact row count and `api GET /api/profiles/QA_Titles3`. Expected: remove the row despite the cleanup error. Known failure, reproduced twice 2026-10-12 ([#17520](https://github.com/OpenHands/OpenHands/issues/17520)): DELETE returns 200, the following title-cleanup PATCH returns 500 on three retries, the error surfaces, but the row remains (`1`) while the API returns 404. Repeat with `QA_Titles4`. Restore flags before `doctor`; clear the stale title preference through the real settings API. No provider validation or network interception is needed.", "source_line": 112, "expected": [ "Delete asks `Are you sure you want to delete the profile \"\"? This action cannot be undone.` with Cancel focused; Cancel keeps the row, Delete removes it with a toast." ], "notes": [], "annotations": [ "Known failure, reproduced twice 2026-10-12 ([#17520](https://github.com/OpenHands/OpenHands/issues/17520)): DELETE returns 200, the following title-cleanup PATCH returns 500 on three retries, the error surfaces, but the row remains (`1`) while the API returns 404. Repeat with `QA_Titles4`. Restore flags before `doctor`; clear the stale title preference through the real settings API. No provider validation or network interception is needed." ], "states": [ "failure" ] }, { "name": "F10--connection-linked-provider-dispatch", "title": "Connection-linked provider dispatch", "ids": [ "F10.custom-model", "F10.connection-link" ], "body": "From `/settings/llm` Add > Advanced, save `QA_QWEN` with `llm-custom-model-input` `qwen` linked through `llm-provider-connection-input` to an own OpenAI connection at refused loopback. Activate it and start two real Home chats; inspect `conversation events --kinds ConversationErrorEvent --last 2`. Expected: the chosen OpenAI-compatible connection supplies the transport. Known failure, reproduced twice 2026-10-12 ([#17261](https://github.com/OpenHands/OpenHands/issues/17261)): Save succeeds, but each actual chat fails before HTTP with `LLM Provider NOT provided` for `model=qwen`. The reported Windows installation, unlinked preflight and successful local inference are further branches. With a separate own connection whose provider is `custom_openai`, repeat the UI Save/start for `ollama-local/qwen`: the same failure reproduces twice for the namespace reported in [#17195](https://github.com/OpenHands/OpenHands/issues/17195). Its exact exposed model suffix is absent from the report, so `qwen` is an owned diagnostic suffix; this is not a complete 9Router/Ollama inference check. Restore the neutral active profile and remove only the own profiles/connections.", "raw": "**Connection-linked provider dispatch (`F10.custom-model`, `F10.connection-link`).** From `/settings/llm` Add > Advanced, save `QA_QWEN` with `llm-custom-model-input` `qwen` linked through `llm-provider-connection-input` to an own OpenAI connection at refused loopback. Activate it and start two real Home chats; inspect `conversation events --kinds ConversationErrorEvent --last 2`. Expected: the chosen OpenAI-compatible connection supplies the transport. Known failure, reproduced twice 2026-10-12 ([#17261](https://github.com/OpenHands/OpenHands/issues/17261)): Save succeeds, but each actual chat fails before HTTP with `LLM Provider NOT provided` for `model=qwen`. The reported Windows installation, unlinked preflight and successful local inference are further branches. With a separate own connection whose provider is `custom_openai`, repeat the UI Save/start for `ollama-local/qwen`: the same failure reproduces twice for the namespace reported in [#17195](https://github.com/OpenHands/OpenHands/issues/17195). Its exact exposed model suffix is absent from the report, so `qwen` is an owned diagnostic suffix; this is not a complete 9Router/Ollama inference check. Restore the neutral active profile and remove only the own profiles/connections.", "source_line": 115, "expected": [ "the Advanced tab takes a free-text model and Base URL; an empty model is refused with `Model is required`.", "with connections present the editor shows a Provider connection selector; linking hides API key and Base URL, skips validation on Save and regroups the row; None restores the key field." ], "notes": [ "Known issue [OpenHands/software-agent-sdk#4613](https://github.com/OpenHands/software-agent-sdk/issues/4613): A namespaced model such as openai/openai/example-model can have its inner provider prefix stripped again in Responses API and reach the provider as example-model. Reproduction needs an approved OpenAI-compatible endpoint serving the namespaced model and its exact stored API-mode settings; a loopback connection refusal cannot prove the outbound identifier. (`F10.custom-model`).", "Known issue [OpenHands/software-agent-sdk#4250](https://github.com/OpenHands/software-agent-sdk/issues/4250): A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (`F10.custom-model`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#4247](https://github.com/OpenHands/software-agent-sdk/issues/4247): An LM Studio configuration can fail with LLM Provider NOT provided although its models endpoint works. Reproduction needs the reachable LM Studio service and the exact model ID/provider-qualified config, which the report does not supply as text; the reported CLI/Docker SDK 1.16.1 installation is not this source stack. (`F10.custom-model`)." ], "annotations": [ "Known failure, reproduced twice 2026-10-12 ([#17261](https://github.com/OpenHands/OpenHands/issues/17261)): Save succeeds, but each actual chat fails before HTTP with `LLM Provider NOT provided` for `model=qwen`. The reported Windows installation, unlinked preflight and successful local inference are further branches. With a separate own connection whose provider is `custom_openai`, repeat the UI Save/start for `ollama-local/qwen`: the same failure reproduces twice for the namespace reported in [#17195](https://github.com/OpenHands/OpenHands/issues/17195). Its exact exposed model suffix is absent from the report, so `qwen` is an owned diagnostic suffix; this is not a complete 9Router/Ollama inference check. Restore the neutral active profile and remove only the own profiles/connections." ], "states": [ "failure", "issue" ] }, { "name": "F10--bedrock-named-profile-import", "title": "Bedrock named-profile import", "ids": [ "F10.default-used" ], "body": "In the run-owned launcher environment set `AWS_EC2_METADATA_DISABLED=true`, then `restart` and `doctor`; verify that flag without printing other environment contents. Arrange an own profile with `model: bedrock/anthropic.claude-3-haiku-20240307-v1:0`, `aws_profile_name: qa_nonexistent`, region `us-east-1`, retries `0`, and both `base_url` and `aws_bedrock_runtime_endpoint` `http://127.0.0.1:1/v1`. A dummy-only API-key presence marker is needed for Canvas's send gate; the SDK filters that non-Bedrock key on this IAM path. Activate and start a real Home chat; inspect the error privately. Expected: the dependency imports and the missing named profile is reported before any AWS/provider request. A connection error cannot prove successful Bedrock inference. Restore the launcher environment and active profile, then remove only the own fixture.", "raw": "**Bedrock named-profile import (`F10.default-used`).** In the run-owned launcher environment set `AWS_EC2_METADATA_DISABLED=true`, then `restart` and `doctor`; verify that flag without printing other environment contents. Arrange an own profile with `model: bedrock/anthropic.claude-3-haiku-20240307-v1:0`, `aws_profile_name: qa_nonexistent`, region `us-east-1`, retries `0`, and both `base_url` and `aws_bedrock_runtime_endpoint` `http://127.0.0.1:1/v1`. A dummy-only API-key presence marker is needed for Canvas's send gate; the SDK filters that non-Bedrock key on this IAM path. Activate and start a real Home chat; inspect the error privately. Expected: the dependency imports and the missing named profile is reported before any AWS/provider request. A connection error cannot prove successful Bedrock inference. Restore the launcher environment and active profile, then remove only the own fixture.", "source_line": 117, "expected": [ "a conversation started after creating a profile in the UI runs on the Default profile's model." ], "notes": [ "Known issue [#17706](https://github.com/OpenHands/OpenHands/issues/17706): A freshly resolved npm-launcher environment can reject openhands/claude-* conversations because LiteLLM sends prompt_cache_key. Reproduction needs an approved OpenHands proxy key with an Anthropic model and the affected fresh transitive dependency resolution; a keyless initialization cannot prove provider acceptance. (`F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#4934](https://github.com/OpenHands/software-agent-sdk/issues/4934): Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (`F10.schema-validation`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#4250](https://github.com/OpenHands/software-agent-sdk/issues/4250): A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (`F10.custom-model`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#5890](https://github.com/OpenHands/software-agent-sdk/issues/5890): Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (`F10.edit`, `F10.default-used`).", "Known issue [OpenHands/software-agent-sdk#5838](https://github.com/OpenHands/software-agent-sdk/issues/5838): A live LLM switch during a running tool step can be acknowledged before the new LLM is persisted for GET/restart. Blocked: needs a genuine model-issued bounded tool step (for example `sleep 20`) and an approved starting model. Refused-loopback initialization never enters that step. (`F10.default-used`)." ], "annotations": [], "states": [ "issue", "blocked" ] }, { "name": "F10--unavailable-subscription-upstream-is-handled", "title": "Unavailable subscription upstream is handled", "ids": [ "F10.subscription" ], "body": "Expected: device-start failure reports502/504 rather than a server crash. Known failure (reproduced twice 2026-10-16, [OpenHands/software-agent-sdk#5880](https://github.com/OpenHands/software-agent-sdk/issues/5880)): real Connect returned500 with the backend offline. The block uses an owned offline proxy; browser offline alone does not cut the backend request. The cached runtime must already exist: `UV_OFFLINE=1` prevents its dependency resolver from reaching the blocked network. The block changes only this run's environment, keeps model defaults on refused loopback, and restores/restarts it in `EXIT`. The poll branch remains blocked without a genuine pending device token.\n\n ```sh known-bug F10.subscription offline-start\n QA_SUB_ENV=\"$OH_VERIFY_RUN/private/qa-known-subscription-env.json\"\n cp \"$OH_VERIFY_RUN/private/launcher-env.json\" \"$QA_SUB_ENV\"\n trap 'cp \"$QA_SUB_ENV\" \"$OH_VERIFY_RUN/private/launcher-env.json\"; rm -f \"$QA_SUB_ENV\"; control-openhands restart --timeout 120 >/dev/null; control-openhands browser goto /settings/llm >/dev/null' EXIT\n python3 - <<'PYENV'\n from pathlib import Path\n import os,json\n p=Path(os.environ['OH_VERIFY_RUN'])/'private/launcher-env.json'\n d=json.loads(p.read_text())\n assert d.get('OPENAI_BASE_URL')=='http://127.0.0.1:1/v1'\n assert d.get('OPENAI_API_BASE')=='http://127.0.0.1:1/v1'\n d['UV_OFFLINE']='1'\n for key in ['HTTP_PROXY','HTTPS_PROXY','ALL_PROXY','http_proxy','https_proxy','all_proxy']: d[key]='http://127.0.0.1:1'\n d['NO_PROXY']=d['no_proxy']='127.0.0.1,localhost,::1'\n p.write_text(json.dumps(d))\n PYENV\n control-openhands restart --timeout 120\n control-openhands doctor\n control-openhands browser viewport desktop\n control-openhands browser goto /settings/llm\n control-openhands browser click 'testid=add-llm-profile'\n control-openhands browser click 'testid=sdk-section-basic-toggle'\n control-openhands browser choose 'testid=llm-auth-type-input' 'ChatGPT subscription'\n control-openhands browser network --clear\n control-openhands browser click 'testid=subscription-connect' --observe 'role=status' --observe-ms 1000\n control-openhands assert --field recent.0.status --matches '^(502|504)$' --bug 'recent.0.status eq 500 && recent.0.path eq \"/api/llm/subscription/openai/device/start\"' -- browser network --filter '/api/llm/subscription/openai/device/start' # bug\n ```", "raw": "**Unavailable subscription upstream is handled (`F10.subscription`).** Expected: device-start failure reports502/504 rather than a server crash. Known failure (reproduced twice 2026-10-16, [OpenHands/software-agent-sdk#5880](https://github.com/OpenHands/software-agent-sdk/issues/5880)): real Connect returned500 with the backend offline. The block uses an owned offline proxy; browser offline alone does not cut the backend request. The cached runtime must already exist: `UV_OFFLINE=1` prevents its dependency resolver from reaching the blocked network. The block changes only this run's environment, keeps model defaults on refused loopback, and restores/restarts it in `EXIT`. The poll branch remains blocked without a genuine pending device token.\n\n ```sh known-bug F10.subscription offline-start\n QA_SUB_ENV=\"$OH_VERIFY_RUN/private/qa-known-subscription-env.json\"\n cp \"$OH_VERIFY_RUN/private/launcher-env.json\" \"$QA_SUB_ENV\"\n trap 'cp \"$QA_SUB_ENV\" \"$OH_VERIFY_RUN/private/launcher-env.json\"; rm -f \"$QA_SUB_ENV\"; control-openhands restart --timeout 120 >/dev/null; control-openhands browser goto /settings/llm >/dev/null' EXIT\n python3 - <<'PYENV'\n from pathlib import Path\n import os,json\n p=Path(os.environ['OH_VERIFY_RUN'])/'private/launcher-env.json'\n d=json.loads(p.read_text())\n assert d.get('OPENAI_BASE_URL')=='http://127.0.0.1:1/v1'\n assert d.get('OPENAI_API_BASE')=='http://127.0.0.1:1/v1'\n d['UV_OFFLINE']='1'\n for key in ['HTTP_PROXY','HTTPS_PROXY','ALL_PROXY','http_proxy','https_proxy','all_proxy']: d[key]='http://127.0.0.1:1'\n d['NO_PROXY']=d['no_proxy']='127.0.0.1,localhost,::1'\n p.write_text(json.dumps(d))\n PYENV\n control-openhands restart --timeout 120\n control-openhands doctor\n control-openhands browser viewport desktop\n control-openhands browser goto /settings/llm\n control-openhands browser click 'testid=add-llm-profile'\n control-openhands browser click 'testid=sdk-section-basic-toggle'\n control-openhands browser choose 'testid=llm-auth-type-input' 'ChatGPT subscription'\n control-openhands browser network --clear\n control-openhands browser click 'testid=subscription-connect' --observe 'role=status' --observe-ms 1000\n control-openhands assert --field recent.0.status --matches '^(502|504)$' --bug 'recent.0.status eq 500 && recent.0.path eq \"/api/llm/subscription/openai/device/start\"' -- browser network --filter '/api/llm/subscription/openai/device/start' # bug\n ```", "source_line": 119, "expected": [ "Authentication → ChatGPT subscription swaps the model fields for a subscription model list and a device-login card (code, sign-in link, poll, cancel); switching back restores the API-key model." ], "notes": [ "Known issue [#17391](https://github.com/OpenHands/OpenHands/issues/17391): ChatGPT Business subscription login can be refused or use a fallback model with a 404. Completing the entitlement and selected-model checks needs an approved ChatGPT Business workspace login and the reported Docker subscription runtime. (`F10.subscription`)." ], "annotations": [ "Known failure (reproduced twice 2026-10-16, [OpenHands/software-agent-sdk#5880](https://github.com/OpenHands/software-agent-sdk/issues/5880)): real Connect returned500 with the backend offline. The block uses an owned offline proxy; browser offline alone does not cut the backend request. The cached runtime must already exist: `UV_OFFLINE=1` prevents its dependency resolver from reaching the blocked network. The block changes only this run's environment, keeps model defaults on refused loopback, and restores/restarts it in `EXIT`. The poll branch remains blocked without a genuine pending device token." ], "states": [ "failure", "issue", "blocked" ] }, { "name": "F10--contract-f10-schema-unavailable", "title": "Schema unavailable", "ids": [ "F10.schema-unavailable" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "a backend without the settings-schema endpoints shows `SDK settings schema unavailable.` with an upgrade or session-key hint." ], "notes": [], "annotations": [], "states": [ "contract" ] }, { "name": "F10--contract-f10-cloud-readonly", "title": "Cloud readonly", "ids": [ "F10.cloud-readonly" ], "body": "", "raw": "", "contract_only": true, "source_line": null, "expected": [ "Cloud org members without edit rights see the list read-only (no Add buttons, no row menus, no Provider connections section)." ], "notes": [], "annotations": [], "states": [ "contract" ] } ] }, { "id": "F11", "title": "Provider connections", "file": "F11-provider-connections.md", "sha256": "0af9aeea1923e3785c8e70855b8bbb0e5f5d9fd69d77efccc481b677c6a95882", "overview": "A provider connection is a named, shared credential (provider, API key,\noptional base URL) that many LLM profiles link to instead of carrying their own\nkey. Under Settings → LLM, below **Available Profiles**, a user lists\nconnections with their linked-model count, adds them, edits them (rename,\nprovider, base URL, key rotation with a blank key meaning \"keep\"), bulk-adds\nlinked profiles from a row's menu, and deletes them; the server refuses to\ndelete a connection that a profile or the active settings still reference.", "entry": "- Sidebar settings gear (`backend-selector-settings-link`), then **LLM** in the settings navigation (`sidebar-settings-/settings/llm`); the section sits below **Available Profiles**.\n- Direct URL `/settings/llm`.\n- Command menu (`Control+k`/`Meta+k`, or `command-menu-trigger`): search `LLM`, choose **LLM profiles**. Searching `provider connection` finds nothing.\n- **Add connection** (`add-provider-connection`) opens the create modal; each row's `...` (`provider-connection-menu-trigger`) opens **Bulk add** / **Edit** / **Delete**.\n- Elsewhere (other families): Model Router's editor has its own **Add connection** (`meta-profile-add-provider-connection`, see F12); the LLM profile editor links a profile to a connection and **Add from provider connections** bulk-adds without a preselected connection (see F10).", "given": "Preconditions:\n\n- A fresh `control-openhands launch --new`, doctored, `onboard --skip` done, desktop viewport. Do not run `llm preset deepseek` before this family: the recipes need no LLM profile and no connection (`control-openhands api GET /api/profiles` lists none, `control-openhands api GET /api/llm/provider-connections` is `[]`), so the profile added in Bulk add becomes the active one.\n- A DeepSeek key in a file (``). Pass it only with `browser fill ... --value-file `; never on the command line.\n- `F11.agent-uses-connection` and `F11.rotate` start three tiny conversations on `deepseek/deepseek-flash`.\n- `F11.load-error` is arranged by making the run's own connections store unreadable (a shell arrange step that writes only `$OH_VERIFY_RUN/private/provider-connections/provider_connections.json`, restored afterwards); `service stop agent-server` does not work because it replaces the whole page with `agent-server-onboarding-screen`.\n- Blocked here: `F11.cloud` (needs a Cloud account with and without an organization, as owner and as member).", "gotchas": "- Select the Provider by clicking an option (`testid=provider-item-` exists only for verified providers; use `'role=option[name=\"