Engel’s notebook

Agent profiles

F13 · 22 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F13 Agent profiles

22 behaviors · 28 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

Settings → Agent is a library of named agent profiles. Each profile is either an OpenHands agent (an LLM profile, a system prompt, a tool set, parallel tool calls, and MCP-server and secret scopes) or an ACP external agent (a preset such as Claude Code, Codex, Gemini CLI, Pi or OpenCode, its launch command, a model the agent itself reports, and provider credentials). One profile is active; new conversations started from Home launch from it.

How a user gets here

  • Sidebar settings gear (backend-selector-settings-link): on desktop /settings redirects to /settings/agents. In the settings navigation, Agent (sidebar-settings-/settings/agents).
  • Direct URL /settings/agents; the retired /settings/agent redirects there.
  • Command menu (Control+k/Meta+k or command-menu-trigger): Agent settings.
  • Getting-started checklist Customize your agent and the composer + → Switch agent profile → Manage agent profiles (both mapped in F09/F05).
  • On Home, the composer + → Switch agent profile list activates a profile like Set as active (F05.agent-profile-switch).
  • All per-profile actions are in the row's ... menu (agent-profile-menu-trigger). The editor has no route: it replaces the list in place.

Given

Preconditions:

  • Baseline state (launched, doctored, onboard --skip done) on a fresh run. F13.editor-no-llm must run before control-openhands llm preset deepseek; everything from F13.stale-llm-ref on needs the preset (deepseek-flash active, deepseek-pro).
  • A fresh run has one profile, default, active, whose LLM profile is default (which does not exist). control-openhands api GET /api/agent-profiles shows it.
  • F13.mcp-scope needs two configured MCP servers. Arrange them with control-openhands api POST /api/settings/mcp/qa_mcp_a --data '{"transport":"stdio","command":"qa-no-such-command","enabled":false}' --write and the same for qa_mcp_b.
  • F13.acp-conversation needs the provider's agent CLI and an approved account; F13.cloud-read-only needs a Cloud organization member login. Both are blocked here.
  • Dropdowns are autocompletes. Pick an option with control-openhands browser choose 'testid=<testid>' '<label>' (proven on agent-type-selector, agent-preset-selector, agent-model-selector (Custom), agent-profile-llm-selector, agent-settings-mcp-mode and agent-settings-secrets-mode). The manual opener, control-openhands browser click 'div:has(> div > input[data-testid=<testid>]) >> role=button[name="Show suggestions"]' then browser click 'role=option[name="<label>"]', works for most of them but is flaky right after an editor opens (see Gotchas); clicking the input itself can open an empty, filtered list.
ListF13.listMapped behavior

Scenario F13--list · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Settings”.
  2. Do
    Open /.
  3. Do
    Press Control+k.
  4. Do
    Enter “Agent settings” in “Command menu” / the combobox.
  5. Do
    Press Enter.
  6. Wait
    Wait for the element identified in the CLI recipe to appear.
  7. Do
    Open /settings/agent.

Expect

under Available Profiles, rows (sorted by name) show the profile name, then its LLM profile (OpenHands) or ACP (external subprocess), a Default badge on the active profile and a ... menu (Edit, Set as active, Delete).

Exact CLI recipe & bug checks

From / run control-openhands browser click 'testid=backend-selector-settings-link' --expect-url '/settings', then control-openhands browser url (/settings/agents) and control-openhands browser text 'testid=agent-profile-row': default, its LLM profile default, Default on three lines. control-openhands browser count 'testid=agent-profile-active-badge' is 1; control-openhands browser screenshot --feature F13.list --name list. Command menu: run control-openhands browser goto / (the browser is on /settings/agents, where the URL wait would pass at once), control-openhands browser press Control+k, control-openhands browser type 'testid=command-menu >> role=combobox' 'Agent settings', control-openhands browser press Enter, control-openhands browser wait-url '/settings/agents(\?|$)'. Legacy URL: control-openhands browser goto /settings/agent returns .../settings/agents.

No LLM profileF13.editor-no-llmMapped behavior

Scenario F13--no-llm-profile · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add agent profile”.
  2. Do
    Enter “QA_NoLlm” in “Agent profile name”.
  3. Do
    Choose “Cancel agent profile btn”.

Expect

with no LLM profiles, an OpenHands profile shows Create an LLM profile first to define an OpenHands agent. and Save toasts Select an LLM profile for this agent. without creating anything.

Exact CLI recipe & bug checks

Before the preset, run control-openhands browser click 'testid=add-agent-profile', then control-openhands browser text 'testid=agent-profile-no-llm': Create an LLM profile first to define an OpenHands agent. control-openhands browser text 'testid=agent-profile-editor-title' is Add agent profile and control-openhands browser text 'testid=agent-profile-editor-description' is Configure the agent below, then click Save to create this profile.. Run control-openhands browser fill 'testid=agent-profile-name-input' QA_NoLlm; Save turns enabled (control-openhands browser enabled 'testid=save-agent-profile-btn'). Click testid=save-agent-profile-btn and run control-openhands browser toasts: Select an LLM profile for this agent. api GET /api/agent-profiles lists no QA_NoLlm. Leave with control-openhands browser click 'testid=cancel-agent-profile-btn'.

Name rulesF13.name-validationMapped behavior

Scenario F13--name-rules · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    In the Add editor, fill testid=agent-profile-name-input with '', default, 'bad name!' and QA_ok in turn; after each run control-openhands browser attr 'testid=agent-profile-name-input' aria-invalid and control-openhands browser enabled 'testid=save-agent-profile-btn'. Results: '' → true/false; default (duplicate) → true/false; 'bad name!' → true/false; QA_ok → false/true. The rule under the field is always shown (grey) and turns the theme danger colour (text-danger, #18199) while the name is invalid; it reads Profile name must start with a letter or digit, and contain only letters, digits, dots, underscores, or hyphens (max 64 characters). For the duplicate, control-openhands browser count 'role=alert >> text=already exists' is 1: the alert is the field's description, in the danger colour (rgb(231, 106, 94) on OpenHands-Neutral, rgb(209, 52, 56) on Light+), while the rule text stays grey. It reads An agent profile with this name already exists. Choose a different name, or edit the existing one. Take control-openhands browser screenshot --feature F13.name-validation --name duplicate with default filled. In Edit the same message appears for another profile's name and not for the profile's own name (F13.edit).

Expect

an empty or malformed name marks the field invalid (red rule) and disables Save; a name another profile already has also marks the field invalid, shows a red taken-name message about agent profiles under it (the rule stays grey) and disables Save (Edit accepts the profile's own name).

Exact CLI recipe & bug checks

In the Add editor, fill testid=agent-profile-name-input with '', default, 'bad name!' and QA_ok in turn; after each run control-openhands browser attr 'testid=agent-profile-name-input' aria-invalid and control-openhands browser enabled 'testid=save-agent-profile-btn'. Results: '' → true/false; default (duplicate) → true/false; 'bad name!' → true/false; QA_ok → false/true. The rule under the field is always shown (grey) and turns the theme danger colour (text-danger, #18199) while the name is invalid; it reads Profile name must start with a letter or digit, and contain only letters, digits, dots, underscores, or hyphens (max 64 characters). For the duplicate, control-openhands browser count 'role=alert >> text=already exists' is 1: the alert is the field's description, in the danger colour (rgb(231, 106, 94) on OpenHands-Neutral, rgb(209, 52, 56) on Light+), while the rule text stays grey. It reads An agent profile with this name already exists. Choose a different name, or edit the existing one. Take control-openhands browser screenshot --feature F13.name-validation --name duplicate with default filled. In Edit the same message appears for another profile's name and not for the profile's own name (F13.edit).

ACP formF13.acp-formMapped behavior

Scenario F13--acp-form · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Agent type selector”.
  2. Do
    Choose “ACP (external subprocess)”.
  3. Do
    Choose the element identified in the CLI recipe.
  4. Do
    Press Escape.
  5. Do
    Enter “npx -y qa-custom-acp --stdio” in “Agent command”.
  6. Do
    Choose “Custom” in “Agent model selector”.
  7. Do
    Choose “Custom”.

Expect

choosing ACP (external subprocess) shows Preset, Command, Credentials and Model; each preset prefills its command and its model choice (Agent default, or Gemini CLI's preferred gemini-2.5-pro), Custom empties the command and disables Save until one is typed, typing a preset's exact command selects that preset, and the Model list's Custom opens a free-text model field.

Exact CLI recipe & bug checks

In the Add editor run control-openhands browser click 'testid=agent-type-selector' and control-openhands browser click 'role=option[name="ACP (external subprocess)"]'. control-openhands browser value 'testid=agent-preset-selector' is Claude Code, browser value 'testid=agent-command-input' is npx -y --prefer-offline @agentclientprotocol/claude-agent-acp@0.63.0 and browser value 'testid=agent-model-selector' is Agent default; agent-profile-no-llm is gone (ACP needs no LLM profile). The preset list (control-openhands browser click 'div:has(> div > input[data-testid=agent-preset-selector]) >> role=button[name="Show suggestions"]', then control-openhands browser snapshot 'role=listbox') is Claude Code, Codex, Gemini CLI, Pi, OpenCode, Custom (browser press Escape). Pick Codex and Gemini CLI from testid=agent-preset-selector: the command becomes npx -y --prefer-offline @agentclientprotocol/codex-acp@1.10.0 / npx -y --prefer-offline @google/gemini-cli@0.46.0 --acp and the model Agent default / gemini-2.5-pro (the models come from the agent since #18131: F13.acp-model-discovery). Pick Custom: the command is empty and Save is disabled. control-openhands browser fill 'testid=agent-command-input' 'npx -y qa-custom-acp --stdio' enables Save. Filling the exact Codex command flips the preset to Codex; appending --verbose flips it back to Custom. Model override: with a built-in preset, run control-openhands browser choose 'testid=agent-model-selector' 'Custom' (or open the model list with its toggle button and control-openhands browser click 'role=option[name="Custom"][exact]'); control-openhands browser count 'testid=agent-model-input' is 1 and empty. The Custom preset shows agent-model-input directly (no model list).

Create an ACP profileF13.createMapped behavior

Scenario F13--create-an-acp-profile · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Wait
    Wait for the element identified in the CLI recipe to appear.
  2. Do
    Reload the page.

Expect

Add agent profile opens the editor; Save toasts Profile "<name>" created and the row is listed after a reload.

Exact CLI recipe & bug checks

With preset Custom, fill testid=agent-profile-name-input with QA_acp_custom, testid=agent-command-input with 'npx -y qa-custom-acp --stdio' and testid=agent-model-input with qa-model, click testid=save-agent-profile-btn, then control-openhands browser wait-text 'Profile "QA_acp_custom" created' --timeout 10000, control-openhands browser reload and control-openhands browser text 'testid=agent-profile-row >> has-text=QA_acp_custom': QA_acp_custom / ACP (external subprocess). control-openhands api GET /api/agent-profiles/QA_acp_custom shows acp_server custom, the command and acp_model qa-model.

Pi and OpenCode presetsF13.acp-local-presetsBlocked

Scenario F13--pi-and-opencode-presets · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add agent profile”.
  2. Do
    Choose “ACP (external subprocess)” in “Agent type selector”.
  3. Do
    Choose “<preset>” in “Agent preset selector”.
  4. Wait
    Wait for “Agent model discovering” to disappear.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Reload the page.

Expect

on a local backend the Preset list also offers Pi and OpenCode (with their own commands and credentials); a profile saved on them stores acp_server pi/opencode with acp_command null. On Cloud both are hidden (blocked here).

Exact CLI recipe & bug checks

For each of OpenCode and Pi: run control-openhands browser click 'testid=add-agent-profile', control-openhands browser choose 'testid=agent-type-selector' 'ACP (external subprocess)' and control-openhands browser choose 'testid=agent-preset-selector' '<preset>'. browser value 'testid=agent-command-input' is npx -y --prefer-offline opencode-ai@1.18.23 acp / npx -y --prefer-offline --package=pi-acp@0.0.33 --package=@earendil-works/pi-coding-agent@0.83.0 pi-acp (the versions follow @openhands/typescript-client). Fill testid=agent-profile-name-input with QA_opencode / QA_pi, run control-openhands browser wait 'testid=agent-model-discovering' --state detached --timeout 120000, click testid=save-agent-profile-btn and control-openhands browser wait-text 'Profile "<name>" created' --timeout 10000. control-openhands api GET /api/agent-profiles/<name> has acp_server opencode / pi, acp_command null and acp_model null, and after browser reload both rows read <name> / ACP (external subprocess). Cloud hides both presets (blocked here: needs a Cloud backend).

Models from the agentF13.acp-model-discoveryKnown issueBlocked

Scenario F13--models-from-the-agent · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Wait
    Wait for “Agent model discovering” to disappear.
  2. Do
    Choose the element identified in the CLI recipe.
  3. Do
    Choose “OpenCode/Exo Free” in “Agent model selector”.

Expect

for a built-in preset with its default command the Model list comes from the agent: Asking <agent> which models it offers… while the agent-server asks it, then its models with Agent default (<model>) first; an agent that wants a login says Sign in or add credentials above to see the models <agent> offers., one that cannot start says <agent>'s models appear here after your first conversation with it. …; a picked model is saved as acp_model and reopens selected.

Linked known issues

Known issue OpenHands/software-agent-sdk#5995: Claude Code ACP first turns with claude-opus-5-5 can fail because the pinned adapter bundles an older Claude Code. Blocked: needs an approved Claude login/API key entitled to Opus5.5 and a real first turn; current SDK1.55 still pins the reported adapter0.63.0. No authenticated turn was attempted. (F13.acp-conversation, F13.acp-model-discovery).

Exact CLI recipe & bug checks

In a new ACP editor, each built-in preset pick first shows (unless a successful lookup for that preset and secret set is cached, about 5 minutes: OpenCode then skips it) control-openhands browser text 'testid=agent-model-discovering' Asking <agent> which models it offers… (the agent-server runs the agent CLI through npx for POST /api/acp/models; the first lookup can take a minute while npx downloads it). Wait with control-openhands browser wait 'testid=agent-model-discovering' --state detached --timeout 120000, then read control-openhands browser testids --hidden --filter agent-model. In this sandbox (no Claude, Codex, Gemini or Pi login for the agent-server): Claude Code fails to start and shows agent-model-list-after-first-conversation (Claude Code's models appear here after your first conversation with it. You can also switch models from the chat.); Codex, Gemini CLI and Pi need a login and show agent-model-discovery-needs-auth (Sign in or add credentials above to see the models <agent> offers.), with the model left at Agent default (gemini-2.5-pro for Gemini); <run>/private/stack.log logs ACP model discovery failed: server=<key> code=ACPInitError|ACPAuthRequired. OpenCode needs no key: browser value 'testid=agent-model-selector' is Agent default (OpenCode/Big Pickle) and the list (open it with browser click 'div:has(> div > input[data-testid=agent-model-selector]) >> role=button[name="Show suggestions"]', then browser snapshot 'role=listbox') has that entry, a non-empty list of the models OpenCode reports live (12 on 2026-10-13, such as OpenCode/Big Pickle and OpenCode/Exo Free) and Custom (screenshot --feature F13.acp-model-discovery --name opencode-models). Edit QA_opencode: Save is disabled until control-openhands browser choose 'testid=agent-model-selector' 'OpenCode/Exo Free'; Save (Profile "QA_opencode" updated) stores acp_model opencode/exo-free, and reopening shows OpenCode/Exo Free with Save disabled. With a saved OPENAI_API_KEY (even a dummy, F13.acp-credentials) Codex lists its models and shows Agent default (GPT-6-Astra). Lookups never add a conversation to the list (api GET "/api/conversations/search?limit=20" stays empty).

Edit and renameF13.editMapped behavior

Scenario F13--edit-and-rename · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Agent profile row” / containing “QA_acp_custom” / “Agent profile menu trigger”.
  2. Wait
    Wait for the element identified in the CLI recipe to appear.
  3. Do
    Reload the page.

Expect

Edit reopens the stored values with Save disabled until something changes; renaming keeps the profile id; Back and Cancel leave without saving.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=agent-profile-row >> has-text=QA_acp_custom >> testid=agent-profile-menu-trigger'; control-openhands browser snapshot 'testid=agent-profile-actions-menu' lists Edit, Set as active, Delete. Click testid=agent-profile-edit: the title is Edit agent profile, the description Editing profile "QA_acp_custom" - save to apply changes, the type/preset/command/model values are the stored ones, and Save is disabled. Note the id from api GET /api/agent-profiles, fill the name with QA_acp_renamed, click Save, control-openhands browser wait-text 'Profile "QA_acp_renamed" updated' --timeout 10000 and browser reload: browser count 'testid=agent-profile-row >> has-text=QA_acp_custom' is 0, the renamed row is 1 and api GET /api/agent-profiles shows the same id under the new name. Back/Cancel: open any editor, change a field, click testid=back-to-agent-profiles (or testid=cancel-agent-profile-btn); browser count 'testid=add-agent-profile' is 1 and the API is unchanged.

CredentialsF13.acp-credentialsKnown failureKnown issue

Scenario F13--credentials · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Scroll “Acp credential conflict warning” as specified in the recipe.

Expect

built-in presets show a Credentials section (password fields), an auth banner (signed in / configured), a conflict warning for mutually exclusive Claude credentials, and Already saved — leave blank to keep for stored values. Credentials are saved as global secrets.

Known bug observations

Known failure in Light+ (reproduced 2026-10-13 on TARGET and BASE, #18004): the auth banners use raw text-green-200 (settings-acp-auth-detected) and text-blue-200 (settings-acp-auth-configured) on a pale tint over white (acp-auth-status-banner.tsx:63/93), measuring 1.10:1 and 1.26:1; expected: readable banner text in every theme (screenshot --feature F13.acp-credentials --name detected-banner-lightplus). Cancel; api GET /api/settings/secrets gains nothing. Saved credential: new profile QA_codex, ACP, preset Codex, fill testid=settings-acp-secret-OPENAI_API_KEY with qa-dummy-openai, Save (Profile "QA_codex" created); api GET /api/settings/secrets now lists OPENAI_API_KEY and the profile has acp_server codex, acp_command null, acp_model null (the agent's own default since #18131). Reopen it: control-openhands browser testids --hidden --filter settings-acp-auth shows settings-acp-auth-configured (Credentials for Codex are configured — you can leave the fields below blank.) and control-openhands browser attr 'testid=settings-acp-secret-OPENAI_API_KEY' placeholder is Already saved — leave blank to keep with an empty value.

Linked known issues

Known issue #16296: Claude Code credential fields remain visible below the already-connected banner, leading users to think keys are required. The connected-state check needs an approved existing Claude CLI login on the agent-server host; a dummy credential does not prove that signed-in state. (F13.acp-credentials).

Exact CLI recipe & bug checks

In a new ACP editor with preset Claude Code, control-openhands browser testids --hidden --filter acp lists settings-acp-secret-CLAUDE_CODE_OAUTH_TOKEN, -ANTHROPIC_API_KEY, -ANTHROPIC_BASE_URL and one banner (settings-acp-auth-detected when the agent-server machine has a Claude CLI login: You're already signed in to Claude Code — you can leave the fields below blank.). control-openhands browser attr 'testid=settings-acp-secret-ANTHROPIC_API_KEY' type is password. Fill settings-acp-secret-CLAUDE_CODE_OAUTH_TOKEN with qa-dummy-oauth and settings-acp-secret-ANTHROPIC_API_KEY with qa-dummy-key; control-openhands browser text 'testid=acp-credential-conflict-warning' is Setting ANTHROPIC_API_KEY alongside CLAUDE_CODE_OAUTH_TOKEN breaks its authentication — leave one of them blank. (screenshot --feature F13.acp-credentials --name conflict after browser scroll 'testid=acp-credential-conflict-warning'). Known failure in Light+ (reproduced 2026-10-13 on TARGET and BASE, #18004): the auth banners use raw text-green-200 (settings-acp-auth-detected) and text-blue-200 (settings-acp-auth-configured) on a pale tint over white (acp-auth-status-banner.tsx:63/93), measuring 1.10:1 and 1.26:1; expected: readable banner text in every theme (screenshot --feature F13.acp-credentials --name detected-banner-lightplus). Cancel; api GET /api/settings/secrets gains nothing. Saved credential: new profile QA_codex, ACP, preset Codex, fill testid=settings-acp-secret-OPENAI_API_KEY with qa-dummy-openai, Save (Profile "QA_codex" created); api GET /api/settings/secrets now lists OPENAI_API_KEY and the profile has acp_server codex, acp_command null, acp_model null (the agent's own default since #18131). Reopen it: control-openhands browser testids --hidden --filter settings-acp-auth shows settings-acp-auth-configured (Credentials for Codex are configured — you can leave the fields below blank.) and control-openhands browser attr 'testid=settings-acp-secret-OPENAI_API_KEY' placeholder is Already saved — leave blank to keep with an empty value.

Secrets scopeF13.secrets-scopeMapped behavior

Scenario F13--secrets-scope · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Choose secrets”.
  2. Do
    Reload the page.
  3. Do
    Choose “Agent settings secret list” / “OPENHANDS_AUTOMATION_API_KEY”.

Expect

Secrets offers No profile restriction (every switch on and disabled) or Choose secrets; on an ACP profile, Choose preselects that provider's secret names; the selection persists.

Exact CLI recipe & bug checks

In the QA_codex editor, control-openhands browser eval "[...document.querySelectorAll('[data-testid=agent-settings-secret-list] input')].map(i=>i.closest('li').innerText.split('\n')[0]+'='+i.checked+(i.disabled?'(disabled)':''))" shows every name =true(disabled) under No profile restriction. Open testid=agent-settings-secrets-mode, control-openhands browser click 'role=option[name="Choose secrets"]' and rerun the eval: OPENHANDS_AUTOMATION_API_KEY=false, OPENAI_API_KEY=true, CODEX_AUTH_JSON=true, OPENAI_BASE_URL=true. Save (Profile "QA_codex" updated); api GET /api/agent-profiles/QA_codex has those three in secret_refs, and after browser reload and Edit the mode and switches are the same. On an OpenHands profile, Choose starts with nothing selected; a switch toggles by its name: in the Add editor pick Choose secrets, then control-openhands browser click 'testid=agent-settings-secret-list >> text=OPENHANDS_AUTOMATION_API_KEY' turns document.querySelector('[data-testid=agent-settings-secret-OPENHANDS_AUTOMATION_API_KEY]').checked from false to true.

Stale LLM referenceF13.stale-llm-refMapped behavior

Scenario F13--stale-llm-reference · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Configure or inspect the model profiles specified in the recipe.
  2. Do
    Reload the page.
  3. Wait
    Wait for “Agent profile llm selector” to appear.
  4. Do
    Reload the page.

Expect

editing a profile whose LLM profile no longer exists preselects the active LLM profile and enables Save, so saving heals the reference.

Exact CLI recipe & bug checks

After control-openhands llm preset deepseek, run control-openhands browser reload, open testid=agent-profile-row >> has-text=default >> testid=agent-profile-menu-trigger → testid=agent-profile-edit, control-openhands browser wait 'testid=agent-profile-llm-selector'. browser value 'testid=agent-profile-llm-selector' is deepseek-flash (deepseek/deepseek-flash) and Save is enabled although nothing was touched. Save (Profile "default" updated), browser reload: the row reads default / deepseek-flash / Default and api GET /api/agent-profiles/default has llm_profile_ref deepseek-flash with the same id. Screenshot --feature F13.stale-llm-ref --name fallback before saving.

OpenHands optionsF13.openhands-options · F13.createKnown issue

Scenario F13--openhands-options · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Choose tools” in “Agent settings tools mode”.
  2. Do
    Choose “deepseek-pro (deepseek/deepseek-v4-pro)” in “Agent profile llm selector”.
  3. Do
    Reload the page.
  4. Do
    Choose “Default + your instructions” in “Agent settings system prompt mode”.
  5. Do
    Reload the page.

Expect

OpenHands profiles offer an LLM profile dropdown (the active LLM profile by default), a System prompt choice (OpenHands default, Default + your instructions or Custom persona, each saved on the profile), Parallel tool calls (default 1, minimum 1) and Tools: Standard tools (default) lists the server's standard set read-only, Choose tools turns the server's tool catalog into switches; values persist and reopen.

Add agent profile opens the editor; Save toasts Profile "<name>" created and the row is listed after a reload.

Linked known issues

Known issue OpenHands/software-agent-sdk#5899: The browser tool can be reported unusable when genuine Chromium exists only under PLAYWRIGHT_BROWSERS_PATH. Blocked on this host: it has standard user Chrome. The distinguishing fixture needs a host/container with no standard/default-cache/PATH browser, plus a genuine custom-cache Chromium; never remove the user browser to create it. (F13.openhands-options; operational browser path F27).

Exact CLI recipe & bug checks

Click testid=add-agent-profile, fill the name QA_oh. Defaults: browser value on testid=agent-profile-llm-selector is the active LLM profile (deepseek-flash (deepseek/deepseek-flash)), on testid=agent-settings-system-prompt-mode OpenHands default, on testid=agent-settings-tools-mode Standard tools and on testid=sdk-settings-tool_concurrency_limit 1; control-openhands browser text 'testid=agent-settings-tool-list' lists file_editor, task_tracker, terminal and switch_llm with their descriptions (the list follows the Agent Server; it also includes browser_tool_set when usable). Run control-openhands browser choose 'testid=agent-settings-tools-mode' 'Choose tools'; control-openhands browser eval "[...document.querySelectorAll('[data-testid=agent-settings-tool-list] input')].map(i=>i.dataset.testid.replace('agent-settings-tool-','')+'='+i.checked)" is the selected standard tools, including file_editor=true, task_tracker=true, terminal=true and switch_llm=true; task_tool_set=false, glob=false and grep=false. Additional rows follow the current server catalog. Click 'testid=agent-settings-tool-list >> text=switch_llm' and 'testid=agent-settings-tool-list >> text=task_tool_set' (the eval flips both). Fill testid=sdk-settings-tool_concurrency_limit with 0 and click Save: browser toasts shows Parallel tool calls must be at least 1 and browser eval "document.querySelector('[data-testid=sdk-settings-tool_concurrency_limit]').validationMessage" is Value must be greater than or equal to 1. Fill 2, control-openhands browser choose 'testid=agent-profile-llm-selector' 'deepseek-pro (deepseek/deepseek-v4-pro)', Save (Profile "QA_oh" created), browser reload: the row reads QA_oh / deepseek-pro, and api GET /api/agent-profiles/QA_oh has llm_profile_ref deepseek-pro, tools retaining the selected standard tools plus task_tool_set (including task_tracker), enable_classify_and_switch_llm_tool false and tool_concurrency_limit 2. Reopening shows Choose tools with the same switches, 2 and deepseek-pro, with Save disabled. (There are no Enable sub-agents or Let the agent switch LLM profiles switches: browser count 'text=Enable sub-agents' is 0.) System prompt: reopen QA_oh, control-openhands browser choose 'testid=agent-settings-system-prompt-mode' 'Default + your instructions' and fill testid=agent-settings-system-prompt with Use terse QA answers.. Save, browser reload and reopen: api GET /api/agent-profiles/QA_oh has system_message_suffix Use terse QA answers. and persona null, and the editor keeps the mode and text with Save disabled. Then choose Custom persona, fill You are the disposable QA persona., Save, reload and reopen: persona is that text, system_message_suffix is null, and the editor keeps the mode and text. Choose OpenHands default and Save (both fields null) before later checks that use this profile.

Set as activeF13.set-activeMapped behavior

Scenario F13--set-as-active · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Agent profile row” / containing “QA_oh” / “Agent profile menu trigger”.
  2. Do
    Choose “Agent profile set active”.
  3. Wait
    Wait for the element identified in the CLI recipe to appear.
  4. Do
    Reload the page.
  5. Do
    Press Escape.

Expect

Set as active toasts Switched to profile "<name>", moves the Default badge and is disabled on the active row.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=agent-profile-row >> has-text=QA_oh >> testid=agent-profile-menu-trigger', control-openhands browser click 'testid=agent-profile-set-active' and control-openhands browser wait-text 'Switched to profile "QA_oh"' --timeout 10000. After browser reload, browser count 'testid=agent-profile-row >> has-text=QA_oh >> testid=agent-profile-active-badge' and browser count 'testid=agent-profile-active-badge' are both 1. Reopen the QA_oh menu: browser enabled 'testid=agent-profile-set-active' is false (screenshot --feature F13.set-active --name menu); control-openhands browser press Escape closes it (browser count 'testid=agent-profile-actions-menu' is 0).

Pill wins over a pinned LLMF13.llm-pill-precedenceMapped behavior

Scenario F13--pill-wins-over-a-pinned-llm · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Start a conversation with the prompt “Reply with only: qa-profile-ok. Do not run tools.”.

Expect

when the active profile pins a different LLM profile than the Home LLM pill, a backend advertising per-launch LLM override uses the pill's LLM while preserving the named profile's other settings and identity.

Exact CLI recipe & bug checks

With QA_oh active (pinned to deepseek-pro) and the Home pill on deepseek-flash (browser text 'testid=chat-input-llm-profile' on /), run control-openhands conversation start --prompt "Reply with only: qa-profile-ok. Do not run tools." --wait --timeout 240. The output's model is deepseek/deepseek-flash. On the current backend, control-openhands api GET /api/conversations/<id> (id from that output) has launched_agent_profile.agent_profile_id equal to QA_oh's id, launched_agent_profile.llm_profile_ref deepseek-flash, and agent.tool_concurrency_limit 2: the per-launch LLM override keeps the named profile's other settings. These metadata checks can be made before a successful reply; a refused-loopback fixture proves the launch configuration only.

Active profile drives the launchF13.active-drives-conversationMapped behavior

Scenario F13--active-profile-drives-the-launch · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/agents.
  2. Do
    Start a conversation as specified in the recipe.
  3. Do
    Choose “deepseek-flash (deepseek/deepseek-flash)” in “Agent profile llm selector”.
  4. Do
    Start a conversation as specified in the recipe.

Expect

a conversation started from Home after activation launches from that profile (its id and settings are stamped on the conversation).

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/agents (the previous conversation start left the browser on the conversation), edit QA_oh, run control-openhands browser choose 'testid=agent-profile-llm-selector' 'deepseek-flash (deepseek/deepseek-flash)' (the toggle-button opener failed 5 of 5 times here: see Gotchas), check browser value 'testid=agent-profile-llm-selector', Save (Profile "QA_oh" updated). Run the same conversation start again; api GET /api/conversations/<id> now has launched_agent_profile.agent_profile_id equal to QA_oh's id from api GET /api/agent-profiles and agent.tool_concurrency_limit 2; control-openhands conversation events <id> --kinds MessageEvent contains qa-profile-ok.

MCP scopeF13.mcp-scopeMapped behavior

Scenario F13--mcp-scope · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/agents.
  2. Do
    Choose “Agent settings mcp list” / “qa_mcp_b”.
  3. Setup
    Arrange the prerequisite backend state through the API.
  4. Do
    Reload the page.
  5. Do
    Open /.
  6. Do
    Reload the page.
  7. Do
    Enter “Reply with only: qa-dangling” in “Message”.
  8. Do
    Choose “Submit”.
  9. Do
    Enter “” in “Message”.
  10. Do
    Open /settings/agents.
  11. Do
    Choose “Agent settings mcp list” / “qa_mcp_a”.

Expect

MCP servers offers All servers (every configured server listed, switched on and disabled) or Choose servers (a switch per configured server); the choice persists; a chosen server that was removed stays listed with a red warning, and launching with it is refused.

Exact CLI recipe & bug checks

With qa_mcp_a and qa_mcp_b arranged, browser goto /settings/agents, Edit QA_oh: browser value 'testid=agent-settings-mcp-mode' is All servers and the eval below (with +(i.disabled?'(disabled)':'') appended) shows both =true(disabled). Pick Choose servers from testid=agent-settings-mcp-mode; control-openhands browser eval "[...document.querySelectorAll('[data-testid=agent-settings-mcp-list] input')].map(i=>i.dataset.testid+'='+i.checked)" shows both =true. Run control-openhands browser click 'testid=agent-settings-mcp-list >> text=qa_mcp_b' (now false), Save; api GET /api/agent-profiles/QA_oh has mcp_server_refs ["qa_mcp_a"]. Dangling: arrange control-openhands api DELETE /api/settings/mcp/qa_mcp_a --write, browser reload, Edit QA_oh: control-openhands browser text 'testid=agent-settings-mcp-dangling' is qa_mcp_a is no longer configured. Saving with it selected will stop this agent from starting — clear it or re-add the server. (screenshot --feature F13.mcp-scope --name dangling). Leave without saving, browser goto /, browser reload, control-openhands browser fill 'testid=chat-input' 'Reply with only: qa-dangling', control-openhands browser click 'testid=submit-button' --observe 'role=status' --observe-ms 4000 and control-openhands browser toasts --history (QA_oh must still be active; on older backends without per-launch LLM override it must also be pinned to the pill's LLM): the launch is refused (one POST /api/conversations 422 in browser errors --app-only; earlier failures in the session are listed too): browser toasts --history shows Creating conversation…, then one toast carrying the server's detail.message (MCP server(s) not configured: qa_mcp_a on Agent Server 1.53.0), and browser text 'testid=chat-input' still reads Reply with only: qa-dangling. Clear the composer with browser fill 'testid=chat-input' '', then clear the ref: control-openhands browser goto /settings/agents, Edit QA_oh, browser click 'testid=agent-settings-mcp-list >> text=qa_mcp_a' (the warning count drops to 0), pick All servers, Save; mcp_server_refs is null.

Credential fields per presetF13.acp-preset-credentialsMapped behavior

Scenario F13--credential-fields-per-preset · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “ACP (external subprocess)” in “Agent type selector”.
  2. Do
    Choose “<preset>” in “Agent preset selector”.
  3. Wait
    Wait for “Settings acp auth checking” to disappear.
  4. Do
    Scroll “Settings acp secret google application credentials json” as specified in the recipe.

Expect

the Credentials fields follow the preset: Claude Code CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL; Codex OPENAI_API_KEY, CODEX_AUTH_JSON (multi-line), OPENAI_BASE_URL; Gemini CLI GOOGLE_APPLICATION_CREDENTIALS_JSON (multi-line), GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION, GOOGLE_GENAI_USE_VERTEXAI, GEMINI_API_KEY, GEMINI_BASE_URL; Pi PI_AUTH_JSON (multi-line), ANTHROPIC_API_KEY; OpenCode OPENCODE_AUTH_CONTENT (multi-line), OPENCODE_API_KEY; Custom has no Credentials section. With no host login and no stored credential (Gemini here) no auth banner shows.

Exact CLI recipe & bug checks

In the Add editor, control-openhands browser choose 'testid=agent-type-selector' 'ACP (external subprocess)', then for each of Codex, Gemini CLI and Custom run control-openhands browser choose 'testid=agent-preset-selector' '<preset>' and control-openhands browser testids --hidden --filter settings-acp: Codex lists settings-acp-secret-OPENAI_API_KEY (input), -CODEX_AUTH_JSON (textarea), -OPENAI_BASE_URL; Gemini CLI lists -GOOGLE_APPLICATION_CREDENTIALS_JSON (textarea), -GOOGLE_CLOUD_PROJECT, -GOOGLE_CLOUD_LOCATION, -GOOGLE_GENAI_USE_VERTEXAI, -GEMINI_API_KEY, -GEMINI_BASE_URL; Custom lists nothing and browser eval "document.body.innerText.includes('Credentials')" is false. For Gemini, control-openhands browser wait 'testid=settings-acp-auth-checking' --state detached --timeout 15000, then browser testids --hidden --filter settings-acp-auth has count 0 (no login, no stored credential); browser eval "document.querySelector('[data-testid=settings-acp-secret-GEMINI_API_KEY]').type" is password, GOOGLE_CLOUD_PROJECT is text. Screenshot --feature F13.acp-preset-credentials --name gemini after browser scroll 'testid=settings-acp-secret-GOOGLE_APPLICATION_CREDENTIALS_JSON'. Cancel (Codex shows settings-acp-auth-configured instead while OPENAI_API_KEY from F13.acp-credentials is still saved).

Menu keyboardF13.menu-keyboardMapped behavior

Scenario F13--menu-keyboard · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/agents.
  2. Do
    Choose “Add agent profile”.
  3. Do
    Enter “QA_kb” in “Agent profile name”.
  4. Do
    Choose “Save agent profile btn”.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Choose “Agent profile row” / containing “QA_kb” / “Agent profile menu trigger”.
  7. Do
    Press ArrowDown.
  8. Do
    Press ArrowUp.
  9. Do
    Press Escape.
  10. Do
    Press Tab.
  11. Do
    Use browser focus as specified in the exact recipe.
  12. Do
    Press Enter.
  13. Do
    Use browser mouse click as specified in the exact recipe.
  14. Do
    Choose the element identified in the CLI recipe.
  15. Do
    Press ArrowDown.
  16. Do
    Press ArrowUp.
  17. Setup
    Use the phone viewport.
  18. Do
    Use browser focus as specified in the exact recipe.
  19. Do
    Press Enter.
  20. Do
    Choose “Cancel agent profile btn”.
  21. Do
    Choose “Agent profile delete”.
  22. Do
    Choose “Delete agent profile confirm”.
  23. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

opening the row's ... menu focuses Edit; ArrowDown/ArrowUp move focus between the enabled items (wrapping, and skipping the disabled Set as active on the active row), Enter runs the focused item, and Tab, Escape or a click outside close the menu.

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/agents. It needs a non-active row: run control-openhands browser click 'testid=add-agent-profile', control-openhands browser fill 'testid=agent-profile-name-input' QA_kb, control-openhands browser click 'testid=save-agent-profile-btn' and control-openhands browser wait-text 'Profile "QA_kb" created' --timeout 10000. Open its menu with control-openhands browser click 'testid=agent-profile-row >> has-text=QA_kb >> testid=agent-profile-menu-trigger' and right away run control-openhands browser eval "document.activeElement?.dataset?.testid": agent-profile-edit (opening the menu moves focus to its first enabled item, as the LLM profile menu does in F10.actions-menu). Then control-openhands browser press ArrowDown followed by that eval four times: agent-profile-set-active, agent-profile-delete, agent-profile-edit, agent-profile-set-active; browser press ArrowUp returns to agent-profile-edit. control-openhands browser press Escape closes the menu (control-openhands browser count 'testid=agent-profile-actions-menu' is 0) and returns focus to the row's own trigger (control-openhands browser eval "(()=>{const a=document.activeElement;const r=a.closest('[data-testid=agent-profile-row]');return a.dataset.testid+'@'+(r?r.innerText.split('\\n')[0]:'-')})()" is agent-profile-menu-trigger@QA_kb); reopen it and control-openhands browser press Tab: closed, and focus moves on past the trigger to the next row's trigger; keyboard open (control-openhands browser focus on the trigger, then browser press Enter) also lands on agent-profile-edit; reopen it and run control-openhands browser mouse-click 900 600: closed. Active row: open its menu with control-openhands browser click '[data-testid=agent-profile-row]:has([data-testid=agent-profile-active-badge]) >> testid=agent-profile-menu-trigger' (control-openhands browser snapshot 'testid=agent-profile-actions-menu' shows menuitem "Set as active" [disabled]; focus is already on agent-profile-edit), then control-openhands browser press ArrowDown and the eval, then control-openhands browser press ArrowUp and the eval: agent-profile-delete (the disabled Set as active is skipped), then agent-profile-edit (control-openhands browser screenshot --feature F13.menu-keyboard --name active-row-menu). At browser viewport phone the menu opens inside the viewport with the same focus order. With the menu still open, run control-openhands browser focus 'testid=agent-profile-edit' and control-openhands browser press Enter: control-openhands browser text 'testid=agent-profile-editor-title' is Edit agent profile. Leave the editor with control-openhands browser click 'testid=cancel-agent-profile-btn' (browser count 'testid=add-agent-profile' is 1). Delete QA_kb: reopen its menu, control-openhands browser click 'testid=agent-profile-delete', control-openhands browser click 'testid=delete-agent-profile-confirm' and control-openhands browser wait-text 'Profile "QA_kb" deleted' --timeout 10000.

DeleteF13.deleteMapped behavior

Scenario F13--delete · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Agent profile row” / containing “QA_acp_renamed” / “Agent profile menu trigger”.
  2. Do
    Choose “Agent profile delete”.
  3. Do
    Choose the dialog / “Cancel”.
  4. Do
    Choose “Delete agent profile confirm”.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Reload the page.
  7. Do
    Reload the page.

Expect

Delete asks for confirmation; Cancel keeps the row, Delete removes it. Deleting the active profile selects default immediately; deleting the last profile makes the server re-seed and activate default.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=agent-profile-row >> has-text=QA_acp_renamed >> testid=agent-profile-menu-trigger', control-openhands browser click 'testid=agent-profile-delete'; control-openhands browser text 'role=dialog' reads Delete Profile and Are you sure you want to delete the profile "QA_acp_renamed"? This action cannot be undone. control-openhands browser click 'role=dialog >> role=button[name="Cancel"]' keeps the row (count 1). Repeat and control-openhands browser click 'testid=delete-agent-profile-confirm', browser wait-text 'Profile "QA_acp_renamed" deleted' --timeout 10000, browser reload: count 0. Delete the active QA_oh the same way: after browser reload, browser count 'testid=agent-profile-active-badge' is 1, the default row has that badge, and api GET /api/agent-profiles has active_agent_profile_id equal to default's id. Delete QA_codex, QA_opencode, QA_pi and default: the list again shows one row, default / deepseek-flash / Default, with a new id (the server re-seeds it).

PhoneF13.phoneMapped behavior

Scenario F13--phone · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Use the phone viewport.
  2. Do
    Open /settings/agents.
  3. Do
    Press Escape.
  4. Do
    Scroll “Settings acp secret anthropic api key” as specified in the recipe.
  5. Setup
    Use the desktop viewport.

Expect

the list, row menu and editor fit a 390 px viewport without horizontal overflow.

Exact CLI recipe & bug checks

Run control-openhands browser viewport phone, control-openhands browser goto /settings/agents, control-openhands browser bbox 'testid=agent-profile-row' (insideViewport true, pageHorizontalOverflow false) and browser screenshot --feature F13.phone --name list. Open the row menu: browser bbox 'testid=agent-profile-actions-menu' is inside the viewport. browser press Escape, click testid=add-agent-profile: browser bbox 'testid=agent-settings-screen' is 348 px wide at x 16 with no page overflow (insideViewport is false only because the form is taller than the screen). Switch to ACP, browser scroll 'testid=settings-acp-secret-ANTHROPIC_API_KEY' and screenshot --name acp-credentials. Cancel and control-openhands browser viewport desktop.

ErrorsF13 · setup / cleanupSupport step

Scenario F13--errors · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    After the family, control-openhands browser errors --app-only shows no page errors and no [i18n] Missing translation warnings (the Parallel tool calls label and description are translated since #18035), only the deliberate 422s from the dangling-MCP launch.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

After the family, control-openhands browser errors --app-only shows no page errors and no [i18n] Missing translation warnings (the Parallel tool calls label and description are translated since #18035), only the deliberate 422s from the dangling-MCP launch.

BlockedF13.acp-conversation · F13.cloud-read-onlyKnown issueBlocked

Scenario F13--blocked · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Follow the environment or fixture command in the exact recipe.

Expect

a conversation launched from an active ACP profile runs the external agent. Blocked without an approved provider account.

on a Cloud backend, organization members see rows without the ... menu or Add agent profile. Blocked without a Cloud account.

Linked known issues

Known issue #16905: An enabled built-in OpenHands skill can be absent from a new Claude Code ACP session. The enabled/disabled instruction-delivery check needs an approved authenticated Claude Code account, its ACP subprocess and the configured Code Review automation; no provider session is authorized here. (F13.acp-conversation).

Known issue #17051: Codex ACP can work in New Chat while Issue-to-PR automation fails with missing OpenAI API credentials. Reproduction needs an approved Codex subscription login plus a configured Issue-to-PR automation and its repository credentials; the report does not identify the saved automation/profile settings. (F13.acp-conversation).

Known issue OpenHands/software-agent-sdk#5167: On a headless Docker host an already usable Codex ChatGPT login can still time out during ACP authenticate. Reproduction needs an approved ChatGPT login in the reported headless/read-only-auth-file deployment and its adapter; a desktop or bogus login cannot establish this conditional trigger. (F13.acp-conversation).

Known issue OpenHands/software-agent-sdk#5995: Claude Code ACP first turns with claude-opus-5-5 can fail because the pinned adapter bundles an older Claude Code. Blocked: needs an approved Claude login/API key entitled to Opus5.5 and a real first turn; current SDK1.55 still pins the reported adapter0.63.0. No authenticated turn was attempted. (F13.acp-conversation, F13.acp-model-discovery).

Exact CLI recipe & bug checks

Record control-openhands evidence add --feature F13.acp-conversation --result blocked ... naming the missing provider CLI/account, and the same for F13.cloud-read-only naming the Cloud member login.

CleanupF13 · setup / cleanupSupport step

Scenario F13--cleanup · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/secrets.
  2. Do
    Choose “Secret row” / containing “OPENAI_API_KEY” / “Delete secret”.
  3. Do
    Choose “Confirmation dialog” / “Confirm”.
  4. Setup
    Arrange the prerequisite backend state through the API.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

Delete the OPENAI_API_KEY credential through Secrets: control-openhands browser goto /settings/secrets, control-openhands browser click 'testid=secret-item >> has-text=OPENAI_API_KEY >> testid=delete-secret-button', control-openhands browser click 'testid=confirmation-modal >> testid=confirm-button'. Remove qa_mcp_b with control-openhands api DELETE /api/settings/mcp/qa_mcp_b --write. Leave default active.

Default wordingF13.list · F13.set-activeKnown failure

Scenario F13--default-wording · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/agents.

Expect

under Available Profiles, rows (sorted by name) show the profile name, then its LLM profile (OpenHands) or ACP (external subprocess), a Default badge on the active profile and a ... menu (Edit, Set as active, Delete).

Set as active toasts Switched to profile "<name>", moves the Default badge and is disabled on the active row.

Known bug observations

Known failure, reproduced 2026-10-12 (#17426): the badge reads Default and the menu reads Set as active. Close with Escape, reopen once to repeat, then Escape; no activation or model call is required.

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/agents, control-openhands browser text 'testid=agent-profile-active-badge', open the active row's testid=agent-profile-menu-trigger, and control-openhands browser snapshot 'testid=agent-profile-actions-menu'. Expected: the menu uses the same Default wording as the badge. Known failure, reproduced 2026-10-12 (#17426): the badge reads Default and the menu reads Set as active. Close with Escape, reopen once to repeat, then Escape; no activation or model call is required.

Seeded default key propagationF13.active-drives-conversationMapped behavior

Scenario F13--seeded-default-key-propagation · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Start a conversation with the prompt “Reply with exactly pong. Do not run tools.”.

Expect

a conversation started from Home after activation launches from that profile (its id and settings are stamped on the conversation).

Exact CLI recipe & bug checks

With the seeded default agent profile active and the disposable QA_LOOPBACK LLM profile active, control-openhands api GET /api/settings --pick llm_api_key_is_set is true. Run control-openhands conversation start --prompt 'Reply with exactly pong. Do not run tools.' --timeout 60 (the dummy endpoint refuses connections). Read only boolean presence of agent.llm.api_key in the run-owned persisted base_state.json, keeping the value private, and inspect conversation status <id>. Expected: the key is populated and initialization reaches the refused loopback endpoint rather than an absent-key error. This checks propagation before provider authentication or a successful reply.

Gemini auth file is not sufficientF13.acp-credentialsKnown failureKnown issue

Scenario F13--gemini-auth-file-is-not-sufficient · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    In the isolated run HOME only, create an empty {} file at private/home/.gemini/oauth_creds.json (never replace a real host login). Open Add agent profile → ACP → Gemini CLI and read testid=settings-acp-auth-detected; Cancel and repeat a fresh Add form. Expected: no authenticated banner for an empty file. Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#4629): You're already signed in to Gemini CLI — you can leave the fields below blank. No provider or model call is needed. Remove the own dummy auth file; SDK auth-method priority and upstream CLI-version branches remain outside this check.

Expect

built-in presets show a Credentials section (password fields), an auth banner (signed in / configured), a conflict warning for mutually exclusive Claude credentials, and Already saved — leave blank to keep for stored values. Credentials are saved as global secrets.

Known bug observations

Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#4629): You're already signed in to Gemini CLI — you can leave the fields below blank. No provider or model call is needed. Remove the own dummy auth file; SDK auth-method priority and upstream CLI-version branches remain outside this check.

Linked known issues

Known issue #16296: Claude Code credential fields remain visible below the already-connected banner, leading users to think keys are required. The connected-state check needs an approved existing Claude CLI login on the agent-server host; a dummy credential does not prove that signed-in state. (F13.acp-credentials).

Exact CLI recipe & bug checks

In the isolated run HOME only, create an empty {} file at private/home/.gemini/oauth_creds.json (never replace a real host login). Open Add agent profile → ACP → Gemini CLI and read testid=settings-acp-auth-detected; Cancel and repeat a fresh Add form. Expected: no authenticated banner for an empty file. Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#4629): You're already signed in to Gemini CLI — you can leave the fields below blank. No provider or model call is needed. Remove the own dummy auth file; SDK auth-method priority and upstream CLI-version branches remain outside this check.

Published stale ACP binary on PATHF13.acp-conversationKnown failureKnown issueBlocked

Scenario F13--published-stale-acp-binary-on-path · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Install the real published @zed-industries/codex-acp@0.16.0 under the owned run's workspace/qa-stale-adapter (no global install). Create/activate a real Codex ACP profile through Settings > Agents with only a dummy API-key presence marker and loopback Base URL. Prepend that package's node_modules/.bin only during the own backend restart; keep its private HOME, set default OpenAI routes plus HTTP(S)/ALL proxy to refused loopback and NO_PROXY=127.0.0.1,localhost, with cached dependencies offline. Doctor, then start two actual Home chats and inspect the sanitized runtime command and initialization version. Expected: the SDK uses the pinned 1.10.0 adapter. Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#5466): acp_command became bare codex-acp, and actual ACP initialization reported agent_version=0.16.0; the version warning still wrongly blamed npx fallback. Both then stopped at Authentication required. This proves selection/warning before authentication, not a usable session, the historical model-cache error or the Claude counterpart. Restore the launcher environment/PATH and default agent, delete the own profile/dummy secrets/package fixture, then doctor and stop.

Expect

a conversation launched from an active ACP profile runs the external agent. Blocked without an approved provider account.

Known bug observations

Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#5466): acp_command became bare codex-acp, and actual ACP initialization reported agent_version=0.16.0; the version warning still wrongly blamed npx fallback. Both then stopped at Authentication required. This proves selection/warning before authentication, not a usable session, the historical model-cache error or the Claude counterpart. Restore the launcher environment/PATH and default agent, delete the own profile/dummy secrets/package fixture, then doctor and stop.

Linked known issues

Known issue #16905: An enabled built-in OpenHands skill can be absent from a new Claude Code ACP session. The enabled/disabled instruction-delivery check needs an approved authenticated Claude Code account, its ACP subprocess and the configured Code Review automation; no provider session is authorized here. (F13.acp-conversation).

Known issue #17051: Codex ACP can work in New Chat while Issue-to-PR automation fails with missing OpenAI API credentials. Reproduction needs an approved Codex subscription login plus a configured Issue-to-PR automation and its repository credentials; the report does not identify the saved automation/profile settings. (F13.acp-conversation).

Known issue OpenHands/software-agent-sdk#5167: On a headless Docker host an already usable Codex ChatGPT login can still time out during ACP authenticate. Reproduction needs an approved ChatGPT login in the reported headless/read-only-auth-file deployment and its adapter; a desktop or bogus login cannot establish this conditional trigger. (F13.acp-conversation).

Known issue OpenHands/software-agent-sdk#5995: Claude Code ACP first turns with claude-opus-5-5 can fail because the pinned adapter bundles an older Claude Code. Blocked: needs an approved Claude login/API key entitled to Opus5.5 and a real first turn; current SDK1.55 still pins the reported adapter0.63.0. No authenticated turn was attempted. (F13.acp-conversation, F13.acp-model-discovery).

Exact CLI recipe & bug checks

Install the real published @zed-industries/codex-acp@0.16.0 under the owned run's workspace/qa-stale-adapter (no global install). Create/activate a real Codex ACP profile through Settings > Agents with only a dummy API-key presence marker and loopback Base URL. Prepend that package's node_modules/.bin only during the own backend restart; keep its private HOME, set default OpenAI routes plus HTTP(S)/ALL proxy to refused loopback and NO_PROXY=127.0.0.1,localhost, with cached dependencies offline. Doctor, then start two actual Home chats and inspect the sanitized runtime command and initialization version. Expected: the SDK uses the pinned 1.10.0 adapter. Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#5466): acp_command became bare codex-acp, and actual ACP initialization reported agent_version=0.16.0; the version warning still wrongly blamed npx fallback. Both then stopped at Authentication required. This proves selection/warning before authentication, not a usable session, the historical model-cache error or the Claude counterpart. Restore the launcher environment/PATH and default agent, delete the own profile/dummy secrets/package fixture, then doctor and stop.

Suffixed profile keeps its identityF13.editKnown failure

Scenario F13--suffixed-profile-keeps-its-identity · Permalink · Source

Given

Use F13’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    A normal edit preserves the profile UUID and increments its revision. Known failure (reproduced 2026-10-16, OpenHands/software-agent-sdk#5873): two real concurrency-only saves of a .json name each minted a new UUID at revision0. SDK1.55 repaired the active pointer to default; do not claim it remained dangling. The credential-free block checks the revision symptom and cleans its own fixtures.

Expect

Edit reopens the stored values with Save disabled until something changes; renaming keeps the profile id; Back and Cancel leave without saving.

Known bug observations

Known failure (reproduced 2026-10-16, OpenHands/software-agent-sdk#5873): two real concurrency-only saves of a .json name each minted a new UUID at revision0. SDK1.55 repaired the active pointer to default; do not claim it remained dangling. The credential-free block checks the revision symptom and cleans its own fixtures.

Exact CLI recipe & bug checks

A normal edit preserves the profile UUID and increments its revision. Known failure (reproduced 2026-10-16, OpenHands/software-agent-sdk#5873): two real concurrency-only saves of a .json name each minted a new UUID at revision0. SDK1.55 repaired the active pointer to default; do not claim it remained dangling. The credential-free block checks the revision symptom and cleans its own fixtures.

sh known-bug F13.edit json-suffix

  trap 'control-openhands api DELETE /api/agent-profiles/QA_KNOWN_SUFFIX --write >/dev/null' EXIT
  control-openhands browser viewport desktop
  control-openhands browser goto /settings/agents
  control-openhands browser click 'testid=add-agent-profile'
  control-openhands browser fill 'testid=agent-profile-name-input' QA_KNOWN_SUFFIX.json
  control-openhands browser click 'testid=save-agent-profile-btn'
  control-openhands browser wait-text 'Profile "QA_KNOWN_SUFFIX.json" created' --timeout 10000
  control-openhands assert --field value --equals 0 -- api GET /api/agent-profiles/QA_KNOWN_SUFFIX --pick profile.revision
  control-openhands browser click 'testid=agent-profile-row >> has-text=QA_KNOWN_SUFFIX >> testid=agent-profile-menu-trigger'
  control-openhands browser click 'testid=agent-profile-edit'
  control-openhands browser fill 'testid=sdk-settings-tool_concurrency_limit' 2
  control-openhands browser click 'testid=save-agent-profile-btn'
  control-openhands browser wait-text 'Profile "QA_KNOWN_SUFFIX.json" updated' --timeout 10000
  control-openhands assert --field value --equals 1 --bug-shows 0 -- api GET /api/agent-profiles/QA_KNOWN_SUFFIX --pick profile.revision  # bug
Family notes, prerequisites for known issues & limits Known issueBlocked
  • The editor is not a route: browser url stays /settings/agents and browser reload drops unsaved edits back to the list. The settings page header (settings-page-subtitle) is hidden while it is open.
  • Autocomplete dropdowns: clicking an input that already shows a value can open an empty listbox (the text filters it), and an option click then times out. The toggle-button opener does not always open the list either: on the Edit editor's agent-profile-llm-selector (bottom of the form) the option click timed out 5 of 5 times right after opening the editor, and the model list's Custom option once detached mid-click after a preset change. browser choose avoids the input-filtering problem, but an option may still remain unstable immediately after opening an editor. Reload and reopen before retrying; the suffix-identity check can use the already-selected active LLM without changing that dropdown. A listbox also closes if you spend a command (snapshot, eval) in between, so click the option right after opening.
  • A taken name marks the field invalid with an alert reading An agent profile with this name already exists. ….
  • has-text=default also matches the active row's Default badge (Playwright's text match ignores case), so with several profiles 'testid=agent-profile-row >> has-text=default' hits two rows; select the seeded profile with 'testid=agent-profile-row >> nth=-1' (it sorts last) or by its badge.
  • Model lookups (F13.acp-model-discovery) spawn the provider CLI on the agent-server machine through npx with the profile's allowed secrets; they send no prompt. A failed lookup is retried each time the preset is picked again; a successful one is cached for about five minutes per preset and secret set.
  • Switch rows are sr-only checkboxes inside labels (keyboard reachable): click the label text ('testid=agent-settings-tool-list >> text=<tool>', 'testid=agent-settings-mcp-list >> text=<name>') and read .checked with browser eval. A bare label selector segment is parsed as label=, not CSS.
  • The Default badge, the Set as active item and the Switched to profile toast all refer to the same active pointer; the wording differs from the LLM page (Set as default).
  • The OpenHands row's grey text is the LLM profile name, not the kind. A fresh run's default shows default there although no LLM profile of that name exists until it is edited (F13.stale-llm-ref).
  • Local seeded OpenHands default launches through global agent settings so the Skills selection applies; its per-profile fields do not apply on that path. Named profiles use their saved settings. Home launches follow the LLM pill. On a backend advertising per-launch LLM override, a named profile pinned to another LLM keeps its tool, concurrency and MCP settings; read launched_agent_profile and agent on the conversation (F13.llm-pill-precedence). Only older backends without that capability use the legacy settings fallback when the profile and pill differ.
  • The ACP auth probe runs the provider CLI on the agent-server machine. If that machine has a Claude CLI login, Claude Code can show "already signed in". Use only an explicitly approved account for authenticated ACP checks.
  • ACP credentials are global secrets, not profile fields: cancelling the editor saves nothing, but saving any ACP profile with a typed credential writes it to Settings → Secrets (F14) for every conversation.
  • Choose secrets on an ACP profile also selects provider secret names that are not saved (CODEX_AUTH_JSON, OPENAI_BASE_URL); they show as switches without descriptions.
  • The empty state (agent-profiles-empty) and the load-error state (agent-profiles-load-error) are not reachable on a local backend: deleting the last profile re-seeds default, and with the agent-server stopped the app swaps to Manage backends after a reload, or keeps the cached list without one.
  • The refused-launch toast text comes from the Agent Server and changes between versions (MCP server(s) not configured: <name> on 1.53.0): assert that one toast names the server and the prompt stays, not the exact wording.
  • Opening a row menu focuses its first enabled item. Arrows skip disabled Set as active; Escape closes the menu and returns focus to its trigger, while Tab closes it and continues normal tab order.
  • The Add agent editor has its own agent description. Parallel tool calls label and description are translated.
  • Known issue #16905: An enabled built-in OpenHands skill can be absent from a new Claude Code ACP session. The enabled/disabled instruction-delivery check needs an approved authenticated Claude Code account, its ACP subprocess and the configured Code Review automation; no provider session is authorized here. (F13.acp-conversation).
  • Known issue #16296: Claude Code credential fields remain visible below the already-connected banner, leading users to think keys are required. The connected-state check needs an approved existing Claude CLI login on the agent-server host; a dummy credential does not prove that signed-in state. (F13.acp-credentials).
  • Known issue #17051: Codex ACP can work in New Chat while Issue-to-PR automation fails with missing OpenAI API credentials. Reproduction needs an approved Codex subscription login plus a configured Issue-to-PR automation and its repository credentials; the report does not identify the saved automation/profile settings. (F13.acp-conversation).
  • Known issue OpenHands/software-agent-sdk#5167: On a headless Docker host an already usable Codex ChatGPT login can still time out during ACP authenticate. Reproduction needs an approved ChatGPT login in the reported headless/read-only-auth-file deployment and its adapter; a desktop or bogus login cannot establish this conditional trigger. (F13.acp-conversation).
  • Known issue OpenHands/software-agent-sdk#5995: Claude Code ACP first turns with claude-opus-5-5 can fail because the pinned adapter bundles an older Claude Code. Blocked: needs an approved Claude login/API key entitled to Opus5.5 and a real first turn; current SDK1.55 still pins the reported adapter0.63.0. No authenticated turn was attempted. (F13.acp-conversation, F13.acp-model-discovery).
  • Known issue OpenHands/software-agent-sdk#5899: The browser tool can be reported unusable when genuine Chromium exists only under PLAYWRIGHT_BROWSERS_PATH. Blocked on this host: it has standard user Chrome. The distinguishing fixture needs a host/container with no standard/default-cache/PATH browser, plus a genuine custom-cache Chromium; never remove the user browser to create it. (F13.openhands-options; operational browser path F27).

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.