Engel’s notebook

Provider connections

F11 · 21 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F11 Provider connections

21 behaviors · 27 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

A provider connection is a named, shared credential (provider, API key, optional base URL) that many LLM profiles link to instead of carrying their own key. Under Settings → LLM, below Available Profiles, a user lists connections with their linked-model count, adds them, edits them (rename, provider, base URL, key rotation with a blank key meaning "keep"), bulk-adds linked profiles from a row's menu, and deletes them; the server refuses to delete a connection that a profile or the active settings still reference.

How a user gets here

  • Sidebar settings gear (backend-selector-settings-link), then LLM in the settings navigation (sidebar-settings-/settings/llm); the section sits below Available Profiles.
  • Direct URL /settings/llm.
  • Command menu (Control+k/Meta+k, or command-menu-trigger): search LLM, choose LLM profiles. Searching provider connection finds nothing.
  • Add connection (add-provider-connection) opens the create modal; each row's ... (provider-connection-menu-trigger) opens Bulk add / Edit / Delete.
  • Elsewhere (other families): Model Router's editor has its own Add connection (meta-profile-add-provider-connection, see F12); the LLM profile editor links a profile to a connection and Add from provider connections bulk-adds without a preselected connection (see F10).

Given

Preconditions:

  • A fresh control-openhands launch --new, doctored, onboard --skip done, desktop viewport. Do not run llm preset deepseek before this family: the recipes need no LLM profile and no connection (control-openhands api GET /api/profiles lists none, control-openhands api GET /api/llm/provider-connections is []), so the profile added in Bulk add becomes the active one.
  • A DeepSeek key in a file (<key file>). Pass it only with browser fill ... --value-file <key file>; never on the command line.
  • F11.agent-uses-connection and F11.rotate start three tiny conversations on deepseek/deepseek-flash.
  • F11.load-error is arranged by making the run's own connections store unreadable (a shell arrange step that writes only $OH_VERIFY_RUN/private/provider-connections/provider_connections.json, restored afterwards); service stop agent-server does not work because it replaces the whole page with agent-server-onboarding-screen.
  • Blocked here: F11.cloud (needs a Cloud account with and without an organization, as owner and as member).
Empty stateF11.list-emptyMapped behavior

Scenario F11--empty-state · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Settings”.
  2. Do
    Choose “Llm settings”.

Expect

with no connections the section shows the heading Provider connections, the subline Share one API key across multiple models., Add connection and No provider connections yet. Add one to share an API key across models.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=backend-selector-settings-link' --expect-url '/settings', control-openhands browser click 'testid=sidebar-settings-/settings/llm' --expect-url '/settings/llm', then control-openhands browser text 'testid=provider-connections-empty' and control-openhands browser snapshot main. The text is No provider connections yet. Add one to share an API key across models.; the snapshot shows heading Provider connections, paragraph Share one API key across multiple models. and button Add connection (screenshot --feature F11.list-empty --name empty).

Other entry pointsF11.listMapped behavior

Scenario F11--other-entry-points · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /.
  2. Do
    Press Control+k.
  3. Do
    Enter “LLM” in “Command menu” / the combobox.
  4. Do
    Press Enter.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Open /settings/llm.

Expect

each row shows the name, the provider, <n> model(s) (profiles linked to it), a green key-set icon and a ... menu labelled Provider connection menu; the section is reachable from the settings nav, the command menu and the URL.

Exact CLI recipe & bug checks

Run control-openhands browser goto / (Empty state ends on /settings/llm, where the URL wait would pass at once), control-openhands browser press Control+k, control-openhands browser type 'testid=command-menu >> role=combobox' LLM (browser snapshot 'testid=command-menu' shows the selected option LLM profiles Manage models, providers, and API keys. Go), control-openhands browser press Enter, control-openhands browser wait-url '/settings/llm', then control-openhands browser count 'testid=add-provider-connection' (1). control-openhands browser goto /settings/llm shows the same section.

Validation and provider pickerF11.create-validation · F11.provider-pickerMapped behavior

Scenario F11--validation-and-provider-picker · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add provider connection”.
  2. Do
    Enter “QA_conn” in “Provider connection name”.
  3. Do
    Choose “Provider connection provider”.
  4. Do
    Enter “deep” in “Provider connection provider”.
  5. Do
    Choose “Provider item deepseek”.
  6. Do
    Enter “ ” in “Provider connection api key”.
  7. Do
    Enter “deepse” in “Provider connection provider”.
  8. Do
    Choose “Provider connection name”.
  9. Do
    Press Enter.

Expect

in Add provider connection focus starts on Name; Save stays disabled until Name, a picked Provider and API Key are non-blank (whitespace does not count); a provider typed but not picked is cleared; Enter does not submit.

the Provider combobox lists Verified Models (OpenHands, Anthropic, OpenAI, …, deepseek, …) and Other Models groups and filters as you type.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=add-provider-connection'; control-openhands browser snapshot 'role=dialog' shows Add provider connection with Name, Provider, API Key, Base URL Optional and Save [disabled], and control-openhands browser eval "document.activeElement.getAttribute('data-testid')" is provider-connection-name-input. Fill control-openhands browser fill 'testid=provider-connection-name-input' QA_conn; control-openhands browser enabled 'testid=provider-connection-submit' stays false. Run control-openhands browser click 'testid=provider-connection-provider-input' and control-openhands browser snapshot 'role=listbox': groups Verified Models (OpenHands, Anthropic, OpenAI, Mistral AI, Gemini, deepseek, Moonshot, minimax, glm, nvidia, qwen, OpenRouter; OpenRouter is listed since 2026-10-08 or earlier) and Other Models. The field shows the provider's label, not its id: picking testid=provider-item-openai makes browser value 'testid=provider-connection-provider-input' OpenAI. control-openhands browser fill 'testid=provider-connection-provider-input' deep narrows it to deepseek (verified) and deepgram, DeepInfra; then control-openhands browser click 'testid=provider-item-deepseek' and control-openhands browser value 'testid=provider-connection-provider-input' is deepseek. Save is still disabled; control-openhands browser fill 'testid=provider-connection-api-key-input' ' ' keeps it disabled, and a name of only spaces does too. A provider typed but not picked is cleared: control-openhands browser fill 'testid=provider-connection-provider-input' deepse, control-openhands browser click 'testid=provider-connection-name-input', then browser value 'testid=provider-connection-provider-input' is empty (pick deepseek again afterwards). Enter does not submit: with every field valid (enabled true), control-openhands browser press Enter in the Name field leaves browser count 'testid=provider-connection-modal' at 1 and creates nothing (api GET /api/llm/provider-connections).

CreateF11.createMapped behavior

Scenario F11--create · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “the value specified in the recipe” in “Provider connection api key”.
  2. Do
    Choose “Provider connection submit”.
  3. Wait
    Wait for the element identified in the CLI recipe to appear.
  4. Do
    Reload the page.

Expect

Save creates the connection (toast Connection "<name>" created); the row persists after a reload and a backend restart; the key is stored but never shown.

Exact CLI recipe & bug checks

In the same modal run control-openhands browser fill 'testid=provider-connection-api-key-input' --value-file <key file> (enabled turns true, browser attr 'testid=provider-connection-api-key-input' type is password), control-openhands browser click 'testid=provider-connection-submit' and control-openhands browser wait-text 'Connection "QA_conn" created'. control-openhands browser count 'testid=provider-connection-modal' is 0. After control-openhands browser reload, control-openhands browser text 'testid=provider-connection-row >> has-text=QA_conn' is QA_conn\ndeepseek\n0 model(s) and control-openhands api GET /api/llm/provider-connections lists it with "api_key_set": true and "base_url": null (the key itself is never returned).

RowF11.listMapped behavior

Scenario F11--row · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Run control-openhands browser count 'testid=provider-connection-row >> has-text=QA_conn >> testid=set-indicator' (1), control-openhands browser attr 'testid=provider-connection-row >> has-text=QA_conn >> testid=provider-connection-menu-trigger' aria-label (Provider connection menu) and control-openhands browser screenshot 'testid=provider-connection-row' --feature F11.list --name row (name, muted provider, 0 model(s), green check, vertical dots).

Expect

each row shows the name, the provider, <n> model(s) (profiles linked to it), a green key-set icon and a ... menu labelled Provider connection menu; the section is reachable from the settings nav, the command menu and the URL.

Exact CLI recipe & bug checks

Run control-openhands browser count 'testid=provider-connection-row >> has-text=QA_conn >> testid=set-indicator' (1), control-openhands browser attr 'testid=provider-connection-row >> has-text=QA_conn >> testid=provider-connection-menu-trigger' aria-label (Provider connection menu) and control-openhands browser screenshot 'testid=provider-connection-row' --feature F11.list --name row (name, muted provider, 0 model(s), green check, vertical dots).

CancelF11.create-cancelMapped behavior

Scenario F11--cancel · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add provider connection”.
  2. Do
    Choose the dialog / “Cancel”.
  3. Do
    Enter “QA_escape” in “Provider connection name”.
  4. Do
    Press Escape.
  5. Do
    Use browser mouse click as specified in the exact recipe.

Expect

Cancel, Escape and a backdrop click close the modal without creating anything; reopening starts empty.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=add-provider-connection', control-openhands browser click 'role=dialog >> role=button[name="Cancel"]'; browser count 'testid=provider-connection-modal' is 0. Reopen, control-openhands browser fill 'testid=provider-connection-name-input' QA_escape, control-openhands browser press Escape (count 0); reopen: control-openhands browser value 'testid=provider-connection-name-input' is empty; control-openhands browser mouse-click 50 500 closes it too. api GET /api/llm/provider-connections lists no QA_escape.

Server errorF11.create-errorMapped behavior

Scenario F11--server-error · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add provider connection”.
  2. Do
    Enter “QA_$(printf 'x%.0s' $(seq 1 127))” in “Provider connection name”.
  3. Do
    Enter “dummy-key-3” in “Provider connection api key”.
  4. Do
    Choose “Provider connection submit”.

Expect

a server rejection (for example a name over 128 characters) keeps the modal open and toasts a readable message.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=add-provider-connection', control-openhands browser fill 'testid=provider-connection-name-input' "QA_$(printf 'x%.0s' $(seq 1 127))" (130 characters), pick deepseek as above, control-openhands browser fill 'testid=provider-connection-api-key-input' dummy-key-3, control-openhands browser click 'testid=provider-connection-submit', then control-openhands browser toasts and control-openhands browser count 'testid=provider-connection-modal'. The modal stays open (1) and the toast reads String should have at most 128 characters (pass on main@3500d9e with Agent Server 1.53.0; screenshot --feature F11.create-error --name toast). Close with role=dialog >> role=button[name="Cancel"].

Second connectionF11 · setup / cleanupSupport step

Scenario F11--second-connection · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “not a url” in “Provider connection base url”.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

Create QA_spare the same way with dummy-key-2 as key and control-openhands browser fill 'testid=provider-connection-base-url-input' 'not a url' (accepted: base URLs are not validated). Toast Connection "QA_spare" created.

Row menuF11.row-menu · F11.row-menu-focusKnown failure

Scenario F11--row-menu · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Provider connection row” / containing “QA_spare” / “Provider connection menu trigger”.
  2. Do
    Press Tab.
  3. Do
    Press ArrowDown.
  4. Do
    Press Tab.
  5. Do
    Press Escape.
  6. Do
    Choose “Share one API key across multiple models.”.

Expect

the ... menu offers Bulk add, Edit, Delete; Escape, a click outside or a second trigger click close it; ArrowUp/ArrowDown move and wrap; Tab closes it.

opening the menu moves focus to its first item (Bulk add).

Known bug observations

Known failure (reproduced twice 2026-10-16, #18204): it stays provider-connection-menu-trigger and ArrowDown does nothing until control-openhands browser press Tab moves focus to Bulk add. From there browser press ArrowDown focuses provider-connection-edit, two ArrowUp wrap to provider-connection-delete, and browser press Tab closes the menu (browser count 'testid=provider-connection-actions-menu' 0). Reopen and browser press Escape (0); reopen and click the trigger again (0); reopen and control-openhands browser click 'text=Share one API key across multiple models.' (0).

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=provider-connection-row >> has-text=QA_spare >> testid=provider-connection-menu-trigger' and control-openhands browser snapshot 'testid=provider-connection-actions-menu': menuitems Bulk add, Edit, Delete (screenshot --feature F11.row-menu --name open). control-openhands browser eval "document.activeElement.getAttribute('data-testid')" should be provider-connection-add-models; Known failure (reproduced twice 2026-10-16, #18204): it stays provider-connection-menu-trigger and ArrowDown does nothing until control-openhands browser press Tab moves focus to Bulk add. From there browser press ArrowDown focuses provider-connection-edit, two ArrowUp wrap to provider-connection-delete, and browser press Tab closes the menu (browser count 'testid=provider-connection-actions-menu' 0). Reopen and browser press Escape (0); reopen and click the trigger again (0); reopen and control-openhands browser click 'text=Share one API key across multiple models.' (0).

EditF11.editMapped behavior

Scenario F11--edit · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Provider connection actions menu” / “Provider connection edit”.
  2. Do
    Enter “QA_edited” in “Provider connection name”.
  3. Do
    Enter “https://api.deepseek.com” in “Provider connection base url”.
  4. Do
    Choose “Provider connection submit”.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Reload the page.
  7. Do
    Enter “” in “Provider connection base url”.
  8. Wait
    Wait for “Provider connection modal” to disappear.

Expect

Edit opens Edit provider connection prefilled (key blank with placeholder <hidden> and hint Leave blank to keep the current key.); rename, provider and base URL changes persist (toast Connection "<name>" updated), a cleared base URL is stored as none, a blank name disables Save, and a rename shows in the linked profiles' group header.

Exact CLI recipe & bug checks

Run the trigger click above, then control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-edit' and control-openhands browser snapshot 'role=dialog': Edit provider connection, Name QA_spare, Provider deepseek, API Key Leave blank to keep the current key. with placeholder <hidden>, Base URL not a url; browser value 'testid=provider-connection-api-key-input' is empty and Save is enabled. Run control-openhands browser fill 'testid=provider-connection-name-input' QA_edited, control-openhands browser fill 'testid=provider-connection-base-url-input' 'https://api.deepseek.com', control-openhands browser click 'testid=provider-connection-submit' and control-openhands browser wait-text 'Connection "QA_edited" updated'. After browser reload the row reads QA_edited\ndeepseek\n0 model(s), the QA_spare count is 0, and api GET /api/llm/provider-connections shows base_url https://api.deepseek.com and api_key_set true. Edit again: fill the name with ' ' (browser enabled 'testid=provider-connection-submit' false), restore QA_edited, control-openhands browser fill 'testid=provider-connection-base-url-input' '', Save, browser wait 'testid=provider-connection-modal' --state detached; the API shows base_url null.

Bulk addF11.add-modelsMapped behavior

Scenario F11--bulk-add · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Provider connection row” / containing “QA_conn” / “Provider connection menu trigger”.
  2. Do
    Choose “Provider connection actions menu” / “Provider connection add models”.
  3. Do
    Use browser uncheck as specified in the exact recipe.
  4. Do
    Use browser check as specified in the exact recipe.
  5. Do
    Choose “Add models submit”.
  6. Wait
    Wait for the element identified in the CLI recipe to appear.
  7. Do
    Reload the page.

Expect

Bulk add opens Add models as profiles with the row's provider and connection preselected; the profiles it adds are linked and the row's count goes up.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=provider-connection-row >> has-text=QA_conn >> testid=provider-connection-menu-trigger' and control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-add-models'. The dialog Add models as profiles has provider deepseek selected, control-openhands browser eval "[...document.querySelectorAll('[data-testid=add-models-connection] option')].map(o=>o.textContent+'='+o.selected).join(' | ')" is No connection (keyless)=false | QA_conn=true | QA_edited=false, and browser text 'testid=add-models-submit' is Add 12 profiles (every model checked). Run control-openhands browser uncheck 'testid=add-models-select-all' (Add 0 profiles), control-openhands browser check 'testid=add-models-check-deepseek/deepseek-flash' (Add 1 profiles), control-openhands browser click 'testid=add-models-submit' and control-openhands browser wait-text 'Added 1'. After browser reload the QA_conn row reads QA_conn\ndeepseek\n1 model(s) (QA_edited stays 0 model(s)), and api GET /api/profiles shows deepseek-flash with provider_connection_id and "active_profile": "deepseek-flash".

The agent uses the connection's keyF11.agent-uses-connectionKnown issue

Scenario F11--the-agent-uses-the-connection-s-key · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Wait
    Wait for the element identified in the CLI recipe to appear.
  2. Do
    Start a conversation with the prompt “Reply with exactly the word: pong”.

Expect

a conversation on a profile linked to the connection runs with the connection's key; an Edit with the key left blank keeps it working.

Linked known issues

Known issue OpenHands/software-agent-sdk#5207: OpenRouter linked profiles can lose connection credentials at raw agent/condenser startup or reuse stale credentials after rotation; opaque reasoning can also be lost on persisted turns. End-to-end checks need an approved OpenRouter connection and a model returning reasoning_details, including the linked-profile switch and reload path; loopback failure does not prove them. (F11.agent-uses-connection, F11.rotate).

Exact CLI recipe & bug checks

Open Edit on QA_conn (as above), leave the key blank, click testid=provider-connection-submit and browser wait-text 'Connection "QA_conn" updated'. Run control-openhands conversation start --prompt "Reply with exactly the word: pong" --wait --timeout 180 (prints <id>, "model": "deepseek/deepseek-flash", status finished), then control-openhands conversation events <id> --kinds ConversationErrorEvent,MessageEvent: the agent message is pong and there is no ConversationErrorEvent.

Rotate the keyF11.rotateKnown issue

Scenario F11--rotate-the-key · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/llm.
  2. Do
    Enter “sk-qa-invalid-0000” in “Provider connection api key”.
  3. Wait
    Wait for the element identified in the CLI recipe to appear.
  4. Do
    Start a conversation with the prompt “Reply with exactly the word: ping”.
  5. Do
    Open /settings/llm.
  6. Do
    Enter “the value specified in the recipe” in “Provider connection api key”.
  7. Do
    Start a conversation with the prompt “Reply with exactly the word: pong”.

Expect

typing a new key rotates it for new conversations: an invalid key makes the next conversation fail with the provider's authentication error and no reply; rotating back restores replies.

Linked known issues

Known issue OpenHands/software-agent-sdk#5497: after a key rotation, new conversations still run agent steps on the old key; only title generation uses the new one (F11.rotate).

Known issue OpenHands/software-agent-sdk#5207: OpenRouter linked profiles can lose connection credentials at raw agent/condenser startup or reuse stale credentials after rotation; opaque reasoning can also be lost on persisted turns. End-to-end checks need an approved OpenRouter connection and a model returning reasoning_details, including the linked-profile switch and reload path; loopback failure does not prove them. (F11.agent-uses-connection, F11.rotate).

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/llm, open Edit on QA_conn, control-openhands browser fill 'testid=provider-connection-api-key-input' sk-qa-invalid-0000, Save and browser wait-text 'Connection "QA_conn" updated'. Run control-openhands conversation start --prompt "Reply with exactly the word: ping" --wait --timeout 180 and control-openhands conversation events <id> --kinds ConversationErrorEvent,MessageEvent. Expected: a ConversationErrorEvent and no agent reply. Today the conversation shows the banner litellm.BadRequestError: DeepseekException - ... Authentication Fails, Your api key: ****0000 is invalid (so the new key was used), yet the agent still answers ping in the same turn (fail, screenshot --feature F11.rotate --name bad-key). Rotate back: browser goto /settings/llm, Edit QA_conn, control-openhands browser fill 'testid=provider-connection-api-key-input' --value-file <key file>, Save; a new conversation start --prompt "Reply with exactly the word: pong" --wait --timeout 180 has pong and no ConversationErrorEvent.

Rename a linked connectionF11.editMapped behavior

Scenario F11--rename-a-linked-connection · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/llm.
  2. Wait
    Wait for the element identified in the CLI recipe to appear.
  3. Do
    Reload the page.

Expect

Edit opens Edit provider connection prefilled (key blank with placeholder <hidden> and hint Leave blank to keep the current key.); rename, provider and base URL changes persist (toast Connection "<name>" updated), a cleared base URL is stored as none, a blank name disables Save, and a rename shows in the linked profiles' group header.

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/llm, Edit QA_conn, fill the name with QA_shared, Save and browser wait-text 'Connection "QA_shared" updated'. After browser reload, control-openhands browser text 'testid=profile-group-header' is QA_SHARED (CSS upper-case).

Delete is refused while referencedF11.delete-referencedMapped behavior

Scenario F11--delete-is-refused-while-referenced · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Provider connection row” / containing “QA_shared” / “Provider connection menu trigger”.
  2. Do
    Choose “Provider connection actions menu” / “Provider connection delete”.
  3. Do
    Choose “Delete provider connection confirm”.
  4. Do
    Reload the page.

Expect

deleting a connection that profiles or the active settings reference is refused with the server's message naming them; the dialog stays open and the row stays.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=provider-connection-row >> has-text=QA_shared >> testid=provider-connection-menu-trigger', control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-delete', control-openhands browser click 'testid=delete-provider-connection-confirm', then control-openhands browser toasts: Provider connection cannot be deleted while it is referenced by LLM profile(s): deepseek-flash and referenced by the active agent settings. Update those references before deleting it. The dialog stays open (browser count 'role=dialog' > 0); close it with role=dialog >> role=button[name="Cancel"]; after browser reload the row is still there.

DeleteF11.deleteMapped behavior

Scenario F11--delete · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Provider connection row” / containing “QA_edited” / “Provider connection menu trigger”.
  2. Do
    Choose “Provider connection actions menu” / “Provider connection delete”.
  3. Do
    Choose the dialog / “Cancel”.
  4. Do
    Press Escape.
  5. Do
    Choose “Delete provider connection confirm”.
  6. Wait
    Wait for the element identified in the CLI recipe to appear.
  7. Do
    Reload the page.

Expect

Delete asks Are you sure you want to delete the connection "<name>"? (focus on Cancel); Cancel and Escape keep it; Delete removes it (toast Connection "<name>" deleted).

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=provider-connection-row >> has-text=QA_edited >> testid=provider-connection-menu-trigger', control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-delete' and control-openhands browser snapshot 'role=dialog': Delete provider connection, Are you sure you want to delete the connection "QA_edited"?, buttons Cancel and Delete; browser eval "document.activeElement.textContent" is Cancel (screenshot --feature F11.delete --name confirm). control-openhands browser click 'role=dialog >> role=button[name="Cancel"]' keeps the row; reopen and browser press Escape keeps it too. Reopen, control-openhands browser click 'testid=delete-provider-connection-confirm', control-openhands browser wait-text 'Connection "QA_edited" deleted', browser reload: browser count 'testid=provider-connection-row >> has-text=QA_edited' is 0.

Phone layoutF11.phoneMapped behavior

Scenario F11--phone-layout · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Use the phone viewport.
  2. Do
    Scroll “Add provider connection” as specified in the recipe.
  3. Setup
    Use the desktop viewport.

Expect

at 390 px the rows, the row menu and the modals fit without horizontal overflow.

Exact CLI recipe & bug checks

Run control-openhands browser viewport phone, control-openhands browser scroll 'testid=add-provider-connection', control-openhands browser bbox 'testid=provider-connection-row >> nth=0' (insideViewport true, pageHorizontalOverflow false) and browser screenshot --feature F11.phone --name list. Open the QA_shared row menu: browser bbox 'testid=provider-connection-actions-menu' is inside the viewport. Click testid=provider-connection-actions-menu >> testid=provider-connection-edit: browser bbox 'testid=provider-connection-modal' and browser bbox 'testid=provider-connection-submit' are inside the viewport (screenshot --name edit-modal). Cancel, then control-openhands browser viewport desktop.

Long name and provider changeF11.long-name · F11.editMapped behavior

Scenario F11--long-name-and-provider-change · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Use browser tooltip as specified in the exact recipe.
  2. Setup
    Use the phone viewport.
  3. Do
    Scroll “Add provider connection” as specified in the recipe.
  4. Setup
    Use the desktop viewport.
  5. Do
    Choose “Anthropic” in “Provider connection provider”.
  6. Wait
    Wait for “Provider connection modal” to disappear.
  7. Do
    Reload the page.

Expect

a name at the 128-character limit is accepted; the row truncates the name and provider with an ellipsis, shows the full name as a hover tooltip (title) and keeps the ... trigger in view at desktop and phone width.

Edit opens Edit provider connection prefilled (key blank with placeholder <hidden> and hint Leave blank to keep the current key.); rename, provider and base URL changes persist (toast Connection "<name>" updated), a cleared base URL is stored as none, a blank name disables Save, and a rename shows in the linked profiles' group header.

Exact CLI recipe & bug checks

Create a connection named "QA_long$(printf 'x%.0s' $(seq 1 121))" (128 characters) with deepseek and dummy-key-5 as above: toast Connection "QA_long…" created, modal closed. control-openhands browser tooltip 'testid=provider-connection-row >> has-text=QA_long >> span[title]' returns the full 128-character name (source title attribute), browser eval on that span shows scrollWidth > clientWidth with text-overflow: ellipsis, and browser bbox 'testid=provider-connection-row >> has-text=QA_long' is insideViewport true, pageHorizontalOverflow false (screenshot --feature F11.long-name --name row). At browser viewport phone after browser scroll 'testid=add-provider-connection', browser bbox 'testid=provider-connection-row >> has-text=QA_long >> testid=provider-connection-menu-trigger' is inside the viewport with no horizontal overflow (screenshot --name phone); back to browser viewport desktop. Open Edit on QA_long: browser count 'testid=provider-connection-modal >> testid=set-indicator' is 1 (green key-set check beside API Key). control-openhands browser choose 'testid=provider-connection-provider-input' Anthropic, Save, browser wait 'testid=provider-connection-modal' --state detached, browser reload: the row reads QA_long…\nanthropic\n0 model(s) and api GET /api/llm/provider-connections shows "provider": "anthropic".

Persistence across a backend restartF11.createMapped behavior

Scenario F11--persistence-across-a-backend-restart · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Restart the run-owned stack as specified.
  2. Do
    Reload the page.

Expect

Save creates the connection (toast Connection "<name>" created); the row persists after a reload and a backend restart; the key is stored but never shown.

Exact CLI recipe & bug checks

Run control-openhands restart and control-openhands browser reload; browser eval "[...document.querySelectorAll('[data-testid=provider-connection-row]')].map(r=>r.innerText.replace(/\n/g,'|')).join(' ; ')" lists the same rows and counts as before (for example QA_shared|deepseek|1 model(s) ; QA_long…|anthropic|0 model(s)).

Unlisted providerF11.edit-unlisted-providerMapped behavior

Scenario F11--unlisted-provider · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Arrange the prerequisite backend state through the API.
  2. Do
    Reload the page.
  3. Do
    Choose the current page.
  4. Do
    Enter “qa_” in the element identified in the CLI recipe.

Expect

a connection whose provider is not in the provider catalog (created through the API) opens in Edit with that provider preselected and listed under Other Models, and saving keeps it.

Exact CLI recipe & bug checks

Arrange (not proof) control-openhands api POST /api/llm/provider-connections --data '{"display_name":"QA_custom","provider":"qa_unlisted","api_key":"dummy-key-6","base_url":null}' --write (201), then browser reload: the row reads QA_custom\nqa_unlisted\n0 model(s). Open Edit on QA_custom: browser value 'testid=provider-connection-provider-input' is qa_unlisted; browser click that input, browser fill ... qa_ and browser snapshot 'role=listbox' shows only group Other Models with option qa_unlisted [selected]. Click 'role=option[name="qa_unlisted"]', fill the name with QA_custom2, Save: toast Connection "QA_custom2" updated and the API still shows "provider": "qa_unlisted". Delete QA_custom2 and QA_long through their row menus and testid=delete-provider-connection-confirm (wait for the confirm button --state detached between them); api GET /api/llm/provider-connections lists only QA_shared.

Stale reference and cleanupF11.delete-stale-referenceMapped behavior

Scenario F11--stale-reference-and-cleanup · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Choose “Profile actions menu” / “Profile delete”.
  3. Do
    Choose “Delete profile confirm”.
  4. Do
    Reload the page.
  5. Setup
    Configure or inspect the model profiles specified in the recipe.
  6. Do
    Reload the page.
  7. Do
    Reload the page.

Expect

once its last linked profile is deleted (row shows 0 model(s)) the connection can be deleted, or the UI says how to clear the remaining reference.

Exact CLI recipe & bug checks

Delete the linked profile (F10): control-openhands browser click '[data-testid=profile-row]:has([title="deepseek-flash"]) >> testid=profile-menu-trigger', control-openhands browser click 'testid=profile-actions-menu >> testid=profile-delete', control-openhands browser click 'testid=delete-profile-confirm' (toast Profile "deepseek-flash" deleted). After browser reload the QA_shared row reads 0 model(s). Delete QA_shared through its row menu and testid=delete-provider-connection-confirm. Expected: it is deleted. Today the toast is Provider connection cannot be deleted while it is referenced by the active agent settings. Update those references before deleting it. and nothing on the page clears that reference (fail, screenshot --feature F11.delete-stale-reference --name refused). To finish cleanup, close the dialog, arrange an unlinked active profile with control-openhands llm preset deepseek --api-key-file <key file> (this is the arrange step, not proof), browser reload, delete QA_shared through the row menu (toast Connection "QA_shared" deleted), browser reload: testid=provider-connections-empty is back and api GET /api/llm/provider-connections is [].

Load errorF11.load-errorMapped behavior

Scenario F11--load-error · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Reload the page.
  2. Do
    Reload the page.

Expect

if listing connections fails (for example an unreadable connections store, GET /api/llm/provider-connections 400), the section shows Failed to load provider connections. (provider-connections-load-error) in red while the profiles list and Add connection still render.

Exact CLI recipe & bug checks

With the baseline restored (connections []), run the shell arrange step (it writes only this run's state; the file exists only once the run has created a connection, as the bullets above do, so on a run that never created one cp fails: create and delete one through Add connection first) F=$OH_VERIFY_RUN/private/provider-connections/provider_connections.json; cp $F $F.bak; printf '{not json' > $F. control-openhands api GET /api/llm/provider-connections is now 400 (Provider connections file is unreadable: ...) while api GET /api/profiles is 200. Run control-openhands browser reload, control-openhands browser text 'testid=provider-connections-load-error' (Failed to load provider connections.), browser count 'testid=add-provider-connection' (1) and browser count 'testid=profile-row' (> 0; screenshot --feature F11.load-error --name error). The message uses the theme's error colour (browser eval "getComputedStyle(document.querySelector('[data-testid=provider-connections-load-error] p')).color": rgb(231, 106, 94) on OpenHands-Neutral, rgb(192, 40, 45) on Light+, #18330). The profile rows render only because no profile links to a connection at this point: with a linked profile GET /api/profiles fails with the same 400 and the list above shows Failed to load profiles instead (F10.list-states). Restore with cp $F.bak $F && rm $F.bak, browser reload: testid=provider-connections-empty is back and provider-connections-load-error count is 0.

Errors sweepF11 · setup / cleanupSupport step

Scenario F11--errors-sweep · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    control-openhands browser errors --app-only shows pageErrors 0; the HTTP errors are the deliberate POST 422 (long name), two DELETE 409 (referenced, then the stale reference) and the GET 400 from the load-error arrange, all on /api/llm/provider-connections. Run browser errors --clear afterwards.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

control-openhands browser errors --app-only shows pageErrors 0; the HTTP errors are the deliberate POST 422 (long name), two DELETE 409 (referenced, then the stale reference) and the GET 400 from the load-error arrange, all on /api/llm/provider-connections. Run browser errors --clear afterwards.

Redacted key is rejectedF11.create-validation · F11.editKnown failure

Scenario F11--redacted-key-is-rejected · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Use browser focus as specified in the exact recipe.
  2. Do
    Press Space.

Expect

in Add provider connection focus starts on Name; Save stays disabled until Name, a picked Provider and API Key are non-blank (whitespace does not count); a provider typed but not picked is cleared; Enter does not submit.

Edit opens Edit provider connection prefilled (key blank with placeholder <hidden> and hint Leave blank to keep the current key.); rename, provider and base URL changes persist (toast Connection "<name>" updated), a cleared base URL is stored as none, a blank name disables Save, and a rename shows in the linked profiles' group header.

Known bug observations

Known failure, reproduced 2026-10-12 (OpenHands/software-agent-sdk#5624): creation succeeds with api_key_set: false. Editing that row, typing the same placeholder and keyboard-saving succeeds and changes it to true. A second create under QA_SENTINEL_2 again succeeds with the key unset. The UI already prevents a whitespace-only key; this recipe drives the sentinel branch, not provider authentication. Remove both unreferenced fixture connections afterwards.

Exact CLI recipe & bug checks

Add QA_SENTINEL, choose OpenAI, fill the API Key with the literal ********** and Base URL with http://127.0.0.1:1/v1. Use the native keyboard Save (browser focus 'testid=provider-connection-submit', browser press Space; the provider listbox can intercept pointer clicks). Read api GET /api/llm/provider-connections --pick 0.api_key_set. Expected: reject the placeholder. Known failure, reproduced 2026-10-12 (OpenHands/software-agent-sdk#5624): creation succeeds with api_key_set: false. Editing that row, typing the same placeholder and keyboard-saving succeeds and changes it to true. A second create under QA_SENTINEL_2 again succeeds with the key unset. The UI already prevents a whitespace-only key; this recipe drives the sentinel branch, not provider authentication. Remove both unreferenced fixture connections afterwards.

Store lock keeps unrelated routes responsiveF11.listKnown failure

Scenario F11--store-lock-keeps-unrelated-routes-responsive · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Hold only the own private/provider-connections/.provider-connections.lock from an external run-owned process using fcntl.flock(..., LOCK_EX) for 34 seconds. Start an authenticated provider-connections list request, then request the own Agent Server's /alive two seconds later; reload the actual LLM page and capture its stalled loading state. Expected: /alive answers within two seconds while the store request waits. Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#5632): /alive returned 200 after 25.416 and 28.060 seconds. A direct request with a 45-second client deadline received the store's 503 after 30.052 seconds; the CLI's shorter request instead timed out. Release/close the own lock and run doctor; do not hold or alter any other run's stores.

Expect

each row shows the name, the provider, <n> model(s) (profiles linked to it), a green key-set icon and a ... menu labelled Provider connection menu; the section is reachable from the settings nav, the command menu and the URL.

Known bug observations

Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#5632): /alive returned 200 after 25.416 and 28.060 seconds. A direct request with a 45-second client deadline received the store's 503 after 30.052 seconds; the CLI's shorter request instead timed out. Release/close the own lock and run doctor; do not hold or alter any other run's stores.

Exact CLI recipe & bug checks

Hold only the own private/provider-connections/.provider-connections.lock from an external run-owned process using fcntl.flock(..., LOCK_EX) for 34 seconds. Start an authenticated provider-connections list request, then request the own Agent Server's /alive two seconds later; reload the actual LLM page and capture its stalled loading state. Expected: /alive answers within two seconds while the store request waits. Known failure, reproduced twice 2026-10-12 (OpenHands/software-agent-sdk#5632): /alive returned 200 after 25.416 and 28.060 seconds. A direct request with a 45-second client deadline received the store's 503 after 30.052 seconds; the CLI's shorter request instead timed out. Release/close the own lock and run doctor; do not hold or alter any other run's stores.

Connection URL edit reaches next chatF11.edit · F11.agent-uses-connectionKnown failureKnown issue

Scenario F11--connection-url-edit-reaches-next-chat · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Arrange a disposable OpenAI connection QA_URL_PATCH with a dummy key and http://127.0.0.1:1/v1, plus an activated linked QA_URL_LINKED profile without inline key/Base URL. From /settings/llm, open the exact connection row menu > provider-connection-edit, change provider-connection-base-url-input to http://127.0.0.1:1/v2 and submit; api GET /api/llm/provider-connections confirms the stored URL. Start a fresh Home chat with a bounded no-tools prompt, then read only agent.llm.base_url in its run-owned base_state.json. Repeat the real edit to /v3 and start another chat. Expected: each new runtime uses the newly saved URL. Known failure, reproduced twice 2026-10-12 (#17803): the connection persisted /v2 and /v3, but settings and both new runtimes retained the activated /v1 snapshot. All model endpoints refuse on loopback; no provider success is needed to observe resolution. Restore the neutral active profile, then delete only the own linked profile and connection.

Expect

Edit opens Edit provider connection prefilled (key blank with placeholder <hidden> and hint Leave blank to keep the current key.); rename, provider and base URL changes persist (toast Connection "<name>" updated), a cleared base URL is stored as none, a blank name disables Save, and a rename shows in the linked profiles' group header.

a conversation on a profile linked to the connection runs with the connection's key; an Edit with the key left blank keeps it working.

Known bug observations

Known failure, reproduced twice 2026-10-12 (#17803): the connection persisted /v2 and /v3, but settings and both new runtimes retained the activated /v1 snapshot. All model endpoints refuse on loopback; no provider success is needed to observe resolution. Restore the neutral active profile, then delete only the own linked profile and connection.

Linked known issues

Known issue OpenHands/software-agent-sdk#5207: OpenRouter linked profiles can lose connection credentials at raw agent/condenser startup or reuse stale credentials after rotation; opaque reasoning can also be lost on persisted turns. End-to-end checks need an approved OpenRouter connection and a model returning reasoning_details, including the linked-profile switch and reload path; loopback failure does not prove them. (F11.agent-uses-connection, F11.rotate).

Exact CLI recipe & bug checks

Arrange a disposable OpenAI connection QA_URL_PATCH with a dummy key and http://127.0.0.1:1/v1, plus an activated linked QA_URL_LINKED profile without inline key/Base URL. From /settings/llm, open the exact connection row menu > provider-connection-edit, change provider-connection-base-url-input to http://127.0.0.1:1/v2 and submit; api GET /api/llm/provider-connections confirms the stored URL. Start a fresh Home chat with a bounded no-tools prompt, then read only agent.llm.base_url in its run-owned base_state.json. Repeat the real edit to /v3 and start another chat. Expected: each new runtime uses the newly saved URL. Known failure, reproduced twice 2026-10-12 (#17803): the connection persisted /v2 and /v3, but settings and both new runtimes retained the activated /v1 snapshot. All model endpoints refuse on loopback; no provider success is needed to observe resolution. Restore the neutral active profile, then delete only the own linked profile and connection.

CloudF11.cloudBehavior contract

Scenario F11--contract-f11-cloud · Permalink · Source

Given

Use F11’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do

This behavior has no separately named recipe in this snapshot. Use the family’s entry points and the contract below; the page does not invent a replay.

Expect

on Cloud the section appears only with an organization bound and for members allowed to manage profiles; local users always see it.

Family notes, prerequisites for known issues & limits Known issue
  • Select the Provider by clicking an option (testid=provider-item-<name> exists only for verified providers; use 'role=option[name="<label>"]' for the others), or with browser choose 'testid=provider-connection-provider-input' <label>. browser fill alone types a filter that is cleared on blur and leaves Save disabled.
  • browser fill ... --value-file <key file> needs no positional value and prints only the length; keep keys out of commands and screenshots (the field is a password input).
  • Duplicate display names are accepted (two rows called QA_conn), and so is any base URL text; rows then differ only by order ('testid=provider-connection-row >> nth=1'). Keep fixture names unique.
  • Edit's Save is enabled even with no changes, and a blank key on Edit is omitted from the request (the stored key is kept); there is no way to clear a key.
  • The delete spinner (aria-busy) lasts under 100 ms on a local stack; --observe sees only Delete then <absent>.
  • role=dialog matches two nested elements in these modals; count > 0, do not expect 1.
  • The first profile added on a run with no profiles becomes the active one. Activating a linked profile copies the resolved key into the active agent settings, which keep the connection id: that is why the server still counts a reference after every linked profile is gone (see F11.delete-stale-reference) (OpenHands/software-agent-sdk#5498).
  • Delete a connection's linked profiles before the connection; then also make an unlinked profile active, or the delete is refused.
  • Rotation results are read from the conversation (conversation events ... --kinds ConversationErrorEvent,MessageEvent and the banner in the chat), not from the settings page, which never shows the key.
  • Model Router's Add connection reuses this modal (F12); its toasts and fields are the same.
  • F11.load-error needs a real server failure: the recipe corrupts this run's own provider_connections.json and restores it. Always restore it (keep the .bak), or every later family on the run sees the error; never touch another run's state.
  • At phone width a long name squeezes the provider column to one letter (d…); the full values stay in the API and the name's title.
  • Known issue OpenHands/software-agent-sdk#5497: after a key rotation, new conversations still run agent steps on the old key; only title generation uses the new one (F11.rotate).
  • Known issue OpenHands/software-agent-sdk#5207: OpenRouter linked profiles can lose connection credentials at raw agent/condenser startup or reuse stale credentials after rotation; opaque reasoning can also be lost on persisted turns. End-to-end checks need an approved OpenRouter connection and a model returning reasoning_details, including the linked-profile switch and reload path; loopback failure does not prove them. (F11.agent-uses-connection, F11.rotate).

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.