Engel’s notebook

LLM profiles

F10 · 34 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F10 LLM profiles

34 behaviors · 43 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

An LLM profile is a named model configuration: provider and model (or a custom model and base URL), an API key or a ChatGPT subscription, an optional provider-connection link, and every advanced LLM option. Under Settings → LLM a user lists profiles (grouped by provider connection once any are linked), adds and edits them in an embedded form whose Save first runs a one-token validation call, renames, duplicates, deletes and sets one as Default, and bulk-adds models from a provider. The Default profile is the model new conversations run on. Provider connections themselves (add, edit, delete) are F11.

How a user gets here

  • Sidebar settings gear (backend-selector-settings-link, opens /settings/agents), then LLM in the settings navigation (sidebar-settings-/settings/llm).
  • Direct URL /settings/llm.
  • Command menu (Control+k or Meta+k, or command-menu-trigger in the sidebar): search "LLM", choose LLM profiles ("Manage models, providers, and API keys.").
  • On the list: Add LLM Profile (add-llm-profile), Add from provider connections (add-models-from-provider), each row's ... (profile-menu-trigger), and a provider connection row's ... → Bulk add (provider-connection-add-models).
  • The create/edit editor has no URL of its own: it replaces the list in place at /settings/llm; Back (back-to-profiles) and Cancel (cancel-profile-btn) return to the list.

Given

Preconditions:

  • Baseline state (launched, doctored, onboard --skip done). Do not run llm preset deepseek first: a fresh run has no profiles, which the empty state and "first profile becomes Default" need.
  • A DeepSeek key for real saves, either in the environment as DEEPSEEK_API_KEY or in a file. Pass it only with browser fill ... --value-env DEEPSEEK_API_KEY or --value-file <key file>; never on the command line.
  • No profile or connection named QA_*, deepseek-chat or deepseek-v3 exists (control-openhands api GET /api/profiles, control-openhands api GET /api/llm/provider-connections).
  • Select a row by exact name with the CSS form '[data-testid=profile-row]:has([title="<name>"])'; has-text=QA_pro also matches QA_pro-copy.
  • Read toasts with control-openhands browser wait-text '<text>' (or browser toasts) right after the click; they disappear after a few seconds.
  • Blocked here: completing F10.subscription sign-in (needs a ChatGPT Plus/Pro account), F10.cloud-readonly (needs a Cloud org member account), F10.cloud-provider-pagination (needs a Cloud organization backend where the user can manage profiles), and F10.schema-unavailable (needs an agent server without the schema endpoints, for example an older launch --sdk-ref). The F10.list-states error is arranged by the Load error bullet (a linked profile plus this run's corrupted connections store).
Empty listF10.list-emptyMapped behavior

Scenario F10--empty-list · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Settings”.
  2. Do
    Choose “Llm settings”.

Expect

with no profiles the list reads No profiles saved yet. Add a profile to save your LLM configurations. and both Add buttons stay available.

Exact CLI recipe & bug checks

On a fresh run, navigate from the sidebar: control-openhands browser click 'testid=backend-selector-settings-link' --expect-url '/settings', control-openhands browser click 'testid=sidebar-settings-/settings/llm' --expect-url '/settings/llm', then control-openhands browser text 'testid=profiles-empty'. The text is No profiles saved yet. Add a profile to save your LLM configurations. and control-openhands browser testids main lists add-llm-profile, add-models-from-provider and profiles-empty.

Open the editorF10.createMapped behavior

Scenario F10--open-the-editor · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add llm profile”.

Expect

Add LLM Profile → provider, model and API key → Save shows Validating... then Saving..., toasts Profile "<name>" created and returns to the list; the first profile on an empty list becomes Default.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=add-llm-profile'. control-openhands browser text 'testid=profile-editor-title' is Add LLM Profile, control-openhands browser text 'testid=profile-editor-description' is Configure your LLM settings below, then click Save to create this profile., and the form opens on the Basic tab with provider OpenAI and model gpt-5.6-sol (browser value 'testid=llm-provider-input', browser value 'testid=llm-model-input').

API key help linkF10.api-key-helpMapped behavior

Scenario F10--api-key-help-link · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section basic toggle”.
  2. Do
    Choose “Llm api key help anchor” / the link.
  3. Do
    Use browser tabs as specified in the exact recipe.
  4. Do
    Use browser close tab as specified in the exact recipe.
  5. Do
    Use browser tab as specified in the exact recipe.

Expect

under the API key field (Basic and Advanced), Don't know your API key? Click here for instructions opens the OpenHands docs on getting an API key in a new tab.

Exact CLI recipe & bug checks

In that editor run control-openhands browser click 'testid=sdk-section-basic-toggle' (a new editor opens on All once the agent's LLM settings carry a non-default option; see F10.create-prefill), then control-openhands browser text 'testid=llm-api-key-help-anchor' is Don't know your API key? Click here for instructions; its link has target _blank (browser eval "document.querySelector('[data-testid=llm-api-key-help-anchor] a, a[data-testid=llm-api-key-help-anchor]').target"). Run control-openhands browser click 'testid=llm-api-key-help-anchor >> role=link', then control-openhands browser tabs (rerun once if it still lists one page): page 1 is https://docs.openhands.dev/.../local-setup#getting-an-api-key. Return with control-openhands browser close-tab 1 and control-openhands browser tab 0. The Advanced tab shows the same link as testid=llm-api-key-help-anchor-advanced.

Provider and model listsF10.model-pickerMapped behavior

Scenario F10--provider-and-model-lists · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Llm provider”.
  2. Do
    Enter “deepseek” in “Llm provider”.
  3. Do
    Choose “Provider item deepseek”.
  4. Do
    Choose “Llm model”.
  5. Do
    Enter “openhands” in “Llm provider”.
  6. Do
    Choose “Provider item openhands”.
  7. Do
    Choose “claude-sonnet-5”.

Expect

the Basic provider and model comboboxes group options as Verified Models and Other Models; changing provider clears the model; the OpenHands provider shows account and key help.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=llm-provider-input' and control-openhands browser snapshot 'role=listbox': groups Verified Models (OpenHands, Anthropic, OpenAI, Mistral AI, Gemini, deepseek, Moonshot, minimax, glm, nvidia, qwen) and Other Models. Pick DeepSeek with control-openhands browser fill 'testid=llm-provider-input' deepseek and control-openhands browser click 'testid=provider-item-deepseek'; browser value 'testid=llm-model-input' is now empty. control-openhands browser click 'testid=llm-model-input' and browser snapshot 'role=listbox' list deepseek-chat, deepseek-v4-pro, … as Verified and deepseek-coder, deepseek-flash, … under Other. For the OpenHands provider (browser fill 'testid=llm-provider-input' openhands, browser click 'testid=provider-item-openhands'), control-openhands browser text 'testid=openhands-account-help' is Need an OpenHands Account? Click here; after control-openhands browser click 'role=option[name="claude-sonnet-5"][exact]' in the model list, control-openhands browser text 'testid=openhands-api-key-help' starts You can find your OpenHands Provider LLM Key in the OpenHands LLM Key section of OpenHands Cloud. Screenshot: browser screenshot --feature F10.model-picker --name openhands-provider.

Every provider is offeredF10.provider-list-completeMapped behavior

Scenario F10--every-provider-is-offered · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Cancel profile btn”.
  2. Do
    Choose “Add llm profile”.
  3. Do
    Choose “Sdk section basic toggle”.
  4. Do
    Choose “Llm provider”.
  5. Do
    Choose “xai”.
  6. Do
    Choose “Llm model”.
  7. Do
    Press Escape.
  8. Do
    Choose “Cancel profile btn”.

Expect

the Basic provider combobox offers every provider the backend reports or has verified models for (over a hundred, xai included); picking a late-sorting one fills the field and loads its models.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=cancel-profile-btn' (the Provider and model lists bullet leaves its editor open; skip this when the list is showing), control-openhands browser click 'testid=add-llm-profile', control-openhands browser click 'testid=sdk-section-basic-toggle' and control-openhands browser click 'testid=llm-provider-input'. control-openhands browser count 'role=listbox >> role=option' is the size of the union of the two backend lists the page merges: the names in control-openhands api GET /api/llm/providers --pick providers (under value; control-openhands api GET /api/llm/providers --pick providers | jq '.value | length' is 160 today) and the provider keys of control-openhands api GET /api/llm/models/verified --max-bytes 200000 (its body is {"models": {<provider>: [<model>, …]}}; control-openhands api GET /api/llm/models/verified --max-bytes 200000 | jq '.body.models | keys | length' is 12 today, four of which, glm, nvidia, openhands and qwen, are missing from the first list). The union is jq -s '([.[0].value[]] + (.[1].body.models | keys)) | unique | length' <(control-openhands api GET /api/llm/providers --pick providers) <(control-openhands api GET /api/llm/models/verified --max-bytes 200000) (bash process substitution): 164 today, and the listbox count is the same 164, with every backend name present under its display label (openhands as OpenHands, mistral as Mistral AI, azure_ai as Azure AI Studio). control-openhands browser count 'role=option[name="xai"][exact]' is 1 (it sorts last but three, before xiaomi_mimo, xinference and zai). Pick it with control-openhands browser click 'role=option[name="xai"][exact]': control-openhands browser value 'testid=llm-provider-input' is xai, control-openhands browser value 'testid=llm-model-input' is empty, and after control-openhands browser click 'testid=llm-model-input' the listbox offers grok-4.20, grok-4.20-0309, … under Other Models (57 today; control-openhands browser screenshot --feature F10.provider-list-complete --name xai-picked). Close with control-openhands browser press Escape (the open listbox would swallow the next click) and control-openhands browser click 'testid=cancel-profile-btn'; the list is back.

Cloud provider paginationF10.cloud-provider-paginationBlocked

Scenario F10--cloud-provider-pagination · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Follow the environment or fixture command in the exact recipe.

Expect

on a Cloud organization backend the provider list is fetched from providers/search page by page, so providers past the first page (xai, openrouter) are offered too (blocked here).

Exact CLI recipe & bug checks

Blocked: needs a Cloud organization backend where the user can manage profiles (local backends serve the whole list in one GET /api/llm/providers). There, the Basic provider list comes from providers/search one page at a time (control-openhands browser network --filter providers/search lists one request per page, the later ones with a page_id), and xai and openrouter, which sort past the first page, are offered and pick like any other provider. Record control-openhands evidence add --feature F10.cloud-provider-pagination --result blocked ... naming the Cloud account.

Create the first profileF10.create · F10.name-autofillMapped behavior

Scenario F10--create-the-first-profile · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Llm model”.
  2. Do
    Choose “Model item deepseek flash”.
  3. Do
    Enter “QA_flash” in “Profile name”.
  4. Do
    Enter “the value specified in the recipe” in “Llm api key”.
  5. Do
    Choose “Save profile btn”.
  6. Wait
    Wait for the element identified in the CLI recipe to appear.
  7. Do
    Reload the page.

Expect

Add LLM Profile → provider, model and API key → Save shows Validating... then Saving..., toasts Profile "<name>" created and returns to the list; the first profile on an empty list becomes Default.

the Profile Name is auto-filled from the chosen model and follows later model changes until the user types a name; clearing the field resumes it.

Exact CLI recipe & bug checks

In a fresh editor pick deepseek as above, then control-openhands browser click 'testid=llm-model-input' and control-openhands browser click 'testid=model-item-deepseek-flash'. control-openhands browser value 'testid=profile-name-input' now follows the model: deepseek-flash (right after the provider pick, while the model is still empty, it still reads gpt-5.6-sol). Run control-openhands browser fill 'testid=profile-name-input' QA_flash, control-openhands browser fill 'testid=llm-api-key-input' --value-file <key file> (or --value-env DEEPSEEK_API_KEY), control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 4000 and control-openhands browser wait-text 'Profile "QA_flash" created' --timeout 60000. The observed button states are Save, Validating..., Saving..., then the editor closes. After control-openhands browser reload, control-openhands browser text '[data-testid=profile-row]:has([title="QA_flash"])' reads QA_flash, deepseek/deepseek-flash, Default, and control-openhands api GET /api/profiles has active_profile QA_flash.

A typed name sticks; a cleared one follows againF10.name-autofillMapped behavior

Scenario F10--a-typed-name-sticks-a-cleared-one-follows-again · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add llm profile”.
  2. Do
    Enter “QA_typed” in “Profile name”.
  3. Do
    Choose “Llm model”.
  4. Do
    Choose “deepseek-v4-pro”.
  5. Do
    Enter “” in “Profile name”.
  6. Do
    Choose “Llm model”.
  7. Do
    Choose “Model item deepseek flash”.

Expect

the Profile Name is auto-filled from the chosen model and follows later model changes until the user types a name; clearing the field resumes it.

Exact CLI recipe & bug checks

In a fresh editor (control-openhands browser click 'testid=add-llm-profile') run control-openhands browser fill 'testid=profile-name-input' QA_typed, then pick deepseek as above, control-openhands browser click 'testid=llm-model-input' and control-openhands browser click 'role=option[name="deepseek-v4-pro"][exact]'; browser value 'testid=profile-name-input' stays QA_typed. Run control-openhands browser fill 'testid=profile-name-input' '', then browser click 'testid=llm-model-input' and browser click 'testid=model-item-deepseek-flash': the name is deepseek-flash; pick role=option[name="deepseek-v4-pro"][exact] again and it is deepseek-v4-pro.

Name rulesF10.name-validationMapped behavior

Scenario F10--name-rules · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “QA bad” in “Profile name”.

Expect

a name that breaks the rule (letter or digit first; letters, digits, ., _, -; max 64) disables Save and turns the rule text red; a name that already exists marks the field invalid, shows a taken-name message about LLM profiles and disables Save (Edit accepts the profile's own name).

Exact CLI recipe & bug checks

In the editor run control-openhands browser fill 'testid=profile-name-input' 'QA bad'; control-openhands browser enabled 'testid=save-profile-btn' is false and control-openhands browser attr 'testid=profile-name-input' aria-invalid is true (the same for -QA and a 65-character name), and the rule text under the field turns to the theme's danger colour: control-openhands browser eval "(()=>{const i=document.querySelector('[data-testid=profile-name-input]');return getComputedStyle(document.getElementById(i.getAttribute('aria-describedby'))).color+' '+getComputedStyle(document.body).getPropertyValue('--oh-color-danger')})()" reads rgb(231, 106, 94) #e76a5e on OpenHands-Neutral and rgb(209, 52, 56) #D13438 on Light+ (F16.color-theme) (#18199); a valid name turns it back to the muted grey. An existing name (QA_flash) also gives enabled false and aria-invalid true, with a role=alert message under the field in the danger colour while the rule text stays grey. The message reads An LLM profile with this name already exists. Choose a different name, or edit the existing one. (browser eval "[...document.querySelectorAll('[role=alert]')].map(a=>a.innerText+' | '+getComputedStyle(a).color)"; screenshot --feature F10.name-validation --name duplicate-neutral). QA_flash2 gives true and no message.

Validation failureF10.create-validationKnown issue

Scenario F10--validation-failure · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Save profile btn”.
  2. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

a Save whose validation call fails toasts the provider error and keeps the editor open with nothing saved.

Linked known issues

Known issue #17785: Saving DeepSeek-v4-flash on Windows Desktop can fail with missing cache_creation_tokens instead of saving. The reported path needs a Windows Desktop 1.24.0 environment with its SDK dependency resolution and an approved DeepSeek key; this macOS source run cannot establish that result. (F10.create-validation).

Known issue OpenHands/software-agent-sdk#5862: DeepSeek invalid-key401 responses can be classified as bad-request errors. Blocked by this pass's loopback-only provider routing: a real invalid-key validation request to DeepSeek must be authorized to capture its actual401 shape. No usable key or successful inference is needed, but a loopback connection error does not prove this classification. (F10.create-validation).

Exact CLI recipe & bug checks

In a fresh editor choose deepseek / model-item-deepseek-flash, fill profile-name-input with QA_nokey, leave the API key empty and run control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 3000 (states Save, Validating..., Save). control-openhands browser wait-text 'Authentication Fails' finds the toast litellm.AuthenticationError: AuthenticationError: DeepseekException - Authentication Fails (...); browser text 'testid=profile-editor-title' is still Add LLM Profile and control-openhands api GET /api/profiles has no QA_nokey.

Custom modelF10.custom-modelKnown issue

Scenario F10--custom-model · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section advanced toggle”.
  2. Do
    Enter “” in “Llm custom model”.
  3. Do
    Enter “QA_adv” in “Profile name”.
  4. Do
    Choose “Save profile btn”.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Enter “deepseek/deepseek-flash” in “Llm custom model”.
  7. Do
    Enter “https://api.deepseek.com” in “Base url”.
  8. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

the Advanced tab takes a free-text model and Base URL; an empty model is refused with Model is required.

Linked known issues

Known issue OpenHands/software-agent-sdk#4613: A namespaced model such as openai/openai/example-model can have its inner provider prefix stripped again in Responses API and reach the provider as example-model. Reproduction needs an approved OpenAI-compatible endpoint serving the namespaced model and its exact stored API-mode settings; a loopback connection refusal cannot prove the outbound identifier. (F10.custom-model).

Known issue OpenHands/software-agent-sdk#4250: A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (F10.custom-model, F10.default-used).

Known issue OpenHands/software-agent-sdk#4247: An LM Studio configuration can fail with LLM Provider NOT provided although its models endpoint works. Reproduction needs the reachable LM Studio service and the exact model ID/provider-qualified config, which the report does not supply as text; the reported CLI/Docker SDK 1.16.1 installation is not this source stack. (F10.custom-model).

Exact CLI recipe & bug checks

In that editor run control-openhands browser click 'testid=sdk-section-advanced-toggle'. browser value 'testid=llm-custom-model-input' carries the Basic choice (deepseek/deepseek-flash; placeholder openai/gpt-5.6-sol), and browser attr 'testid=base-url-input' placeholder is https://api.openai.com. Run control-openhands browser fill 'testid=llm-custom-model-input' '', control-openhands browser fill 'testid=profile-name-input' QA_adv, control-openhands browser click 'testid=save-profile-btn' and control-openhands browser wait-text 'Model is required'. Then control-openhands browser fill 'testid=llm-custom-model-input' deepseek/deepseek-flash, control-openhands browser fill 'testid=base-url-input' https://api.deepseek.com, fill the key as in Create, Save, and control-openhands browser wait-text 'Profile "QA_adv" created' --timeout 60000. api GET /api/profiles shows QA_adv with base_url https://api.deepseek.com.

Second profile from a verified modelF10 · setup / cleanupSupport step

Scenario F10--second-profile-from-a-verified-model · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “deepseek-v4-pro”.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

Repeat Create choosing control-openhands browser click 'role=option[name="deepseek-v4-pro"][exact]' (verified models have no model-item-* test ID) and the name QA_pro. The toast reads Profile "QA_pro" created; the row QA_pro deepseek/deepseek-v4-pro has no badge.

Row menuF10.actions-menuMapped behavior

Scenario F10--row-menu · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Press ArrowDown.
  3. Do
    Press ArrowUp.
  4. Do
    Press Tab.
  5. Do
    Press Escape.
  6. Do
    Choose “Available Profiles”.
  7. Do
    Use browser focus as specified in the exact recipe.
  8. Do
    Press Enter.
  9. Do
    Press Escape.

Expect

the row menu offers Edit, Rename, Duplicate, Set as default (disabled on the Default row) and Delete; it focuses Edit on open (mouse or Enter), arrows cycle over the enabled items (skipping a disabled Set as default), and Escape, Tab or an outside click close it.

Exact CLI recipe & bug checks

Run control-openhands browser click '[data-testid=profile-row]:has([title="QA_flash"]) >> testid=profile-menu-trigger' and control-openhands browser snapshot 'testid=profile-actions-menu': menuitems Edit, Rename, Duplicate, Set as default [disabled], Delete on the Default row (all enabled on other rows). control-openhands browser eval "document.activeElement.dataset.testid" is profile-edit: focus moves into the menu on open. Run control-openhands browser press ArrowDown and that eval four times: profile-rename, profile-duplicate, profile-delete (the disabled Set as default is skipped), profile-edit (wraps); control-openhands browser press ArrowUp twice gives profile-delete, profile-duplicate. control-openhands browser press Tab closes the menu (browser count 'testid=profile-actions-menu' is 0) and moves focus on to the next row's trigger: the row-aware eval control-openhands browser eval "(()=>{const a=document.activeElement;const r=a.closest('[data-testid=profile-row]');return a.dataset.testid+'@'+(r?r.querySelector('[title]').title:'-')})()" is profile-menu-trigger@QA_pro. Reopen the QA_flash menu and control-openhands browser press Escape: count 0, and the row-aware eval is profile-menu-trigger@QA_flash (focus is back on its own trigger). Reopen it and control-openhands browser click 'role=heading[name="Available Profiles"]' closes it too. Keyboard open: control-openhands browser focus '[data-testid=profile-row]:has([title="QA_pro"]) >> testid=profile-menu-trigger' and control-openhands browser press Enter put focus on profile-edit; on this non-default row control-openhands browser snapshot 'testid=profile-actions-menu' lists the same five menuitems with none [disabled], and control-openhands browser screenshot --feature F10.actions-menu --name menu-nondefault shows them with the focus ring on Edit. Five ArrowDowns walk profile-rename, profile-duplicate, profile-set-active, profile-delete, profile-edit. control-openhands browser press Escape closes it, and the row-aware eval is profile-menu-trigger@QA_pro.

Set as defaultF10.set-defaultMapped behavior

Scenario F10--set-as-default · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Choose “Profile actions menu” / “Profile set active”.
  3. Wait
    Wait for the element identified in the CLI recipe to appear.
  4. Do
    Reload the page.
  5. Setup
    Configure or inspect the model profiles specified in the recipe.
  6. Setup
    Configure or inspect the model profiles specified in the recipe.
  7. Do
    Start a conversation with the prompt “Reply with only: adv-ok. Do not run any tools.”.

Expect

Set as default toasts Switched to profile "<name>", moves the badge, persists and becomes the agent's active LLM.

Exact CLI recipe & bug checks

Run control-openhands browser click '[data-testid=profile-row]:has([title="QA_pro"]) >> testid=profile-menu-trigger', control-openhands browser click 'testid=profile-actions-menu >> testid=profile-set-active' and control-openhands browser wait-text 'Switched to profile "QA_pro"'. After control-openhands browser reload, control-openhands browser count 'testid=profile-active-badge' is 1 and control-openhands browser count '[data-testid=profile-row]:has([title="QA_pro"]) >> testid=profile-active-badge' is 1; control-openhands api GET /api/profiles has active_profile QA_pro and control-openhands llm show reports active.model deepseek/deepseek-v4-pro. A profile's Base URL follows it into the agent settings: QA_adv (Custom model above) carries one: Set as active on its row makes control-openhands llm show report active.base_url https://api.deepseek.com, also under agent_settings.llm.base_url in control-openhands api GET /api/settings. With a key, control-openhands conversation start --prompt "Reply with only: adv-ok. Do not run any tools." --wait --timeout 240 then shows the pill QA_adv and the reply adv-ok (driven 2026-10-08; QA_pro likewise replied pro-ok on deepseek/deepseek-v4-pro). Make QA_pro the Default again before Duplicate.

DuplicateF10.duplicateMapped behavior

Scenario F10--duplicate · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

Duplicate saves <name>-copy, then <name>-copy-1, with the same model and stored key; the copy is not Default.

Exact CLI recipe & bug checks

From the QA_pro row menu click testid=profile-actions-menu >> testid=profile-duplicate and control-openhands browser wait-text 'Profile duplicated as "QA_pro-copy"'; repeat on QA_pro for Profile duplicated as "QA_pro-copy-1", and on QA_pro-copy for Profile duplicated as "QA_pro-copy-copy". After a reload control-openhands browser text 'testid=profile-row >> has-text=QA_pro-copy-1' reads QA_pro-copy-1 deepseek/deepseek-v4-pro, the badge stays on QA_pro, and api GET /api/profiles has api_key_set: true on every copy.

RenameF10.renameMapped behavior

Scenario F10--rename · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “-bad” in “Rename profile”.
  2. Do
    Choose “Rename Profile” / “Cancel”.
  3. Do
    Press Enter.
  4. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

the Rename modal is prefilled and focused; invalid names disable Rename; an unchanged name or Cancel just closes; Enter submits; success toasts and persists, and the Default badge follows a renamed default.

Exact CLI recipe & bug checks

Open the QA_pro-copy-copy row menu and click testid=profile-actions-menu >> testid=profile-rename. control-openhands browser value 'testid=rename-profile-input' is QA_pro-copy-copy and browser eval "document.activeElement.dataset.testid" is rename-profile-input. control-openhands browser fill 'testid=rename-profile-input' '-bad' makes control-openhands browser enabled 'role=dialog[name="Rename Profile"] >> role=button[name="Rename"]' false and control-openhands browser attr 'testid=rename-profile-rule' class contains text-danger (the theme danger colour, #18199). Filling the old name and clicking Rename closes the modal with no toast (browser count 'testid=rename-profile-modal' is 0); control-openhands browser click 'role=dialog[name="Rename Profile"] >> role=button[name="Cancel"]' also closes it. To rename, reopen, fill QA_renamed, control-openhands browser press Enter, control-openhands browser wait-text 'Profile renamed to "QA_renamed"', reload, and the count of '[data-testid=profile-row]:has([title="QA_renamed"])' is 1 while the old name's is 0. Renaming the Default row (QA_pro → QA_pro2, submit with testid=rename-profile-submit) keeps its badge (browser text '[data-testid=profile-row]:has([data-testid=profile-active-badge])' reads QA_pro2 … Default) and active_profile follows.

Rename onto an existing nameF10.rename-conflictMapped behavior

Scenario F10--rename-onto-an-existing-name · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Rename Profile” / “Cancel”.

Expect

renaming onto an existing name is refused with the server's message (Profile '<name>' already exists) and the modal stays open.

Exact CLI recipe & bug checks

The Rename step above already renamed QA_pro-copy-copy and QA_pro, so use the remaining fixtures: open the QA_pro-copy row menu, click testid=profile-actions-menu >> testid=profile-rename, fill testid=rename-profile-input with QA_pro2 and click testid=rename-profile-submit. The modal stays open (count 'testid=rename-profile-modal' is 1), control-openhands browser toasts has the server's message Profile 'QA_pro2' already exists (no HTTP request failed prefix, no JSON), and browser screenshot --feature F10.rename-conflict --name toast shows that toast above the open modal. Close it with control-openhands browser click 'role=dialog[name="Rename Profile"] >> role=button[name="Cancel"]'.

EditF10.editKnown issueBlocked

Scenario F10--edit · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Llm model”.
  2. Do
    Choose “Model item deepseek flash”.
  3. Do
    Enter “QA_edited” in “Profile name”.
  4. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

Edit loads the profile (Editing profile "<name>" - save to apply changes) with the stored key masked; Save stays disabled until something changes; a changed name renames before saving; the stored key is kept.

Linked known issues

Known issue OpenHands/software-agent-sdk#5890: Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (F10.edit, F10.default-used).

Known issue #17806: A model-only Save must preserve an untouched key. Current SDK1.55 refused-loopback validation rejects Save before persistence; the stored key marker remains set. Completing Basic/Advanced preserved-key and runtime-switch proof needs an approved route that accepts preflight. Keep dummy traffic on refused loopback until an external route is authorized; a rejected Save does not establish either key loss or successful preservation through Save. (F10.edit).

Exact CLI recipe & bug checks

Open the QA_renamed row menu and click testid=profile-actions-menu >> testid=profile-edit. browser text 'testid=profile-editor-title' is Edit LLM Profile, browser text 'testid=profile-editor-description' is Editing profile "QA_renamed" - save to apply changes, browser attr 'testid=llm-api-key-input' placeholder is <hidden>, browser count 'testid=set-indicator' is 1 and browser enabled 'testid=save-profile-btn' is false (screenshot --feature F10.edit --name loaded). Filling another profile's name (QA_flash) sets aria-invalid true with the taken-name message of F10.name-validation and keeps Save disabled; refilling QA_renamed clears it. Run control-openhands browser click 'testid=llm-model-input', control-openhands browser click 'testid=model-item-deepseek-flash' (Save turns true), control-openhands browser fill 'testid=profile-name-input' QA_edited, Save, and control-openhands browser wait-text 'Profile "QA_edited" updated' --timeout 60000. After a reload browser text '[data-testid=profile-row]:has([title="QA_edited"])' reads QA_edited deepseek/deepseek-flash and QA_renamed is gone. The save passed validation without retyping the key, which proves the stored key was kept.

Basic re-save keeps a hidden Base URLF10.basic-save-keeps-base-urlMapped behavior

Scenario F10--basic-re-save-keeps-a-hidden-base-url · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section advanced toggle”.
  2. Do
    Choose “Sdk section basic toggle”.
  3. Do
    Enter “the value specified in the recipe” in “Llm api key”.
  4. Do
    Choose “Save profile btn”.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Reload the page.
  7. Do
    Reload the page.
  8. Do
    Choose “Cancel profile btn”.
  9. Setup
    Arrange the prerequisite backend state through the API.

Expect

re-saving a profile from the Basic view without changing its model keeps the Base URL that view hides.

Exact CLI recipe & bug checks

QA_adv carries base_url https://api.deepseek.com (Custom model above). Open its row menu, click testid=profile-actions-menu >> testid=profile-edit, then control-openhands browser click 'testid=sdk-section-advanced-toggle': control-openhands browser value 'testid=base-url-input' is https://api.deepseek.com. Run control-openhands browser click 'testid=sdk-section-basic-toggle': control-openhands browser count 'testid=base-url-input' is 0 (the view hides it), control-openhands browser value 'testid=llm-model-input' is deepseek-flash and control-openhands browser enabled 'testid=save-profile-btn' is false (control-openhands browser screenshot --feature F10.basic-save-keeps-base-url --name basic-view). Without touching the model, run control-openhands browser fill 'testid=llm-api-key-input' --value-env DEEPSEEK_API_KEY (Save turns true), control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 4000 (Save, Validating..., Saving...) and control-openhands browser wait-text 'Profile "QA_adv" updated' --timeout 60000. Then control-openhands api GET /api/profiles/QA_adv --pick config.base_url is still https://api.deepseek.com and control-openhands api GET /api/profiles/QA_adv --pick config.model is deepseek/deepseek-flash; after control-openhands browser reload the same GETs agree. Without a key, arrange QA_DUMMY_KEY=qa control-openhands llm set --profile QA_base --model deepseek/deepseek-chat --api-key-env QA_DUMMY_KEY --base-url https://api.deepseek.com --no-validate --no-activate (it prints activated false and control-openhands api GET /api/profiles --pick active_profile is unchanged: the Default stays where it is and the editors below inherit it, as long as some profile is Default; on a run where QA_base is the only profile the page makes it Default at the next load, see Gotchas) and control-openhands browser reload, then drive the same steps on QA_base's row with its own readbacks: its editor opens on Advanced (the Base URL is a non-default value) with control-openhands browser value 'testid=base-url-input' https://api.deepseek.com, Basic hides it (count 0), control-openhands browser value 'testid=llm-model-input' is deepseek-chat and control-openhands browser enabled 'testid=save-profile-btn' is false; fill the key with QA_DUMMY_KEY=qa-dummy-key control-openhands browser fill 'testid=llm-api-key-input' --value-env QA_DUMMY_KEY (Save turns true) and Save: the observed states are Save, Validating... ([disabled] [busy]), Save within a second, control-openhands browser toasts has the provider's message litellm.BadRequestError: DeepseekException - {"error":{"message":"Authentication Fails, Your api key: ****-key is invalid …, browser text 'testid=profile-editor-title' is still Edit LLM Profile, control-openhands api GET /api/profiles/QA_base --pick config.base_url is still https://api.deepseek.com and --pick config.model is deepseek/deepseek-chat; record blocked with DEEPSEEK_API_KEY as the prerequisite, leave with control-openhands browser click 'testid=cancel-profile-btn' and remove the arranged profile with control-openhands api DELETE /api/profiles/QA_base --write (arrange, not proof).

Leave without savingF10.editor-discardMapped behavior

Scenario F10--leave-without-saving · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section all toggle”.
  2. Do
    Enter “0.5” in “Sdk settings llm.top p”.
  3. Do
    Choose “Back to profiles”.

Expect

Back and Cancel return to the list without saving and without an unsaved-changes prompt.

Exact CLI recipe & bug checks

Edit QA_adv, control-openhands browser click 'testid=sdk-section-all-toggle', control-openhands browser fill 'testid=sdk-settings-llm.top_p' 0.5 (Save turns enabled), then control-openhands browser click 'testid=back-to-profiles'. The list returns at once (browser count 'testid=add-llm-profile' is 1) with no prompt (control-openhands browser events --kinds dialog is empty), and reopening Edit → All shows top_p empty. testid=cancel-profile-btn behaves the same.

Schema field validation and re-applyF10.schema-validation · F10.edit-default-reapplyKnown issue

Scenario F10--schema-field-validation-and-re-apply · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section all toggle”.
  2. Do
    Enter “5” in “Sdk settings llm.temperature”.
  3. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

bad values in schema fields (Temperature over 2, Top P over 1, invalid JSON) are refused with a toast naming the field.

saving the Default profile re-applies it to the agent's LLM settings.

Linked known issues

Known issue OpenHands/software-agent-sdk#5181: The SDK can turn an advertised output limit into a requested default cap and silently drop an explicitly enabled reasoning_effort. The narrowed maintainer scope needs an authenticated OpenRouter/proxy route exposing lower-limit endpoints and capability metadata, or a real endpoint accepting the explicit override; a synthetic empty-metadata response would not establish production behavior. (F10.schema-validation).

Known issue OpenHands/software-agent-sdk#4934: Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (F10.schema-validation, F10.default-used).

Known issue OpenHands/software-agent-sdk#4511: Disabling prompt caching can still emit cache_control on preexisting cache-marked tool messages. The report only supplies a Python serializer reproduction; a Canvas check needs the exact persisted/tool-message precondition and a provider request that exposes the cache markers, with approved model access. (F10.schema-validation).

Exact CLI recipe & bug checks

Edit the Default profile and run control-openhands browser click 'testid=sdk-section-all-toggle'. control-openhands browser fill 'testid=sdk-settings-llm.temperature' 5, Save, then control-openhands browser wait-text 'must be at most 2' (toast Temperature must be at most 2). Clear it, fill testid=sdk-settings-llm.top_p with 3: Top P must be at most 1. Clear it, fill testid=sdk-settings-llm.extra_headers with {bad: Invalid JSON for Extra Headers. Clear that, fill temperature 0.3, Save, and wait-text 'Profile "<name>" updated'. control-openhands api GET /api/settings then has agent_settings.llm.temperature 0.3, and reopening Edit → All shows 0.3.

New profile prefillF10.create-prefillMapped behavior

Scenario F10--new-profile-prefill · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add llm profile”.
  2. Do
    Check the stated assertion against live output.
  3. Do
    Enter “deepseek/deepseek-flash” in “Llm custom model”.
  4. Do
    Enter “QA_inherit” in “Profile name”.
  5. Do
    Scroll “Sdk settings llm.temperature” as specified in the recipe.
  6. Do
    Reload the page.
  7. Do
    Scroll “Sdk settings llm.temperature” as specified in the recipe.
  8. Do
    Check the stated assertion against live output.
  9. Setup
    Use the phone viewport.
  10. Setup
    Arrange the prerequisite backend state through the API.

Expect

a new editor prefills the backend default model with a blank key, base URL and connection; the other LLM options it shows come from the agent's current LLM settings, i.e. the last applied Default profile (the editor opens on the tab they need), and Save stores what the form shows.

Exact CLI recipe & bug checks

With that Default (temperature 0.3) run control-openhands browser click 'testid=add-llm-profile'. The editor opens on All (browser eval "[...document.querySelectorAll('[role=tab]')].map(t=>t.innerText+':'+t.getAttribute('aria-selected')).join(',')" is Basic:false,Advanced:false,All:true), browser value 'testid=llm-custom-model-input' is the backend default openai/gpt-5.6-sol, and control-openhands assert --field value --equals 0.3 -- browser value 'testid=sdk-settings-llm.temperature' passes: the temperature is inherited from the Default. Run control-openhands browser fill 'testid=llm-custom-model-input' deepseek/deepseek-flash, browser fill 'testid=profile-name-input' QA_inherit, fill the key as in Create, control-openhands browser scroll 'testid=sdk-settings-llm.temperature', control-openhands browser screenshot --feature F10.create-prefill --name inherited-before-save, Save and wait-text 'Profile "QA_inherit" created' --timeout 60000. The saved profile has the temperature the form showed: control-openhands api GET /api/profiles/QA_inherit --pick config.temperature is 0.3. After browser reload, open QA_inherit's row menu ('[data-testid=profile-row]:has([title="QA_inherit"]) >> testid=profile-menu-trigger'), click testid=profile-actions-menu >> testid=profile-edit and testid=sdk-section-all-toggle, control-openhands browser scroll 'testid=sdk-settings-llm.temperature' and control-openhands assert --field value --equals 0.3 -- browser value 'testid=sdk-settings-llm.temperature' passes (screenshot --feature F10.create-prefill --name edit-after-create); leave with testid=cancel-profile-btn. Repeat at browser viewport phone after control-openhands api DELETE /api/profiles/QA_inherit --write (the desktop pass already took the name); the same holds.

Link to a provider connectionF10.connection-link · F10.list-groupedMapped behavior

Scenario F10--link-to-a-provider-connection · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add provider connection”.
  2. Do
    Enter “QA_ds_conn” in “Provider connection name”.
  3. Do
    Choose “Provider connection provider”.
  4. Do
    Enter “deepseek” in “Provider connection provider”.
  5. Do
    Choose “Provider item deepseek”.
  6. Do
    Enter “the value specified in the recipe” in “Provider connection api key”.
  7. Do
    Choose “Provider connection submit”.
  8. Wait
    Wait for the element identified in the CLI recipe to appear.
  9. Do
    Choose “Llm provider connection”.
  10. Do
    Choose “QA_ds_conn”.
  11. Do
    Choose “Save profile btn”.

Expect

with connections present the editor shows a Provider connection selector; linking hides API key and Base URL, skips validation on Save and regroups the row; None restores the key field.

once any profile links to a provider connection, rows group under the connection's name, with unlinked rows last under Not linked; with no linked rows the list is flat again.

Exact CLI recipe & bug checks

Arrange a connection through the F11 form: control-openhands browser click 'testid=add-provider-connection', control-openhands browser fill 'testid=provider-connection-name-input' QA_ds_conn, control-openhands browser click 'testid=provider-connection-provider-input', control-openhands browser fill 'testid=provider-connection-provider-input' deepseek, control-openhands browser click 'testid=provider-item-deepseek', control-openhands browser fill 'testid=provider-connection-api-key-input' --value-file <key file>, control-openhands browser click 'testid=provider-connection-submit' and control-openhands browser wait-text 'Connection "QA_ds_conn" created'. Edit QA_adv, click testid=sdk-section-basic-toggle, run control-openhands browser click 'testid=llm-provider-connection-input' (options None, QA_ds_conn) and control-openhands browser click 'role=option[name="QA_ds_conn"]'; browser count 'testid=llm-api-key-input' is 0, and on the Advanced tab browser count 'testid=base-url-input' is 0. Picking role=option[name="None"] brings the key field back (1). Re-select QA_ds_conn, then control-openhands browser click 'testid=save-profile-btn' --observe 'testid=save-profile-btn' --observe-ms 3000: states Save, Saving... (no Validating...), and wait-text 'Profile "QA_adv" updated'. After a reload control-openhands browser eval "[...document.querySelectorAll('[data-testid=profile-group-header]')].map(e=>e.innerText).join(' | ')" is QA_DS_CONN | NOT LINKED (CSS upper-case) and control-openhands browser count 'div:has(> [data-testid=profile-group-header]:text-is("QA_ds_conn")) [data-testid=profile-row]:has([title="QA_adv"])' is 1 (screenshot --feature F10.list-grouped --name grouped).

Add models from a providerF10.add-modelsMapped behavior

Scenario F10--add-models-from-a-provider · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Add models from provider”.
  2. Do
    Use browser select as specified in the exact recipe.
  3. Wait
    Wait for “Add models select all” to appear.
  4. Do
    Use browser check as specified in the exact recipe.
  5. Do
    Use browser select as specified in the exact recipe.
  6. Do
    Use browser check as specified in the exact recipe.
  7. Do
    Use browser check as specified in the exact recipe.
  8. Do
    Choose “Add models submit”.
  9. Wait
    Wait for the element identified in the CLI recipe to appear.
  10. Do
    Choose “Provider connection row” / containing “QA_ds_conn” / “Provider connection menu trigger”.
  11. Do
    Choose “Provider connection actions menu” / “Provider connection add models”.
  12. Do
    Choose “Add models as profiles” / “Cancel”.

Expect

"Add from provider connections" (or a connection row's Bulk add) creates one linked or keyless profile per checked model; models that already have a profile of that name are hidden.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=add-models-from-provider'; browser enabled 'testid=add-models-submit' is false. Keyless: control-openhands browser select 'testid=add-models-provider' moonshot and control-openhands browser wait 'testid=add-models-select-all'; browser text 'testid=add-models-keyless-note' is No connection — profiles will need a key added later, rows start unchecked (Add 0 profiles), and control-openhands browser check 'testid=add-models-select-all' makes browser text 'testid=add-models-submit' read Add 12 profiles; uncheck it again. Linked: control-openhands browser select 'testid=add-models-provider' deepseek; the connection field (testid=add-models-connection-field, options No connection (keyless) and QA_ds_conn) is pre-bound to QA_ds_conn. Run control-openhands browser check 'testid=add-models-check-deepseek/deepseek-chat', control-openhands browser check 'testid=add-models-check-deepseek/deepseek-v3', control-openhands browser click 'testid=add-models-submit' and control-openhands browser wait-text 'Added 2' (toast Added 2 profiles). The modal closes; after a reload deepseek-chat and deepseek-v3 sit under the QA_DS_CONN header and api GET /api/profiles shows both with a provider_connection_id. Preselect entry: control-openhands browser click 'testid=provider-connection-row >> has-text=QA_ds_conn >> testid=provider-connection-menu-trigger' (menu: Bulk add, Edit, Delete), control-openhands browser click 'testid=provider-connection-actions-menu >> testid=provider-connection-add-models'; provider deepseek and the connection are preset, every remaining row is checked (Add 10 profiles), and browser count 'testid=add-models-row-deepseek/deepseek-chat' is 0 (already added). Close with control-openhands browser click 'role=dialog[name="Add models as profiles"] >> role=button[name="Cancel"]'.

Broken linkF10.broken-linkMapped behavior

Scenario F10--broken-link · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Arrange the prerequisite backend state through the API.
  2. Do
    Reload the page.

Expect

a profile whose linked connection no longer exists shows a warning Broken link badge whose tooltip says to edit the profile to re-link it; its editor offers the orphaned id in the Provider connection selector so it can be cleared.

Exact CLI recipe & bug checks

The local server refuses to delete a referenced connection (row menu → testid=provider-connection-delete → testid=delete-provider-connection-confirm toasts Provider connection cannot be deleted while it is referenced by LLM profile(s): ...), so arrange the state directly: control-openhands api POST /api/profiles/QA_broken --write --data '{"llm":{"model":"deepseek/deepseek-chat","provider_connection_id":"00000000000000000000000000000000"},"include_secrets":false}'. Then control-openhands browser reload and control-openhands browser text '[data-testid=profile-row]:has([title="QA_broken"])' reads QA_broken deepseek/deepseek-chat Broken link; control-openhands browser attr '[data-testid=profile-row]:has([title="QA_broken"]) >> testid=profile-broken-connection-badge' title is The linked provider connection was deleted. Edit this profile to re-link it. (screenshot --feature F10.broken-link --name badge). Edit it: browser value 'testid=llm-provider-connection-input' is the orphaned id and the listbox offers None, QA_ds_conn and that id. The row's group header shows the raw id, not a name (see Gotchas).

ChatGPT subscription, pre-authF10.subscriptionKnown failureKnown issueBlocked

Scenario F10--chatgpt-subscription-pre-auth · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section basic toggle”.
  2. Do
    Choose “Llm auth type”.
  3. Do
    Choose “ChatGPT subscription”.
  4. Do
    Enter “QA_sub” in “Profile name”.
  5. Do
    Choose “Save profile btn”.
  6. Do
    Choose “Subscription connect”.
  7. Wait
    Wait for “Subscription user code” to appear.
  8. Do
    Use browser tabs as specified in the exact recipe.
  9. Do
    Use browser tabs as specified in the exact recipe.
  10. Do
    Use browser close tab as specified in the exact recipe.
  11. Do
    Use browser tab as specified in the exact recipe.
  12. Do
    Choose “Subscription poll”.
  13. Do
    Choose “Subscription cancel”.

Expect

Authentication → ChatGPT subscription swaps the model fields for a subscription model list and a device-login card (code, sign-in link, poll, cancel); switching back restores the API-key model.

Known bug observations

Known failure (reproduced 2026-10-13, #17135): run control-openhands browser fill 'testid=profile-name-input' QA_sub, control-openhands browser network --clear, control-openhands browser click 'testid=save-profile-btn' and control-openhands browser network --bodies --filter '/api/profiles'. Expected: Save validates and saves the shown gpt-5.5. Actual: one POST /api/profiles/QA_sub/validate whose body has "model":"openai/gpt-5.6-sol", the API-key form's hidden default, although the dropdown shows gpt-5.5 (it shows the first subscription model without writing it to the form); browser toasts --history has OpenAI subscription login is required, because no subscription is connected. Picking a model in testid=llm-subscription-model-input, even gpt-5.5 again, makes the next Save send POST /api/profiles/QA_sub/validate, which answers OpenAI subscription login is required here (browser toasts --history), because no subscription is connected. control-openhands browser click 'testid=subscription-connect' and control-openhands browser wait 'testid=subscription-user-code': the card shows the code, Open sign-in page, I've finished signing in (subscription-poll) and Cancel, and control-openhands browser tabs lists a second page on auth.openai.com (it opens a second or two after the code appears; rerun browser tabs if it lists one page). The card polls on its own, so Sign-in is not complete yet. ... can already show before you click poll. Close it with control-openhands browser close-tab 1 and control-openhands browser tab 0. control-openhands browser click 'testid=subscription-poll' leaves Sign-in is not complete yet. Finish the browser flow and try again. on the card, and control-openhands browser click 'testid=subscription-cancel' removes the code (count 'testid=subscription-user-code' 0). Switching back with role=option[name="API key"] restores provider OpenAI and model gpt-5.6-sol. Completing sign-in is blocked (needs a ChatGPT Plus/Pro account).

Linked known issues

Known issue #17391: ChatGPT Business subscription login can be refused or use a fallback model with a 404. Completing the entitlement and selected-model checks needs an approved ChatGPT Business workspace login and the reported Docker subscription runtime. (F10.subscription).

Exact CLI recipe & bug checks

In a fresh editor run control-openhands browser click 'testid=sdk-section-basic-toggle' first (after the schema step the Default profile has a temperature, so a new editor opens on All; see F10.create-prefill), then control-openhands browser click 'testid=llm-auth-type-input' (options API key, ChatGPT subscription) and control-openhands browser click 'role=option[name="ChatGPT subscription"]'. browser text 'testid=subscription-status' is ChatGPT subscription not connected, browser value 'testid=llm-subscription-model-input' is gpt-5.5 (list: gpt-5.5, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-6-astra), and llm-provider-input and llm-api-key-input are gone (count 0). Known failure (reproduced 2026-10-13, #17135): run control-openhands browser fill 'testid=profile-name-input' QA_sub, control-openhands browser network --clear, control-openhands browser click 'testid=save-profile-btn' and control-openhands browser network --bodies --filter '/api/profiles'. Expected: Save validates and saves the shown gpt-5.5. Actual: one POST /api/profiles/QA_sub/validate whose body has "model":"openai/gpt-5.6-sol", the API-key form's hidden default, although the dropdown shows gpt-5.5 (it shows the first subscription model without writing it to the form); browser toasts --history has OpenAI subscription login is required, because no subscription is connected. Picking a model in testid=llm-subscription-model-input, even gpt-5.5 again, makes the next Save send POST /api/profiles/QA_sub/validate, which answers OpenAI subscription login is required here (browser toasts --history), because no subscription is connected. control-openhands browser click 'testid=subscription-connect' and control-openhands browser wait 'testid=subscription-user-code': the card shows the code, Open sign-in page, I've finished signing in (subscription-poll) and Cancel, and control-openhands browser tabs lists a second page on auth.openai.com (it opens a second or two after the code appears; rerun browser tabs if it lists one page). The card polls on its own, so Sign-in is not complete yet. ... can already show before you click poll. Close it with control-openhands browser close-tab 1 and control-openhands browser tab 0. control-openhands browser click 'testid=subscription-poll' leaves Sign-in is not complete yet. Finish the browser flow and try again. on the card, and control-openhands browser click 'testid=subscription-cancel' removes the code (count 'testid=subscription-user-code' 0). Switching back with role=option[name="API key"] restores provider OpenAI and model gpt-5.6-sol. Completing sign-in is blocked (needs a ChatGPT Plus/Pro account).

DeleteF10.delete · F10.delete-defaultMapped behavior

Scenario F10--delete · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose the element identified in the CLI recipe.
  2. Do
    Choose “Profile actions menu” / “Profile delete”.
  3. Do
    Choose “Delete Profile” / “Cancel”.
  4. Do
    Choose “Delete profile confirm”.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.

Expect

Delete asks Are you sure you want to delete the profile "<name>"? This action cannot be undone. with Cancel focused; Cancel keeps the row, Delete removes it with a toast.

deleting the Default profile promotes another remaining profile to Default.

Exact CLI recipe & bug checks

Run control-openhands browser click '[data-testid=profile-row]:has([title="QA_pro-copy"]) >> testid=profile-menu-trigger' and control-openhands browser click 'testid=profile-actions-menu >> testid=profile-delete'. control-openhands browser snapshot 'role=dialog' shows Delete Profile with Are you sure you want to delete the profile "QA_pro-copy"? This action cannot be undone., and browser eval "document.activeElement.innerText" is Cancel. control-openhands browser click 'role=dialog[name="Delete Profile"] >> role=button[name="Cancel"]' keeps the row (count 1). Repeat, then control-openhands browser click 'testid=delete-profile-confirm', control-openhands browser wait-text 'Profile "QA_pro-copy" deleted', reload, and the count is 0. Deleting the Default row the same way leaves exactly one profile-active-badge on another profile (browser text '[data-testid=profile-row]:has([data-testid=profile-active-badge])'), and api GET /api/profiles names it in active_profile.

Command menu and direct URLF10.listMapped behavior

Scenario F10--command-menu-and-direct-url · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /.
  2. Do
    Press Control+k.
  3. Do
    Enter “LLM” in “Command menu” / the combobox.
  4. Do
    Choose “Command menu” / “LLM profiles”.
  5. Do
    Open /settings/llm.

Expect

the "Available Profiles" list shows each row's name, model, a Default badge on the active profile and a ... menu, sorted by name; reachable from the sidebar, the direct URL and the command menu.

Exact CLI recipe & bug checks

Run control-openhands browser goto / (the bullets above end on /settings/llm, where the URL wait would pass at once), control-openhands browser press Control+k, control-openhands browser type 'testid=command-menu >> role=combobox' LLM and control-openhands browser snapshot 'testid=command-menu': two options, LLM profiles ("Manage models, providers, and API keys.") and Model Router (F12; its keywords include "llm"). Hovering an option selects it, so which one Enter runs depends on where the mouse rests: click the option instead, control-openhands browser click 'testid=command-menu >> role=option[name="LLM profiles"]' --expect-url '/settings/llm(\?|$)'. browser count 'testid=add-llm-profile' is 1 on arrival. control-openhands browser goto /settings/llm shows the same list (browser count 'testid=profile-row', one profile-active-badge; screenshot --feature F10.list --name list).

Phone layoutF10.phoneMapped behavior

Scenario F10--phone-layout · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Use the phone viewport.
  2. Do
    Open /settings/llm.
  3. Do
    Choose “Cancel profile btn”.
  4. Setup
    Use the desktop viewport.

Expect

list, row menu and editor fit a 390 px viewport without horizontal overflow.

Exact CLI recipe & bug checks

Run control-openhands browser viewport phone, control-openhands browser goto /settings/llm, control-openhands browser bbox 'testid=add-llm-profile', control-openhands browser bbox 'testid=profile-row >> nth=0' and control-openhands browser screenshot --feature F10.phone --name list: insideViewport true, pageHorizontalOverflow false. Open a row menu: browser bbox 'testid=profile-actions-menu' is inside the viewport. Open Edit: browser bbox 'testid=profile-name-input' is 348 px wide inside the 390 px viewport and pageHorizontalOverflow is false (llm-settings-screen may report insideViewport false when the edited profile's form is taller than the screen; only the overflow flag matters). Return with control-openhands browser click 'testid=cancel-profile-btn' and control-openhands browser viewport desktop.

Default profile drives a conversationF10.default-usedKnown issueBlocked

Scenario F10--default-profile-drives-a-conversation · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Start a conversation with the prompt “Reply with exactly the word: pong”.
  2. Do
    Open /settings/llm.

Expect

a conversation started after creating a profile in the UI runs on the Default profile's model.

Linked known issues

Known issue #17706: A freshly resolved npm-launcher environment can reject openhands/claude-* conversations because LiteLLM sends prompt_cache_key. Reproduction needs an approved OpenHands proxy key with an Anthropic model and the affected fresh transitive dependency resolution; a keyless initialization cannot prove provider acceptance. (F10.default-used).

Known issue OpenHands/software-agent-sdk#4934: Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (F10.schema-validation, F10.default-used).

Known issue OpenHands/software-agent-sdk#4250: A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (F10.custom-model, F10.default-used).

Known issue OpenHands/software-agent-sdk#5890: Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (F10.edit, F10.default-used).

Known issue OpenHands/software-agent-sdk#5838: A live LLM switch during a running tool step can be acknowledged before the new LLM is persisted for GET/restart. Blocked: needs a genuine model-issued bounded tool step (for example sleep 20) and an approved starting model. Refused-loopback initialization never enters that step. (F10.default-used).

Exact CLI recipe & bug checks

Make a profile with a distinct model Default through the row menu (for example QA_pro-copy-1, deepseek/deepseek-v4-pro, then wait-text 'Switched to profile "QA_pro-copy-1"'), and run control-openhands conversation start --prompt "Reply with exactly the word: pong" --wait --timeout 180. The JSON has "status": "finished" and "model": "deepseek/deepseek-v4-pro". The browser is now on the conversation page: control-openhands browser goto /settings/llm before driving this family again.

Persistence across a restartF10.listMapped behavior

Scenario F10--persistence-across-a-restart · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Restart the run-owned stack as specified.
  2. Do
    Open /settings/llm.

Expect

the "Available Profiles" list shows each row's name, model, a Default badge on the active profile and a ... menu, sorted by name; reachable from the sidebar, the direct URL and the command menu.

Exact CLI recipe & bug checks

control-openhands restart, control-openhands doctor, control-openhands browser goto /settings/llm: the same rows and the same Default badge are back.

Load errorF10.list-statesMapped behavior

Scenario F10--load-error · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Reload the page.
  2. Wait
    Wait for “Profiles load error” to appear.
  3. Wait
    Wait for “Provider connections load error” to appear.
  4. Do
    Reload the page.
  5. Setup
    Change the run-owned service state: stop agent-server.
  6. Do
    Reload the page.
  7. Do
    Restart the run-owned stack as specified.

Expect

a spinner shows while profiles load; a load failure shows Failed to load profiles in the theme's error colour.

Exact CLI recipe & bug checks

With any profile linked to a connection (here QA_adv → QA_ds_conn, from the Link bullet), make this run's connections store unreadable (shell arrange, this run only): F=$OH_VERIFY_RUN/private/provider-connections/provider_connections.json; cp $F $F.bak; printf '{not json' > $F. control-openhands api GET /api/profiles is now 400 (Provider connections file is unreadable: …). Run control-openhands browser reload, control-openhands browser wait 'testid=profiles-load-error' and control-openhands browser wait 'testid=provider-connections-load-error' (it can appear one retry after the profiles error): the first reads Failed to load profiles in the theme's error colour (browser eval "getComputedStyle(document.querySelector('[data-testid=profiles-load-error] p')).color" is rgb(231, 106, 94) on OpenHands-Neutral and rgb(192, 40, 45) on Light+, #18330), with both Add buttons still shown and provider-connections-load-error below it (F11.load-error); screenshot --feature F10.list-states --name load-error-neutral. Restore with cp $F.bak $F && rm $F.bak and browser reload. The backend-down route does not reach it: control-openhands service stop agent-server and control-openhands browser reload do not reach testid=profiles-load-error: the whole app is replaced by the disconnected-backend screen (testid=agent-server-onboarding-screen, toasts An error occurred (twice, one after the other) and Could not connect to the configured agent server. Make sure it is running and reachable, then reload the page. (#18051); the Manage backends row detail still reads HTTP request failed (502 Bad Gateway): "Bad Gateway: connect ECONNREFUSED 127.0.0.1:<agent-server port>"). The list error needs /api/profiles alone to fail. Bring the stack back with control-openhands restart.

CleanupF10 · setup / cleanupSupport step

Scenario F10--cleanup · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/llm.

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/llm (the Load error bullet left the disconnected-backend screen), then delete every QA_*, deepseek-chat and deepseek-v3 profile through the row menu, linked ones first; control-openhands browser count 'testid=profile-group-header' drops to 0 once no linked row is left (F10.list-grouped). Then delete QA_ds_conn (testid=provider-connection-row >> has-text=QA_ds_conn >> testid=provider-connection-menu-trigger, testid=provider-connection-actions-menu >> testid=provider-connection-delete, testid=delete-provider-connection-confirm; toast Connection "QA_ds_conn" deleted). With no profiles left, testid=profiles-empty returns and api GET /api/profiles is {"profiles":[],"active_profile":null}.

Embedded editor starts on BasicF10.create-prefill · F10.editKnown failureKnown issueBlocked

Scenario F10--embedded-editor-starts-on-basic · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/llm.
  2. Do
    Choose “Add llm profile”.

Expect

a new editor prefills the backend default model with a blank key, base URL and connection; the other LLM options it shows come from the agent's current LLM settings, i.e. the last applied Default profile (the editor opens on the tab they need), and Save stores what the form shows.

Edit loads the profile (Editing profile "<name>" - save to apply changes) with the stored key masked; Save stays disabled until something changes; a changed name renames before saving; the stored key is kept.

Known bug observations

Known failure, reproduced 2026-10-12: All is selected. Back, open the exact QA_LOOPBACK row menu and testid=profile-actions-menu >> testid=profile-edit; the tablist also selects All. Repeat Add/Edit once, then Back without saving. This exercises the non-default settings condition; it does not claim that a default-only profile opens on All.

Linked known issues

Known issue OpenHands/software-agent-sdk#5890: Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (F10.edit, F10.default-used).

Known issue #17806: A model-only Save must preserve an untouched key. Current SDK1.55 refused-loopback validation rejects Save before persistence; the stored key marker remains set. Completing Basic/Advanced preserved-key and runtime-switch proof needs an approved route that accepts preflight. Keep dummy traffic on refused loopback until an external route is authorized; a rejected Save does not establish either key loss or successful preservation through Save. (F10.edit).

Exact CLI recipe & bug checks

Arrange an active disposable QA_LOOPBACK profile through the real profiles API with llm.model openai/gpt-4o, a dummy key, llm.base_url http://127.0.0.1:1/v1, num_retries 0 and temperature 0.4 (no completion). Run control-openhands browser goto /settings/llm, control-openhands browser click 'testid=add-llm-profile' and control-openhands browser snapshot 'role=tablist'. Expected by #17801: Basic selected each time the embedded form opens. Known failure, reproduced 2026-10-12: All is selected. Back, open the exact QA_LOOPBACK row menu and testid=profile-actions-menu >> testid=profile-edit; the tablist also selects All. Repeat Add/Edit once, then Back without saving. This exercises the non-default settings condition; it does not claim that a default-only profile opens on All.

Off-catalog Basic modelF10.model-pickerKnown failure

Scenario F10--off-catalog-basic-model · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “OpenRouter” in “Llm provider”.
  2. Do
    Enter “thinkingmachines/inkling-small:free” in “Llm model”.
  3. Do
    Press Tab.

Expect

the Basic provider and model comboboxes group options as Verified Models and Other Models; changing provider clears the model; the OpenHands provider shows account and key help.

Known bug observations

Known failure, reproduced 2026-10-12: the value is empty on both attempts (#17577, concrete companion of #17233). Capture browser screenshot --feature F10.model-picker --name off-catalog-discarded and Back without saving. The Advanced provider-prefix/authenticated-save half needs a working OpenRouter route; this check proves the Basic entry dead end.

Exact CLI recipe & bug checks

From the list open Add, click testid=sdk-section-basic-toggle, then control-openhands browser choose 'testid=llm-provider-input' OpenRouter. Run control-openhands browser fill 'testid=llm-model-input' 'thinkingmachines/inkling-small:free', control-openhands browser press Tab and control-openhands browser value 'testid=llm-model-input'; repeat fill/Tab once. Expected: keep the model entered under the selected provider. Known failure, reproduced 2026-10-12: the value is empty on both attempts (#17577, concrete companion of #17233). Capture browser screenshot --feature F10.model-picker --name off-catalog-discarded and Back without saving. The Advanced provider-prefix/authenticated-save half needs a working OpenRouter route; this check proves the Basic entry dead end.

Azure Base URL on BasicF10.model-pickerKnown failure

Scenario F10--azure-base-url-on-basic · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Azure” in “Llm provider”.
  2. Do
    Choose “Sdk section advanced toggle”.
  3. Do
    Choose “Sdk section basic toggle”.

Expect

the Basic provider and model comboboxes group options as Verified Models and Other Models; changing provider clears the model; the OpenHands provider shows account and key help.

Known bug observations

Known failure, reproduced 2026-10-12 (#16726): Basic count 0, after browser click 'testid=sdk-section-advanced-toggle' count 1, and after browser click 'testid=sdk-section-basic-toggle' count 0 again. Screenshot Basic and Back without saving. No Azure account is needed for this form omission; its reported authenticated conversation error is outside this observation.

Exact CLI recipe & bug checks

Open Add, explicitly select Basic and control-openhands browser choose 'testid=llm-provider-input' Azure. control-openhands browser count 'testid=base-url-input' should be 1 for Azure's required endpoint. Known failure, reproduced 2026-10-12 (#16726): Basic count 0, after browser click 'testid=sdk-section-advanced-toggle' count 1, and after browser click 'testid=sdk-section-basic-toggle' count 0 again. Screenshot Basic and Back without saving. No Azure account is needed for this form omission; its reported authenticated conversation error is outside this observation.

Slow invalid preflightF10.create-validationKnown failureKnown issue

Scenario F10--slow-invalid-preflight · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Arrange the disposable dummy-key QA_LOOPBACK profile from the embedded-editor bullet (all model traffic to http://127.0.0.1:1/v1). Open Add, explicitly select All and fill testid=sdk-settings-llm.num_retries with 5; select Advanced, fill llm-custom-model-input with openai/gpt-4o, base-url-input with the refused loopback URL, the key via --value-file <dummy-key file> and profile-name-input with QA_SLOW. Clear browser network --filter QA_SLOW --clear, click Save with --observe 'testid=save-profile-btn' --observe-ms 1000, then inspect toasts and network after the 30-second client deadline. Expected: refuse the unreachable endpoint and save nothing. Known failure, reproduced twice 2026-10-12 (#15778, slow branch of OpenHands/software-agent-sdk#5100): Validating... lasts until the browser aborts; 30 seconds after the validation POST, a profile POST succeeds with 201 and the profile appears in the list. The frontend deliberately returns no verdict for an abort, then saves. Repeating under QA_SLOW_2 behaves the same; capture the list and browser network --filter QA_SLOW_2 --bodies. With retries 1, the same refused endpoint instead failed in under a second, showed Connection error and left GET /api/profiles/QA_INVALID at 404 on both saves. Delete the two saved QA_SLOW profiles via the real API as fixture cleanup. This does not verify OpenHands/software-agent-sdk#5100's separate exhausted-budget 429 branch, which needs an approved real route returning that provider verdict.

Expect

a Save whose validation call fails toasts the provider error and keeps the editor open with nothing saved.

Known bug observations

Known failure, reproduced twice 2026-10-12 (#15778, slow branch of OpenHands/software-agent-sdk#5100): Validating... lasts until the browser aborts; 30 seconds after the validation POST, a profile POST succeeds with 201 and the profile appears in the list. The frontend deliberately returns no verdict for an abort, then saves. Repeating under QA_SLOW_2 behaves the same; capture the list and browser network --filter QA_SLOW_2 --bodies. With retries 1, the same refused endpoint instead failed in under a second, showed Connection error and left GET /api/profiles/QA_INVALID at 404 on both saves. Delete the two saved QA_SLOW profiles via the real API as fixture cleanup. This does not verify OpenHands/software-agent-sdk#5100's separate exhausted-budget 429 branch, which needs an approved real route returning that provider verdict.

Linked known issues

Known issue #17785: Saving DeepSeek-v4-flash on Windows Desktop can fail with missing cache_creation_tokens instead of saving. The reported path needs a Windows Desktop 1.24.0 environment with its SDK dependency resolution and an approved DeepSeek key; this macOS source run cannot establish that result. (F10.create-validation).

Known issue OpenHands/software-agent-sdk#5862: DeepSeek invalid-key401 responses can be classified as bad-request errors. Blocked by this pass's loopback-only provider routing: a real invalid-key validation request to DeepSeek must be authorized to capture its actual401 shape. No usable key or successful inference is needed, but a loopback connection error does not prove this classification. (F10.create-validation).

Exact CLI recipe & bug checks

Arrange the disposable dummy-key QA_LOOPBACK profile from the embedded-editor bullet (all model traffic to http://127.0.0.1:1/v1). Open Add, explicitly select All and fill testid=sdk-settings-llm.num_retries with 5; select Advanced, fill llm-custom-model-input with openai/gpt-4o, base-url-input with the refused loopback URL, the key via --value-file <dummy-key file> and profile-name-input with QA_SLOW. Clear browser network --filter QA_SLOW --clear, click Save with --observe 'testid=save-profile-btn' --observe-ms 1000, then inspect toasts and network after the 30-second client deadline. Expected: refuse the unreachable endpoint and save nothing. Known failure, reproduced twice 2026-10-12 (#15778, slow branch of OpenHands/software-agent-sdk#5100): Validating... lasts until the browser aborts; 30 seconds after the validation POST, a profile POST succeeds with 201 and the profile appears in the list. The frontend deliberately returns no verdict for an abort, then saves. Repeating under QA_SLOW_2 behaves the same; capture the list and browser network --filter QA_SLOW_2 --bodies. With retries 1, the same refused endpoint instead failed in under a second, showed Connection error and left GET /api/profiles/QA_INVALID at 404 on both saves. Delete the two saved QA_SLOW profiles via the real API as fixture cleanup. This does not verify OpenHands/software-agent-sdk#5100's separate exhausted-budget 429 branch, which needs an approved real route returning that provider verdict.

Deleted title profile after failed cleanupF10.deleteKnown failure

Scenario F10--deleted-title-profile-after-failed-cleanup · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Arrange a non-default dummy loopback profile QA_Titles3, select it on Application's Title generation model dropdown and Save there (this also hydrates the frontend settings cache). Navigate back via sidebar-settings-/settings/llm, open the exact profile row menu and Delete. On macOS arrange a selective real write failure by recording settings.json's original file flags, setting only $OH_VERIFY_RUN/private/settings.json to stat.UF_IMMUTABLE with os.chflags, and restoring its flags in a finally block immediately after the delete observation. Child profiles/ stays writable and settings stay readable; ordinary chmod is insufficient because SDK Save repairs directory permissions. Clear the network log, confirm Delete, read toasts, the exact row count and api GET /api/profiles/QA_Titles3. Expected: remove the row despite the cleanup error. Known failure, reproduced twice 2026-10-12 (#17520): DELETE returns 200, the following title-cleanup PATCH returns 500 on three retries, the error surfaces, but the row remains (1) while the API returns 404. Repeat with QA_Titles4. Restore flags before doctor; clear the stale title preference through the real settings API. No provider validation or network interception is needed.

Expect

Delete asks Are you sure you want to delete the profile "<name>"? This action cannot be undone. with Cancel focused; Cancel keeps the row, Delete removes it with a toast.

Known bug observations

Known failure, reproduced twice 2026-10-12 (#17520): DELETE returns 200, the following title-cleanup PATCH returns 500 on three retries, the error surfaces, but the row remains (1) while the API returns 404. Repeat with QA_Titles4. Restore flags before doctor; clear the stale title preference through the real settings API. No provider validation or network interception is needed.

Exact CLI recipe & bug checks

Arrange a non-default dummy loopback profile QA_Titles3, select it on Application's Title generation model dropdown and Save there (this also hydrates the frontend settings cache). Navigate back via sidebar-settings-/settings/llm, open the exact profile row menu and Delete. On macOS arrange a selective real write failure by recording settings.json's original file flags, setting only $OH_VERIFY_RUN/private/settings.json to stat.UF_IMMUTABLE with os.chflags, and restoring its flags in a finally block immediately after the delete observation. Child profiles/ stays writable and settings stay readable; ordinary chmod is insufficient because SDK Save repairs directory permissions. Clear the network log, confirm Delete, read toasts, the exact row count and api GET /api/profiles/QA_Titles3. Expected: remove the row despite the cleanup error. Known failure, reproduced twice 2026-10-12 (#17520): DELETE returns 200, the following title-cleanup PATCH returns 500 on three retries, the error surfaces, but the row remains (1) while the API returns 404. Repeat with QA_Titles4. Restore flags before doctor; clear the stale title preference through the real settings API. No provider validation or network interception is needed.

Connection-linked provider dispatchF10.custom-model · F10.connection-linkKnown failureKnown issue

Scenario F10--connection-linked-provider-dispatch · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    From /settings/llm Add > Advanced, save QA_QWEN with llm-custom-model-input qwen linked through llm-provider-connection-input to an own OpenAI connection at refused loopback. Activate it and start two real Home chats; inspect conversation events <id> --kinds ConversationErrorEvent --last 2. Expected: the chosen OpenAI-compatible connection supplies the transport. Known failure, reproduced twice 2026-10-12 (#17261): Save succeeds, but each actual chat fails before HTTP with LLM Provider NOT provided for model=qwen. The reported Windows installation, unlinked preflight and successful local inference are further branches. With a separate own connection whose provider is custom_openai, repeat the UI Save/start for ollama-local/qwen: the same failure reproduces twice for the namespace reported in #17195. Its exact exposed model suffix is absent from the report, so qwen is an owned diagnostic suffix; this is not a complete 9Router/Ollama inference check. Restore the neutral active profile and remove only the own profiles/connections.

Expect

the Advanced tab takes a free-text model and Base URL; an empty model is refused with Model is required.

with connections present the editor shows a Provider connection selector; linking hides API key and Base URL, skips validation on Save and regroups the row; None restores the key field.

Known bug observations

Known failure, reproduced twice 2026-10-12 (#17261): Save succeeds, but each actual chat fails before HTTP with LLM Provider NOT provided for model=qwen. The reported Windows installation, unlinked preflight and successful local inference are further branches. With a separate own connection whose provider is custom_openai, repeat the UI Save/start for ollama-local/qwen: the same failure reproduces twice for the namespace reported in #17195. Its exact exposed model suffix is absent from the report, so qwen is an owned diagnostic suffix; this is not a complete 9Router/Ollama inference check. Restore the neutral active profile and remove only the own profiles/connections.

Linked known issues

Known issue OpenHands/software-agent-sdk#4613: A namespaced model such as openai/openai/example-model can have its inner provider prefix stripped again in Responses API and reach the provider as example-model. Reproduction needs an approved OpenAI-compatible endpoint serving the namespaced model and its exact stored API-mode settings; a loopback connection refusal cannot prove the outbound identifier. (F10.custom-model).

Known issue OpenHands/software-agent-sdk#4250: A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (F10.custom-model, F10.default-used).

Known issue OpenHands/software-agent-sdk#4247: An LM Studio configuration can fail with LLM Provider NOT provided although its models endpoint works. Reproduction needs the reachable LM Studio service and the exact model ID/provider-qualified config, which the report does not supply as text; the reported CLI/Docker SDK 1.16.1 installation is not this source stack. (F10.custom-model).

Exact CLI recipe & bug checks

From /settings/llm Add > Advanced, save QA_QWEN with llm-custom-model-input qwen linked through llm-provider-connection-input to an own OpenAI connection at refused loopback. Activate it and start two real Home chats; inspect conversation events <id> --kinds ConversationErrorEvent --last 2. Expected: the chosen OpenAI-compatible connection supplies the transport. Known failure, reproduced twice 2026-10-12 (#17261): Save succeeds, but each actual chat fails before HTTP with LLM Provider NOT provided for model=qwen. The reported Windows installation, unlinked preflight and successful local inference are further branches. With a separate own connection whose provider is custom_openai, repeat the UI Save/start for ollama-local/qwen: the same failure reproduces twice for the namespace reported in #17195. Its exact exposed model suffix is absent from the report, so qwen is an owned diagnostic suffix; this is not a complete 9Router/Ollama inference check. Restore the neutral active profile and remove only the own profiles/connections.

Bedrock named-profile importF10.default-usedKnown issueBlocked

Scenario F10--bedrock-named-profile-import · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    In the run-owned launcher environment set AWS_EC2_METADATA_DISABLED=true, then restart and doctor; verify that flag without printing other environment contents. Arrange an own profile with model: bedrock/anthropic.claude-3-haiku-20240307-v1:0, aws_profile_name: qa_nonexistent, region us-east-1, retries 0, and both base_url and aws_bedrock_runtime_endpoint http://127.0.0.1:1/v1. A dummy-only API-key presence marker is needed for Canvas's send gate; the SDK filters that non-Bedrock key on this IAM path. Activate and start a real Home chat; inspect the error privately. Expected: the dependency imports and the missing named profile is reported before any AWS/provider request. A connection error cannot prove successful Bedrock inference. Restore the launcher environment and active profile, then remove only the own fixture.

Expect

a conversation started after creating a profile in the UI runs on the Default profile's model.

Linked known issues

Known issue #17706: A freshly resolved npm-launcher environment can reject openhands/claude-* conversations because LiteLLM sends prompt_cache_key. Reproduction needs an approved OpenHands proxy key with an Anthropic model and the affected fresh transitive dependency resolution; a keyless initialization cannot prove provider acceptance. (F10.default-used).

Known issue OpenHands/software-agent-sdk#4934: Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (F10.schema-validation, F10.default-used).

Known issue OpenHands/software-agent-sdk#4250: A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (F10.custom-model, F10.default-used).

Known issue OpenHands/software-agent-sdk#5890: Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (F10.edit, F10.default-used).

Known issue OpenHands/software-agent-sdk#5838: A live LLM switch during a running tool step can be acknowledged before the new LLM is persisted for GET/restart. Blocked: needs a genuine model-issued bounded tool step (for example sleep 20) and an approved starting model. Refused-loopback initialization never enters that step. (F10.default-used).

Exact CLI recipe & bug checks

In the run-owned launcher environment set AWS_EC2_METADATA_DISABLED=true, then restart and doctor; verify that flag without printing other environment contents. Arrange an own profile with model: bedrock/anthropic.claude-3-haiku-20240307-v1:0, aws_profile_name: qa_nonexistent, region us-east-1, retries 0, and both base_url and aws_bedrock_runtime_endpoint http://127.0.0.1:1/v1. A dummy-only API-key presence marker is needed for Canvas's send gate; the SDK filters that non-Bedrock key on this IAM path. Activate and start a real Home chat; inspect the error privately. Expected: the dependency imports and the missing named profile is reported before any AWS/provider request. A connection error cannot prove successful Bedrock inference. Restore the launcher environment and active profile, then remove only the own fixture.

Unavailable subscription upstream is handledF10.subscriptionKnown failureKnown issueBlocked

Scenario F10--unavailable-subscription-upstream-is-handled · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Expected: device-start failure reports502/504 rather than a server crash. Known failure (reproduced twice 2026-10-16, OpenHands/software-agent-sdk#5880): real Connect returned500 with the backend offline. The block uses an owned offline proxy; browser offline alone does not cut the backend request. The cached runtime must already exist: UV_OFFLINE=1 prevents its dependency resolver from reaching the blocked network. The block changes only this run's environment, keeps model defaults on refused loopback, and restores/restarts it in EXIT. The poll branch remains blocked without a genuine pending device token.

Expect

Authentication → ChatGPT subscription swaps the model fields for a subscription model list and a device-login card (code, sign-in link, poll, cancel); switching back restores the API-key model.

Known bug observations

Known failure (reproduced twice 2026-10-16, OpenHands/software-agent-sdk#5880): real Connect returned500 with the backend offline. The block uses an owned offline proxy; browser offline alone does not cut the backend request. The cached runtime must already exist: UV_OFFLINE=1 prevents its dependency resolver from reaching the blocked network. The block changes only this run's environment, keeps model defaults on refused loopback, and restores/restarts it in EXIT. The poll branch remains blocked without a genuine pending device token.

Linked known issues

Known issue #17391: ChatGPT Business subscription login can be refused or use a fallback model with a 404. Completing the entitlement and selected-model checks needs an approved ChatGPT Business workspace login and the reported Docker subscription runtime. (F10.subscription).

Exact CLI recipe & bug checks

Expected: device-start failure reports502/504 rather than a server crash. Known failure (reproduced twice 2026-10-16, OpenHands/software-agent-sdk#5880): real Connect returned500 with the backend offline. The block uses an owned offline proxy; browser offline alone does not cut the backend request. The cached runtime must already exist: UV_OFFLINE=1 prevents its dependency resolver from reaching the blocked network. The block changes only this run's environment, keeps model defaults on refused loopback, and restores/restarts it in EXIT. The poll branch remains blocked without a genuine pending device token.

sh known-bug F10.subscription offline-start

  QA_SUB_ENV="$OH_VERIFY_RUN/private/qa-known-subscription-env.json"
  cp "$OH_VERIFY_RUN/private/launcher-env.json" "$QA_SUB_ENV"
  trap 'cp "$QA_SUB_ENV" "$OH_VERIFY_RUN/private/launcher-env.json"; rm -f "$QA_SUB_ENV"; control-openhands restart --timeout 120 >/dev/null; control-openhands browser goto /settings/llm >/dev/null' EXIT
  python3 - <<'PYENV'
  from pathlib import Path
  import os,json
  p=Path(os.environ['OH_VERIFY_RUN'])/'private/launcher-env.json'
  d=json.loads(p.read_text())
  assert d.get('OPENAI_BASE_URL')=='http://127.0.0.1:1/v1'
  assert d.get('OPENAI_API_BASE')=='http://127.0.0.1:1/v1'
  d['UV_OFFLINE']='1'
  for key in ['HTTP_PROXY','HTTPS_PROXY','ALL_PROXY','http_proxy','https_proxy','all_proxy']: d[key]='http://127.0.0.1:1'
  d['NO_PROXY']=d['no_proxy']='127.0.0.1,localhost,::1'
  p.write_text(json.dumps(d))
  PYENV
  control-openhands restart --timeout 120
  control-openhands doctor
  control-openhands browser viewport desktop
  control-openhands browser goto /settings/llm
  control-openhands browser click 'testid=add-llm-profile'
  control-openhands browser click 'testid=sdk-section-basic-toggle'
  control-openhands browser choose 'testid=llm-auth-type-input' 'ChatGPT subscription'
  control-openhands browser network --clear
  control-openhands browser click 'testid=subscription-connect' --observe 'role=status' --observe-ms 1000
  control-openhands assert --field recent.0.status --matches '^(502|504)$' --bug 'recent.0.status eq 500 && recent.0.path eq "/api/llm/subscription/openai/device/start"' -- browser network --filter '/api/llm/subscription/openai/device/start'  # bug
Schema unavailableF10.schema-unavailableBehavior contract

Scenario F10--contract-f10-schema-unavailable · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do

This behavior has no separately named recipe in this snapshot. Use the family’s entry points and the contract below; the page does not invent a replay.

Expect

a backend without the settings-schema endpoints shows SDK settings schema unavailable. with an upgrade or session-key hint.

Cloud readonlyF10.cloud-readonlyBehavior contract

Scenario F10--contract-f10-cloud-readonly · Permalink · Source

Given

Use F10’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do

This behavior has no separately named recipe in this snapshot. Use the family’s entry points and the contract below; the page does not invent a replay.

Expect

Cloud org members without edit rights see the list read-only (no Add buttons, no row menus, no Provider connections section).

Family notes, prerequisites for known issues & limits Known issueBlocked
  • Save always runs a real completion against the provider (except for connection-linked profiles). A bad or missing key fails fast with the provider's message; budget one tiny call per Save.
  • The editor's starting tab is inferred from the values it loads. Edit opens on whichever tab the profile's non-default values need (QA_renamed and QA_adv opened on Basic, QA_broken on All). A new editor opens on Basic only while the agent's LLM settings are all defaults: once the Default profile has a temperature (the schema step) it opens on All, and this survives deleting every profile, because the agent settings keep the last applied values. Click the tab you need before touching its fields, or llm-provider-input / llm-custom-model-input / llm-api-key-help-anchor will not exist.
  • control-openhands service stop agent-server cannot show this page's error states: the whole app switches to the disconnected-backend screen.
  • In Edit, llm-api-key-input holds the stored encrypted key token (about 140 masked characters), not an empty field. Never run browser value on it: it prints the token. Leave it untouched to keep the key.
  • An open provider or model listbox intercepts clicks on the editor's buttons (cancel-profile-btn, save-profile-btn time out with "subtree intercepts pointer events"): browser press Escape first.
  • Provider and model options render in a portal outside the editor and modals: select testid=provider-item-<name> unscoped. Only verified providers have provider-item-* IDs and only Other models have model-item-* IDs; use role=option[name="..."][exact] for the rest. browser testids --filter model-item lists only items scrolled into view; browser count finds the rest.
  • The auto-filled Profile Name never reuses an existing profile's name: picking a model whose name is already a profile's (for example deepseek-flash after llm preset deepseek) empties the field, which disables Save. Recipes fill profile-name-input with a QA_* name anyway.
  • A duplicate profile name is flagged: aria-invalid true and a taken-name alert. The alert has its own wording, An LLM profile with this name already exists. ….
  • The Rename modal does not check duplicates client-side (Rename stays enabled); the server's 409 comes back as the toast Profile '<name>' already exists, and browser errors --app-only counts one http-error and one console.error for it.
  • Set as default (and deleting the Default) rewrites the agent's active LLM settings (control-openhands llm show). Deleting every profile leaves the last applied model and key in agent settings.
  • The app never leaves profiles without a Default: src/routes/root-layout.tsx runs useEnsureActiveProfile (src/hooks/use-ensure-active-profile.ts), which activates the first profile with a key (else the first profile) whenever profiles exist but none is Default, whether never activated or because the Default was deleted (this is what promotes a profile in F10.delete-default). So llm set --no-activate keeps the Default only while another profile is Default: on a run whose only profile is the arranged one, control-openhands api GET /api/profiles --pick active_profile flips from null to it at the next page load (browser reload) and the agent's LLM settings follow (llm show).
  • Profiles created by Add models are named from the model (deepseek-chat, …), not QA_*. The chooser hides a model when a profile with that name exists, not when a profile uses that model. Clean them up before the connection: the server refuses to delete a connection while profiles reference it.
  • A profile linked to a missing connection is grouped under a header showing the raw connection id (00000000…) instead of a name.
  • The All tab logs many [i18n] Missing translation for key "SCHEMA$LLM$..." console warnings; they are not page errors.
  • The list follows the server's order, which sorts by profile file name: QA_pro-copy-1 lists before QA_pro-copy (- sorts before .json's .). Do not assert a strict alphabetical order on names that are prefixes of each other.
  • Toasts last only a few seconds. A browser screenshot taken after a slow command can miss them, so assert with browser wait-text or browser toasts.
  • A new profile keeps the inherited LLM options the editor shows, such as temperature.
  • Subscription models are not loaded yet. uses the translated SETTINGS$SUBSCRIPTION_MODELS_NOT_LOADED copy in settings and onboarding. It shows when a subscription profile is saved before GET /api/llm/subscription/openai/models answers; checking that branch needs the real request to be pending or failed.
  • Known issue #17785: Saving DeepSeek-v4-flash on Windows Desktop can fail with missing cache_creation_tokens instead of saving. The reported path needs a Windows Desktop 1.24.0 environment with its SDK dependency resolution and an approved DeepSeek key; this macOS source run cannot establish that result. (F10.create-validation).
  • Known issue #17706: A freshly resolved npm-launcher environment can reject openhands/claude-* conversations because LiteLLM sends prompt_cache_key. Reproduction needs an approved OpenHands proxy key with an Anthropic model and the affected fresh transitive dependency resolution; a keyless initialization cannot prove provider acceptance. (F10.default-used).
  • Known issue #17391: ChatGPT Business subscription login can be refused or use a fallback model with a 404. Completing the entitlement and selected-model checks needs an approved ChatGPT Business workspace login and the reported Docker subscription runtime. (F10.subscription).
  • Known issue OpenHands/software-agent-sdk#5181: The SDK can turn an advertised output limit into a requested default cap and silently drop an explicitly enabled reasoning_effort. The narrowed maintainer scope needs an authenticated OpenRouter/proxy route exposing lower-limit endpoints and capability metadata, or a real endpoint accepting the explicit override; a synthetic empty-metadata response would not establish production behavior. (F10.schema-validation).
  • Known issue OpenHands/software-agent-sdk#4934: Switching to a MiniMax M3 profile can fail because the SDK sends unsupported reasoning_effort. The runtime switch needs approved working credentials for the starting model and MiniMax route; the reported provider rejection cannot be proved with a loopback connection error. (F10.schema-validation, F10.default-used).
  • Known issue OpenHands/software-agent-sdk#4613: A namespaced model such as openai/openai/example-model can have its inner provider prefix stripped again in Responses API and reach the provider as example-model. Reproduction needs an approved OpenAI-compatible endpoint serving the namespaced model and its exact stored API-mode settings; a loopback connection refusal cannot prove the outbound identifier. (F10.custom-model).
  • Known issue OpenHands/software-agent-sdk#4511: Disabling prompt caching can still emit cache_control on preexisting cache-marked tool messages. The report only supplies a Python serializer reproduction; a Canvas check needs the exact persisted/tool-message precondition and a provider request that exposes the cache markers, with approved model access. (F10.schema-validation).
  • Known issue OpenHands/software-agent-sdk#4250: A Cloud Workers AI configuration can fail on context-window validation or be routed as OpenAI with an invalid Cloudflare key. Reproduction needs a disposable OpenHands Cloud account, its Workers AI endpoint/key and the exact saved model configuration; the report supplies no complete configuration steps. (F10.custom-model, F10.default-used).
  • Known issue OpenHands/software-agent-sdk#4247: An LM Studio configuration can fail with LLM Provider NOT provided although its models endpoint works. Reproduction needs the reachable LM Studio service and the exact model ID/provider-qualified config, which the report does not supply as text; the reported CLI/Docker SDK 1.16.1 installation is not this source stack. (F10.custom-model).
  • Known issue OpenHands/software-agent-sdk#5890: Switching the conversation LLM can lose the managed-key refresh-on401 hook. Blocked: needs an approved managed-key refresh source and an allow-listed real model endpoint that returns401 and accepts the refreshed key. Refused loopback cannot establish the refresh/retry outcome; do not supply a fake successful completion. (F10.edit, F10.default-used).
  • Known issue OpenHands/software-agent-sdk#5862: DeepSeek invalid-key401 responses can be classified as bad-request errors. Blocked by this pass's loopback-only provider routing: a real invalid-key validation request to DeepSeek must be authorized to capture its actual401 shape. No usable key or successful inference is needed, but a loopback connection error does not prove this classification. (F10.create-validation).
  • Known issue OpenHands/software-agent-sdk#5838: A live LLM switch during a running tool step can be acknowledged before the new LLM is persisted for GET/restart. Blocked: needs a genuine model-issued bounded tool step (for example sleep 20) and an approved starting model. Refused-loopback initialization never enters that step. (F10.default-used).
  • Known issue #17806: A model-only Save must preserve an untouched key. Current SDK1.55 refused-loopback validation rejects Save before persistence; the stored key marker remains set. Completing Basic/Advanced preserved-key and runtime-switch proof needs an approved route that accepts preflight. Keep dummy traffic on refused loopback until an external route is authorized; a rejected Save does not establish either key loss or successful preservation through Save. (F10.edit).

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.