Engel’s notebook

Condenser, agent context and verification settings

F15 · 23 mapped behaviors in the Self-Healing Codebase Project. Follow the prerequisites, actions and expected results; expand the original CLI recipe to replay it.

778 behavior IDs · 28 families · 844 named recipes & support steps
16 October 2026 · PR #18334 at 9a3c3a69
Canvas 1.27.1 · Agent Server 1.55.0 · automation 1.20.0

This is the map at the revision above, including its open bug notes. It does not incorporate later changes on main or claim a fresh verification run. Setup & reading guide · Download the scenario inventory

All scenarios & marker key · Open a script to read Given / Do / Expect. Expand the exact recipe for replayable commands.

F15 Condenser, agent context and verification settings

23 behaviors · 28 scripts & contracts · Maintained source

Entry points & prerequisites

What this family covers

Three settings pages are generated from the Agent Server's settings schemas: Condenser (summarize long chats), Agent Context (persistent agent memory) and Verification (confirmation mode, security analyzer and the critic). Each page shows the schema's fields in Basic/Advanced/All tiers (only the tiers that have fields), hides fields whose parent switch is off, and saves through one Save Changes button that stays disabled until a value differs from the stored one.

How a user gets here

  • Sidebar settings gear (backend-selector-settings-link, lands on /settings/agents), then Condenser, Agent Context or Verification in the settings navigation (sidebar-settings-/settings/condenser, sidebar-settings-/settings/agent-context, sidebar-settings-/settings/verification).
  • Direct URLs /settings/condenser, /settings/agent-context, /settings/verification.
  • Command menu (Control+k or Meta+k, or command-menu-trigger in the sidebar): Condenser settings, Agent Context and Verification settings; "memory" finds both Condenser settings and Agent Context.
  • The effects show up in new conversations only: the confirmation prompt in the chat, and the memory instructions in the conversation's system prompt.

Given

Preconditions:

  • Baseline state (launched, doctored, onboard --skip done). No LLM is needed for the settings pages.
  • F15.confirmation-mode-effect and F15.agent-context-effect need an active LLM profile (control-openhands llm preset deepseek).
  • F15.schema-unavailable needs an agent server that serves /api/settings but answers 404 for /api/settings/agent-schema (an older release via launch --new --sdk-ref <old>); not available in this checkout's pinned stack.
  • Stored values are the defaults: control-openhands api GET /api/settings shows agent_settings.condenser.enabled true, max_size 240, condenser_kind llm_summarizing; agent_context.load_memory false; verification.critic_enabled false; conversation_settings.confirmation_mode false.
Open Condenser from the sidebarF15.condenser-pageMapped behavior

Scenario F15--open-condenser-from-the-sidebar · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Settings”.
  2. Do
    Choose “Condenser settings”.

Expect

Condenser shows its subtitle, Basic and All tabs (no Advanced), and in Basic only the Enable Memory Condensation switch.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=backend-selector-settings-link' --expect-url '/settings/agents(\?|$)', control-openhands browser click 'testid=sidebar-settings-/settings/condenser' --expect-url '/settings/condenser(\?|$)', control-openhands browser text 'testid=settings-page-subtitle' (Summarize long chats to stay within context limits.) and control-openhands browser snapshot 'testid=condenser-settings-screen'. The snapshot shows tabs Basic [selected] and All, the text Enable Memory Condensation and button "Save Changes" [disabled]; control-openhands browser count 'testid=sdk-section-advanced-toggle' is 0. Read the switch with control-openhands browser eval "document.querySelector('[data-testid=\"sdk-settings-condenser.enabled\"]').checked" (true).

All tierF15.condenser-all-viewMapped behavior

Scenario F15--all-tier · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section all toggle”.

Expect

the All tab lists the eight condenser fields with Title Case labels, plain descriptions and their stored values (Max Size 240, Keep First Events 2, …).

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=sdk-section-all-toggle', control-openhands browser count 'testid=condenser-settings-screen >> input' (8) and control-openhands browser value 'testid=sdk-settings-condenser.max_size' (240). control-openhands browser snapshot 'testid=condenser-settings-screen' labels the fields Enable Memory Condensation, Max Size, Condenser Kind (combobox value LLM Summarizer), Max Tokens, Keep First Events, Minimum Progress, Hard Reset Retries and Hard Reset Scaling, each with a plain-language paragraph and all but the switch marked Optional (Condenser Kind: How the condenser shortens long conversations. LLM Summarizer replaces older events with a summary written by the LLM; None keeps the full history.).

Dirty stateF15.save-dirty-stateMapped behavior

Scenario F15--dirty-state · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “120” in “Sdk settings condenser.max size”.
  2. Do
    Enter “240” in “Sdk settings condenser.max size”.

Expect

Save is disabled on load, enabled after an edit, and disabled again when the edit is reverted by hand.

Exact CLI recipe & bug checks

On the All tier: control-openhands browser enabled 'testid=save-button' is false; control-openhands browser fill 'testid=sdk-settings-condenser.max_size' 120 makes it true; control-openhands browser fill 'testid=sdk-settings-condenser.max_size' 240 makes it false again.

Save and reloadF15.condenser-saveMapped behavior

Scenario F15--save-and-reload · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Save”.
  2. Do
    Reload the page.

Expect

changing Max Size and saving shows Saving..., then the saved toast; after a reload the page opens on All with the new value.

Exact CLI recipe & bug checks

Fill Max Size with 120, then control-openhands browser click 'testid=save-button' --observe 'testid=save-button' --observe-ms 3000 (observed states Save Changes → Saving... → Save Changes) and control-openhands browser toasts (Settings saved. For old conversations, you will need to stop and restart the conversation to see the changes.). control-openhands browser enabled 'testid=save-button' is false. Run control-openhands browser reload, control-openhands browser snapshot 'role=tablist' (All [selected]: the page opens on the tier that shows the overridden field) and control-openhands browser value 'testid=sdk-settings-condenser.max_size' (120). control-openhands api GET /api/settings shows max_size 120. Keep it for the next bullet.

Dependent fieldsF15.condenser-dependentsMapped behavior

Scenario F15--dependent-fields · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Condenser settings screen” / “Enable Memory Condensation”.

Expect

turning condensation off hides the six size/reset fields (Condenser Kind stays); turning it back on un-dirties Save.

Exact CLI recipe & bug checks

On the All tier run control-openhands browser click 'testid=condenser-settings-screen >> text=Enable Memory Condensation'; the switch reads false and control-openhands browser count 'testid=condenser-settings-screen >> input' is 2 (the switch and Condenser Kind). Click the label again: the count is 8 and browser enabled 'testid=save-button' is false.

Basic save keeps hidden fieldsF15.view-scoped-saveKnown failure

Scenario F15--basic-save-keeps-hidden-fields · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section basic toggle”.
  2. Do
    Choose “Condenser settings screen” / “Enable Memory Condensation”.
  3. Do
    Choose “Save”.
  4. Do
    Reload the page.
  5. Do
    Choose “Save”.
  6. Do
    Reload the page.

Expect

saving from a narrower tier keeps the stored values of fields that tier does not show.

Known bug observations

Known failure (reproduced 2026-10-06): max_size is 240, because the save sends defaults for every field Basic does not show (#18049). After browser reload the page opens on Basic with the switch off. Restore: click the label again, browser click 'testid=save-button', browser reload; the switch reads true and the API shows enabled true (with max_size 240 today, from the reset above).

Exact CLI recipe & bug checks

With Max Size stored as 120, run control-openhands browser click 'testid=sdk-section-basic-toggle', control-openhands browser click 'testid=condenser-settings-screen >> text=Enable Memory Condensation' and control-openhands browser click 'testid=save-button'. control-openhands api GET /api/settings now shows enabled false and max_size 120. Known failure (reproduced 2026-10-06): max_size is 240, because the save sends defaults for every field Basic does not show (#18049). After browser reload the page opens on Basic with the switch off. Restore: click the label again, browser click 'testid=save-button', browser reload; the switch reads true and the API shows enabled true (with max_size 240 today, from the reset above).

Invalid valuesF15.save-errorsMapped behavior

Scenario F15--invalid-values · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “1.5” in “Sdk settings condenser.max size”.
  2. Do
    Choose “Save”.
  3. Do
    Choose “Save”.
  4. Do
    Reload the page.

Expect

a non-integer Max Size is refused in the browser with a toast; a negative one is refused by the server with a 422 toast; nothing is stored.

Exact CLI recipe & bug checks

On the All tier run control-openhands browser fill 'testid=sdk-settings-condenser.max_size' 1.5, control-openhands browser click 'testid=save-button' and control-openhands browser toasts (Expected an integer value, received: 1.5; no request is sent). Then wait for the toast to close, fill -5, control-openhands browser network --clear, control-openhands browser click 'testid=save-button' --observe 'testid=save-button' --observe-ms 2500 (Saving... for about 1.5 s), control-openhands browser toasts (HTTP request failed (422 Unprocessable Entity): {"detail":"Settings validation failed"}) and control-openhands browser network (three PATCH /api/settings). The API still shows max_size 240; browser reload drops the bad draft.

Condenser KindF15.condenser-kindKnown failure

Scenario F15--condenser-kind · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk settings condenser.condenser kind”.
  2. Do
    Choose the listbox / “None”.
  3. Do
    Choose “Save”.
  4. Do
    Reload the page.
  5. Do
    Enter “100” in “Sdk settings condenser.max size”.

Expect

Condenser Kind is a dropdown (LLM Summarizer, None, stored as llm_summarizing and no_op); the choice persists, and switching back restores the summarizer defaults.

Known bug observations

Known failure (reproduced 2026-10-06): Max Size, Max Tokens, Keep First Events, Minimum Progress, Hard Reset Retries and Hard Reset Scaling stay with their defaults, and browser fill 'testid=sdk-settings-condenser.max_size' 100 then Save gives Settings saved while the API still shows only enabled and condenser_kind and a reload shows 240 again (#18049; browser screenshot --feature F15.condenser-kind --name none-shows-summarizer-fields). Restore by choosing LLM Summarizer the same way and saving: after reload the page opens on Basic and the API shows all eight defaults again (condenser_kind llm_summarizing).

Exact CLI recipe & bug checks

On the All tier run control-openhands browser click 'testid=sdk-settings-condenser.condenser_kind', control-openhands browser snapshot 'role=listbox' (options LLM Summarizer [selected] and None), control-openhands browser click 'role=listbox >> role=option[name="None"]' and control-openhands browser click 'testid=save-button'. After browser reload the page opens on All, browser value 'testid=sdk-settings-condenser.condenser_kind' is None, and control-openhands api GET /api/settings --pick agent_settings.condenser is just {"enabled": true, "condenser_kind": "no_op"} (the labels are Canvas copy; the stored values are unchanged). The summarizer fields should hide while None is selected. Known failure (reproduced 2026-10-06): Max Size, Max Tokens, Keep First Events, Minimum Progress, Hard Reset Retries and Hard Reset Scaling stay with their defaults, and browser fill 'testid=sdk-settings-condenser.max_size' 100 then Save gives Settings saved while the API still shows only enabled and condenser_kind and a reload shows 240 again (#18049; browser screenshot --feature F15.condenser-kind --name none-shows-summarizer-fields). Restore by choosing LLM Summarizer the same way and saving: after reload the page opens on Basic and the API shows all eight defaults again (condenser_kind llm_summarizing).

Agent ContextF15.agent-context-pageMapped behavior

Scenario F15--agent-context · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Agent context settings”.

Expect

Agent Context shows a single Persistent Agent Memory switch with its help text and no tab bar.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=sidebar-settings-/settings/agent-context' --expect-url '/settings/agent-context(\?|$)', control-openhands browser text 'testid=settings-page-subtitle' (Let the agent keep notes and recall them in new conversations.), control-openhands browser count 'role=tablist' (0) and control-openhands browser snapshot 'testid=agent-context-settings-screen' (text Persistent Agent Memory, the paragraph about .openhands/memory/, button "Save Changes" [disabled]).

Persistent memory persistsF15.agent-context-saveMapped behavior

Scenario F15--persistent-memory-persists · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Agent context settings screen” / “Persistent Agent Memory”.
  2. Do
    Choose “Save”.
  3. Do
    Reload the page.

Expect

toggling Persistent Agent Memory and saving persists after reload.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=agent-context-settings-screen >> text=Persistent Agent Memory', control-openhands browser click 'testid=save-button', control-openhands browser reload and control-openhands browser eval "document.querySelector('[data-testid=\"sdk-settings-agent_context.load_memory\"]').checked" (true; API agent_context.load_memory true). Restore: click the label, save, reload; the value is false.

VerificationF15.verification-pageMapped behavior

Scenario F15--verification · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Verification settings”.

Expect

Verification shows Basic, Advanced and All tabs; with defaults every tier shows only Confirmation Mode and Enable Critic.

Exact CLI recipe & bug checks

Run control-openhands browser click 'testid=sidebar-settings-/settings/verification' --expect-url '/settings/verification(\?|$)', control-openhands browser text 'testid=settings-page-subtitle' (Confirmation prompts and security checks on actions.) and control-openhands browser snapshot 'testid=verification-settings-screen': tabs Basic [selected], Advanced, All; texts Confirmation Mode and Enable Critic. control-openhands browser count 'testid=verification-settings-screen >> input' is 2 on Advanced and on All too.

Critic and analyzer fieldsF15.verification-critic-fieldsMapped behavior

Scenario F15--critic-and-analyzer-fields · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Verification settings screen” / “Confirmation Mode”.
  2. Do
    Choose “Verification settings screen” / “"Enable Critic"”.
  3. Do
    Choose “Verification settings screen” / “"Enable Iterative Refinement"”.
  4. Do
    Reload the page.

Expect

Confirmation Mode reveals Security Analyzer (Advanced); Enable Critic reveals Iterative Refinement and Critic API Key (Basic) plus mode, server URL and model (All); Iterative Refinement reveals threshold and max iterations.

Exact CLI recipe & bug checks

On All run control-openhands browser click 'testid=verification-settings-screen >> text=Confirmation Mode', control-openhands browser click 'testid=verification-settings-screen >> text="Enable Critic"' and control-openhands browser click 'testid=verification-settings-screen >> text="Enable Iterative Refinement"'. control-openhands browser testids 'testid=verification-settings-screen' --hidden now lists sdk-settings-security_analyzer, sdk-settings-verification.critic_mode, sdk-settings-verification.critic_threshold (0.6), sdk-settings-verification.max_refinement_iterations (3), sdk-settings-verification.critic_server_url, sdk-settings-verification.critic_model_name, sdk-settings-verification.critic_api_key and help-link-verification.critic_api_key. control-openhands browser attr 'testid=sdk-settings-verification.critic_api_key' type is password; control-openhands browser eval "document.querySelector('[data-testid=\"help-link-verification.critic_api_key\"] a')?.href" is https://app.all-hands.dev/settings/api-keys. control-openhands browser screenshot --feature F15.verification-critic-fields --name all-critic-on shows the API key spanning both columns. Basic (sdk-section-basic-toggle) keeps confirmation_mode, critic_enabled, enable_iterative_refinement and critic_api_key; Advanced adds security_analyzer. The three switches are an unsaved draft that survives tab switches: run control-openhands browser reload before the next bullet, otherwise its Confirmation Mode click turns the switch off again and the save stores the critic draft.

Confirmation mode persistsF15.confirmation-mode-saveMapped behavior

Scenario F15--confirmation-mode-persists · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Verification settings screen” / “Confirmation Mode”.
  2. Do
    Choose “Save”.
  3. Do
    Reload the page.
  4. Do
    Choose “Sdk section advanced toggle”.
  5. Do
    Choose “Sdk settings security analyzer”.
  6. Do
    Choose the listbox / “None”.
  7. Do
    Choose “Save”.
  8. Do
    Reload the page.

Expect

Confirmation Mode and Security Analyzer (conversation settings) persist after reload, alongside the agent-settings fields on the same page.

Exact CLI recipe & bug checks

On /settings/verification (Basic) run control-openhands browser click 'testid=verification-settings-screen >> text=Confirmation Mode', control-openhands browser click 'testid=save-button', control-openhands browser reload and control-openhands browser eval "document.querySelector('[data-testid=\"sdk-settings-confirmation_mode\"]').checked" (true). Then control-openhands browser click 'testid=sdk-section-advanced-toggle', control-openhands browser click 'testid=sdk-settings-security_analyzer', control-openhands browser snapshot 'role=listbox' (options LLM [selected], None), control-openhands browser click 'role=listbox >> role=option[name="None"]', control-openhands browser click 'testid=save-button' and control-openhands browser reload: the tab bar shows Advanced [selected] and control-openhands browser value 'testid=sdk-settings-security_analyzer' is None. control-openhands api GET /api/settings shows conversation_settings.confirmation_mode true, security_analyzer none. Keep both for the next bullet.

Confirmation stops the agentF15.confirmation-mode-effectMapped behavior

Scenario F15--confirmation-stops-the-agent · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Start a conversation with the prompt “Run this shell command: echo qa-f15-confirm”.
  2. Wait
    Wait for the element identified in the CLI recipe to appear.
  3. Do
    Choose “Action confirm”.
  4. Wait
    Wait for the conversation to finish.
  5. Do
    Open /settings/verification.
  6. Do
    Choose “Sdk section basic toggle”.
  7. Do
    Choose “Save”.

Expect

with Confirmation Mode on and Security Analyzer None, a new conversation stops before its first command with Do you want to continue with this action? and Cancel/Continue; Continue runs it.

Exact CLI recipe & bug checks

Run control-openhands conversation start --prompt "Run this shell command: echo qa-f15-confirm" --wait --until waiting_for_confirmation,finished,idle,error --timeout 180; the status is waiting_for_confirmation. control-openhands browser wait-text 'Do you want to continue with this action?' --timeout 15000 succeeds and control-openhands browser screenshot --feature F15.confirmation-mode-effect --name awaiting shows Cancel ⇧⌘⌫ and Continue ⌘↩ under the pending command. Run control-openhands browser click 'testid=action-confirm-button', control-openhands conversation wait <id> --until finished,idle,error --timeout 120 (<id> from the start output) and control-openhands conversation events <id> --kinds ObservationEvent: the terminal observation is qa-f15-confirm. If the status is still waiting_for_confirmation the agent asked again (see Gotchas): confirm again. Restore on /settings/verification (control-openhands browser goto /settings/verification; the browser is on the conversation): browser click 'testid=sdk-section-basic-toggle', click the Confirmation Mode label, browser click 'testid=save-button'; the Basic save also resets Security Analyzer to llm (API confirmation_mode false, security_analyzer llm).

Critic values persistF15.critic-saveMapped behavior

Scenario F15--critic-values-persist · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Enter “qa-critic-model” in “Sdk settings verification.critic model name”.
  2. Do
    Enter “qa-dummy-critic-key” in “Sdk settings verification.critic api key”.
  3. Do
    Choose “Save”.
  4. Do
    Reload the page.

Expect

critic text fields persist after reload; the Critic API Key comes back masked; clearing the fields and turning the critic off restores the defaults.

Exact CLI recipe & bug checks

On the All tier click testid=verification-settings-screen >> text="Enable Critic", run control-openhands browser fill 'testid=sdk-settings-verification.critic_model_name' qa-critic-model, control-openhands browser fill 'testid=sdk-settings-verification.critic_api_key' qa-dummy-critic-key, control-openhands browser click 'testid=save-button' and control-openhands browser reload. The page opens on All; browser value 'testid=sdk-settings-verification.critic_model_name' is qa-critic-model and browser value 'testid=sdk-settings-verification.critic_api_key' is **********. Restore on All: fill both fields with '', click text="Enable Critic" again, save, reload: the page opens on Basic and the API shows critic_enabled false, critic_model_name and critic_api_key null.

Critic mode, threshold and iterationsF15.critic-tuningMapped behavior

Scenario F15--critic-mode-threshold-and-iterations · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Sdk section all toggle”.
  2. Do
    Choose “Verification settings screen” / “"Enable Critic"”.
  3. Do
    Choose “Sdk settings verification.critic mode”.
  4. Do
    Choose the listbox / “All Actions”.
  5. Do
    Choose “Verification settings screen” / “"Enable Iterative Refinement"”.
  6. Do
    Enter “0.75” in “Sdk settings verification.critic threshold”.
  7. Do
    Enter “5” in “Sdk settings verification.max refinement iterations”.
  8. Do
    Choose “Save”.
  9. Do
    Reload the page.
  10. Do
    Enter “1.5” in “Sdk settings verification.critic threshold”.
  11. Do
    Choose “Save”.
  12. Do
    Reload the page.
  13. Do
    Choose “Sdk section basic toggle”.
  14. Do
    Choose “Save”.

Expect

Critic Mode is a dropdown (Finish and Message, All Actions); the mode, Critic Threshold and Max Refinement Iterations persist after reload, and an out-of-range threshold (1.5) is refused with the 422 toast.

Exact CLI recipe & bug checks

On /settings/verification run control-openhands browser click 'testid=sdk-section-all-toggle', control-openhands browser click 'testid=verification-settings-screen >> text="Enable Critic"', control-openhands browser click 'testid=sdk-settings-verification.critic_mode' and control-openhands browser snapshot 'role=listbox' (options Finish and Message [selected], All Actions). Run control-openhands browser click 'role=listbox >> role=option[name="All Actions"]', control-openhands browser click 'testid=verification-settings-screen >> text="Enable Iterative Refinement"', control-openhands browser fill 'testid=sdk-settings-verification.critic_threshold' 0.75, control-openhands browser fill 'testid=sdk-settings-verification.max_refinement_iterations' 5, control-openhands browser click 'testid=save-button' and control-openhands browser reload: the page opens on All, browser value reads All Actions, 0.75 and 5, and control-openhands api GET /api/settings --pick agent_settings.verification shows critic_mode all_actions, critic_threshold 0.75, max_refinement_iterations 5. Then browser fill 'testid=sdk-settings-verification.critic_threshold' 1.5, browser network --clear, browser click 'testid=save-button', browser toasts (HTTP request failed (422 Unprocessable Entity): {"detail":"Settings validation failed"}, three PATCH /api/settings with 422 in browser network); the API still shows 0.75. Restore: browser reload, browser click 'testid=sdk-section-basic-toggle', click text="Enable Iterative Refinement" and text="Enable Critic" off, browser click 'testid=save-button': the Basic save resets mode, threshold and iterations, and the API shows the verification defaults.

Tier follows the stored values after a saveF15.tier-after-saveMapped behavior

Scenario F15--tier-follows-the-stored-values-after-a-save · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/condenser.
  2. Do
    Choose “Sdk section all toggle”.
  3. Do
    Enter “3” in “Sdk settings condenser.keep first”.
  4. Do
    Choose “Save”.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Enter “2” in “Sdk settings condenser.keep first”.
  7. Do
    Choose “Save”.
  8. Wait
    Wait for “Sdk settings condenser.keep first” to disappear.

Expect

after a save the tab bar re-derives the tier from the stored values without a reload: saving the last minor field back to its default moves the page from All to Basic.

Exact CLI recipe & bug checks

On /settings/condenser (control-openhands browser goto /settings/condenser; the previous bullet is on /settings/verification) run control-openhands browser click 'testid=sdk-section-all-toggle', control-openhands browser fill 'testid=sdk-settings-condenser.keep_first' 3, control-openhands browser click 'testid=save-button', control-openhands browser wait-text 'Settings saved' --timeout 5000 and control-openhands browser snapshot 'role=tablist' (All [selected]). Then browser fill 'testid=sdk-settings-condenser.keep_first' 2, browser click 'testid=save-button', control-openhands browser wait 'testid=sdk-settings-condenser.keep_first' --state detached --timeout 5000 (detached) and browser snapshot 'role=tablist': Basic [selected] without any reload; the API shows keep_first 2 (restored).

Memory reaches the agentF15.agent-context-effectMapped behavior

Scenario F15--memory-reaches-the-agent · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/agent-context.
  2. Do
    Start a conversation with the prompt “Reply with only: ok”.
  3. Do
    Open /settings/agent-context.
  4. Do
    Start a conversation as specified in the recipe.

Expect

with Persistent Agent Memory on, a new conversation's system prompt carries the .openhands/memory/ instructions; with it off, it does not.

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/agent-context, turn Persistent Agent Memory on and save as in F15.agent-context-save, then run control-openhands conversation start --prompt "Reply with only: ok" --wait --timeout 180 and control-openhands api GET '/api/conversations/<id>/events/search?limit=1'. The first event is the SystemPromptEvent; its text contains Project memory: followed by `.openhands/memory/` under the workspace root. The same call for a conversation started with memory off (for example the one from F15.confirmation-mode-effect) has no .openhands/memory. Restore memory off through the page (control-openhands browser goto /settings/agent-context first: conversation start left the browser on the conversation).

Leaving discards editsF15.unsaved-navigationMapped behavior

Scenario F15--leaving-discards-edits · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Choose “Verification settings”.
  2. Do
    Choose “Sdk section all toggle”.
  3. Do
    Choose “Condenser settings”.
  4. Do
    Use browser dialogs as specified in the exact recipe.
  5. Do
    Choose “Verification settings”.

Expect

leaving a page with unsaved edits shows no prompt and discards them.

Exact CLI recipe & bug checks

Arrange an unsaved draft: control-openhands browser click 'testid=sidebar-settings-/settings/verification' --expect-url '/settings/verification(\?|$)', control-openhands browser click 'testid=sdk-section-all-toggle', then click testid=verification-settings-screen >> text=Confirmation Mode, >> text="Enable Critic" and >> text="Enable Iterative Refinement" (do not save); browser enabled 'testid=save-button' is true. Run control-openhands browser click 'testid=sidebar-settings-/settings/condenser' --expect-url '/settings/condenser(\?|$)', control-openhands browser dialogs (none), control-openhands browser click 'testid=sidebar-settings-/settings/verification' --expect-url '/settings/verification(\?|$)', then control-openhands browser eval "[document.querySelector('[data-testid=\"sdk-settings-confirmation_mode\"]').checked, document.querySelector('[data-testid=\"sdk-settings-verification.critic_enabled\"]').checked]" ([false, false]) and browser enabled 'testid=save-button' (false).

Command menuF15.command-menuMapped behavior

Scenario F15--command-menu · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /.
  2. Do
    Press Control+k.
  3. Do
    Enter “Condenser” in “Command menu” / the combobox.
  4. Do
    Press Enter.
  5. Wait
    Wait for the element identified in the CLI recipe to appear.
  6. Do
    Press Control+k.
  7. Do
    Enter “memory” in “Command menu” / the combobox.
  8. Do
    Press Escape.

Expect

the command menu offers Condenser settings, Agent Context and Verification settings, and each opens its page.

Exact CLI recipe & bug checks

Run control-openhands browser goto /, control-openhands browser press Control+k, control-openhands browser type 'testid=command-menu >> role=combobox' Condenser, control-openhands browser snapshot 'testid=command-menu' (option Condenser settings Tune context condensation behavior. Go [selected]), control-openhands browser press Enter and control-openhands browser wait-url '/settings/condenser(\?|$)'. Repeat with Verification (option Verification settings Adjust risk and confirmation controls. Go, URL /settings/verification) and with Agent Context (option Agent Context Let the agent keep notes and recall them in new conversations. Go [selected], URL /settings/agent-context, subtitle Let the agent keep notes and recall them in new conversations.). Then control-openhands browser press Control+k, control-openhands browser type 'testid=command-menu >> role=combobox' memory and control-openhands browser snapshot 'testid=command-menu': two options, Condenser settings … and Agent Context … (hovering selects an option, so click the one you want rather than pressing Enter); close with control-openhands browser press Escape (browser count 'testid=command-menu' is 0).

Direct URLsF15.direct-urlMapped behavior

Scenario F15--direct-urls · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/condenser.
  2. Wait
    Wait for “Condenser settings screen” to appear.

Expect

each page renders when opened by URL.

Exact CLI recipe & bug checks

Run control-openhands browser goto /settings/condenser and control-openhands browser wait 'testid=condenser-settings-screen'; likewise /settings/agent-context with testid=agent-context-settings-screen and /settings/verification with testid=verification-settings-screen. Each wait returns visible.

PhoneF15.phoneMapped behavior

Scenario F15--phone · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Use the phone viewport.
  2. Do
    Open /settings/verification.
  3. Do
    Choose “Sdk section all toggle”.
  4. Do
    Choose “Verification settings screen” / “"Enable Critic"”.
  5. Setup
    Use the desktop viewport.
  6. Do
    Reload the page.

Expect

at 390 px the fields stack in one column without horizontal overflow.

Exact CLI recipe & bug checks

Run control-openhands browser viewport phone, control-openhands browser goto /settings/verification, control-openhands browser click 'testid=sdk-section-all-toggle', control-openhands browser click 'testid=verification-settings-screen >> text="Enable Critic"' (unsaved), control-openhands browser bbox 'testid=verification-settings-screen' (width 348, pageHorizontalOverflow false) and control-openhands browser screenshot --feature F15.phone --name verification-all (one column; the settings nav is replaced by a back chevron). Do the same for /settings/condenser All. Return with control-openhands browser viewport desktop and browser reload to drop the draft.

Schema unavailable (`F15.schema-unavailable`, blocked)F15.schema-unavailableBlocked

Scenario F15--schema-unavailable-f15-schema-unavailable-blocked · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Change the run-owned service state: stop agent-server.
  2. Do
    Reload the page.
  3. Do
    Restart the run-owned stack as specified.

Expect

a backend without the settings-schema endpoints shows SDK settings schema unavailable. with an upgrade hint instead of the form (blocked: needs such a backend).

Exact CLI recipe & bug checks

control-openhands service stop agent-server does not reach this state: after browser reload the whole app is replaced by the Manage backends dialog (Disconnected, 502). Bring the stack back with control-openhands restart. The state needs a backend whose schema endpoints 404 (see Preconditions); there browser text 'testid=sdk-schema-unavailable' should start with SDK settings schema unavailable.

Errors sweepF15 · setup / cleanupSupport step

Scenario F15--errors-sweep · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    control-openhands browser errors --clear, open the three pages and their All tabs, then control-openhands browser errors --app-only: no page errors and no warnings (the condenser fields are translated since #18035).

Expect

Complete this support step before continuing to the next scenario.

Exact CLI recipe & bug checks

control-openhands browser errors --clear, open the three pages and their All tabs, then control-openhands browser errors --app-only: no page errors and no warnings (the condenser fields are translated since #18035).

Background refetch keeps the draftF15.save-dirty-stateKnown failure

Scenario F15--background-refetch-keeps-the-draft · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    Open /settings/condenser.
  2. Do
    Choose “Sdk section all toggle”.
  3. Do
    Enter “111” in “Sdk settings condenser.max size”.
  4. Setup
    Arrange the prerequisite backend state through the API.
  5. Do
    Use browser clock as specified in the exact recipe.
  6. Do
    Use browser network state as specified in the exact recipe.
  7. Do
    Use browser network state as specified in the exact recipe.
  8. Do
    Use browser clock as specified in the exact recipe.

Expect

Save is disabled on load, enabled after an edit, and disabled again when the edit is reverted by hand.

Known bug observations

Known failure (reproduced 2026-10-12, #16948): control-openhands browser value 'testid=sdk-settings-condenser.max_size' is 300 and control-openhands browser enabled 'testid=save-button' is false; the draft vanished without Save, reset or a scope change. Capture control-openhands browser screenshot --feature F15.save-dirty-state --name refetch-discards-draft. Restore with control-openhands browser clock --offset-ms 0, fill Max Size 240, click Save and confirm control-openhands api GET /api/settings --pick agent_settings.condenser.max_size is 240; networking stays online.

Exact CLI recipe & bug checks

With Max Size stored as 240, run control-openhands browser goto /settings/condenser, control-openhands browser click 'testid=sdk-section-all-toggle' and control-openhands browser fill 'testid=sdk-settings-condenser.max_size' 111. Arrange a changed server baseline with control-openhands api PATCH /api/settings --write --data '{"agent_settings_diff":{"condenser":{"max_size":300}}}'; this API write is a precondition, not the UI proof. Run control-openhands browser clock --offset-ms 360000 (makes the mounted query stale), control-openhands browser network-state offline, control-openhands browser network-state online and control-openhands browser network --filter '/api/settings'. After the real GET refetch, the unsaved 111 should remain and Save should stay enabled. Known failure (reproduced 2026-10-12, #16948): control-openhands browser value 'testid=sdk-settings-condenser.max_size' is 300 and control-openhands browser enabled 'testid=save-button' is false; the draft vanished without Save, reset or a scope change. Capture control-openhands browser screenshot --feature F15.save-dirty-state --name refetch-discards-draft. Restore with control-openhands browser clock --offset-ms 0, fill Max Size 240, click Save and confirm control-openhands api GET /api/settings --pick agent_settings.condenser.max_size is 240; networking stays online.

Negative condenser input floorF15.save-errorsKnown failure

Scenario F15--negative-condenser-input-floor · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    On /settings/condenser select All, fill testid=sdk-settings-condenser.max_size with -5 and read browser enabled 'testid=save-button'. Expected by #15813: the input prevents values below zero. Known failure, reproduced 2026-10-12: -5 remains and Save is enabled. Save shows Settings validation failed, while api GET /api/settings --pick agent_settings.condenser.max_size remains 240; repeat filling -5 and read Save enabled, then fill 240 to discard the invalid draft. The server guard works; the requested client input floor does not.

Expect

a non-integer Max Size is refused in the browser with a toast; a negative one is refused by the server with a 422 toast; nothing is stored.

Known bug observations

Known failure, reproduced 2026-10-12: -5 remains and Save is enabled. Save shows Settings validation failed, while api GET /api/settings --pick agent_settings.condenser.max_size remains 240; repeat filling -5 and read Save enabled, then fill 240 to discard the invalid draft. The server guard works; the requested client input floor does not.

Exact CLI recipe & bug checks

On /settings/condenser select All, fill testid=sdk-settings-condenser.max_size with -5 and read browser enabled 'testid=save-button'. Expected by #15813: the input prevents values below zero. Known failure, reproduced 2026-10-12: -5 remains and Save is enabled. Save shows Settings validation failed, while api GET /api/settings --pick agent_settings.condenser.max_size remains 240; repeat filling -5 and read Save enabled, then fill 240 to discard the invalid draft. The server guard works; the requested client input floor does not.

Default Verification tabsF15.verification-page · F15.verification-critic-fieldsMapped behavior

Scenario F15--default-verification-tabs · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Do
    #15811 asks whether Basic and Advanced should differ. On defaults, snapshot testid=verification-settings-screen in each tab: both show only Confirmation Mode and Enable Critic. Enable Confirmation Mode without saving; browser count 'testid=sdk-settings-security_analyzer' is 1 on Advanced, 0 on Basic and 1 again on Advanced. Reload to discard, then repeat the default/conditional check. Verified 2026-10-12: the default sameness is the current conditional-field behavior, not evidence that the two tiers always render the same fields.

Expect

Verification shows Basic, Advanced and All tabs; with defaults every tier shows only Confirmation Mode and Enable Critic.

Confirmation Mode reveals Security Analyzer (Advanced); Enable Critic reveals Iterative Refinement and Critic API Key (Basic) plus mode, server URL and model (All); Iterative Refinement reveals threshold and max iterations.

Exact CLI recipe & bug checks

#15811 asks whether Basic and Advanced should differ. On defaults, snapshot testid=verification-settings-screen in each tab: both show only Confirmation Mode and Enable Critic. Enable Confirmation Mode without saving; browser count 'testid=sdk-settings-security_analyzer' is 1 on Advanced, 0 on Basic and 1 again on Advanced. Reload to discard, then repeat the default/conditional check. Verified 2026-10-12: the default sameness is the current conditional-field behavior, not evidence that the two tiers always render the same fields.

Datetime refreshed on Canvas launchF15.agent-context-pageMapped behavior

Scenario F15--datetime-refreshed-on-canvas-launch · Permalink · Source

Given

Use F15’s prerequisites and the common setup. Follow the family’s order when a script uses an earlier fixture or says “as above”.

Do — reading outline

  1. Setup
    Arrange the prerequisite backend state through the API.

Expect

Agent Context shows a single Persistent Agent Memory switch with its help text and no tab bar.

Exact CLI recipe & bug checks

Before a new chat, read api GET /api/settings --pick agent_settings.agent_context.current_datetime; record this public timestamp so it can be restored. Arrange an old persisted-state timestamp through the real API (api PATCH /api/settings --write --data '{"agent_settings_diff":{"agent_context":{"current_datetime":"2001-01-01T00:00:00+00:00"}}}'). Start a chat through the Home composer with the dummy loopback profile, then inspect the run-owned runtime agent.agent_context.current_datetime and conversation events <id> --kinds SystemPromptEvent --grep 2001-01-01 --from-start --last 1. Expected: runtime datetime matches the new launch and the old-date event search returns zero. Restore the original settings timestamp through PATCH; no successful model reply is required to inspect initialization.

Family notes, prerequisites for known issues & limits
  • Switches are sr-only checkboxes (keyboard reachable): toggle by clicking the label text scoped to the page, read .checked with browser eval.
  • text=Enable Critic is a case-insensitive substring match and hits the help paragraph Enable critic evaluation for the agent. first, so the click does nothing. Quote it: text="Enable Critic" (same for text="Enable Iterative Refinement").
  • The conversation-owned fields have no verification. prefix: sdk-settings-confirmation_mode, sdk-settings-security_analyzer. Critic fields keep it (sdk-settings-verification.critic_enabled).
  • Saving is tier-scoped: Basic or Advanced saves reset every field the tier hides to its schema default (F15.view-scoped-save). Make all edits on All when other non-default values must survive.
  • The page opens on the least detailed tier that shows every stored override (any minor override → All). Do not assume Basic after a reload.
  • Condenser Kind None (no_op) still shows the summarizer fields with defaults, although the stored condenser no longer has them, and edits to them are accepted (Settings saved) but dropped while None is stored. #18049 covers hiding fields that do not apply to the selected kind.
  • A server-side validation error is retried three times (about 1.5 s of Saving...) before the toast, and the toast only says Settings validation failed without naming the field (#17932).
  • Condenser Kind is served by the Agent Server's settings schema as a user field. Canvas shows its own choice labels and help text rather than raw llm_summarizing/no_op values or reStructuredText. The dropdown (F15.condenser-kind) remains a user setting so additional kinds can be selected. OpenHands/software-agent-sdk#5495, the schema's developer-facing description, remains an upstream issue.
  • Toasts close after a few seconds; read them right after the click, or assert the stored value with api GET /api/settings.
  • Under confirmation mode the first confirmed command sometimes returned The terminal session was reset because the underlying tmux server/session disappeared... while several runs shared one tmux server (OpenHands/OpenHands#17946); launch now isolates them, so treat a reset as a finding. The agent then asks to run it again; a confirmed command took up to two minutes to produce its observation. Confirm again and wait with conversation wait, not a fixed sleep.
  • The confirmation buttons have test ids (action-confirm-button, action-reject-button) but their accessible names are Confirm action / Reject action, not the visible Continue / Cancel: role=button[name*="Continue"] finds nothing.
  • Basic-tier saves on Verification reset Security Analyzer to llm (it is an Advanced field). That is a convenient restore, and a trap if you meant to keep None.
  • The memory instructions are only visible through the conversation's first event; conversation events truncates text, so read api GET /api/conversations/<id>/events/search?limit=1 and search the JSON.
  • A checkout without #18035 shows the condenser minor fields with the schema's English text (Max tokens, Keep first, raw choice values) and logs 14 [i18n] Missing translation warnings on the All tab.
  • The dropdowns' clear (×) button has no accessible name and, in this build, clicking it through role=button >> nth=2 left the value unchanged.
  • After every successful save the selected tab is recomputed from the stored values (F15.tier-after-save), so a follow-up fill on an All-only field can time out because the page silently moved to Basic. Re-click sdk-section-all-toggle before each further edit.
  • Pressing Enter in a field does not save (no toast, API unchanged, Save stays enabled); only the Save Changes button submits.
  • Fields revealed by a parent switch do not exist while the switch is off: fill on critic_threshold with Iterative Refinement off waits for a missing element.
  • Rejecting a pending action (Cancel), the lock chip above the composer and the ⌘↩ / ⇧⌘⌫ shortcuts belong to F06.confirmation-mode and F06.confirmation-shortcuts; this family only proves that the settings turn the prompt on.
  • browser count 'role=status' also counts the backend status dot (backend-status-dot); use browser toasts for toast texts.

Before you start

Use an OpenHands checkout at the source revision, with Node 24+, installed npm dependencies, uv / uvx, and a supported Chromium browser. The verification skill is the complete setup contract.

export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH"
export OH_VERIFY_RUN=$(control-openhands launch --new --print-run)
control-openhands doctor
control-openhands onboard --skip
# Configure a model only for a scenario that needs one:
control-openhands llm preset deepseek
# When the run is complete:
control-openhands stop

F01 walks onboarding instead of skipping it. No-model checks run before model configuration. Keep OH_VERIFY_RUN set so commands address your own run. Desktop is 1440 × 1000; phone is 390 × 844. Model-backed checks need the configured provider key and budget.

Recipes run in source order within each family and can share fixtures. The page preserves that order; “as above” is a dependency on an earlier case. Setup commands and API writes arrange state. They do not prove a UI path. Follow the family cleanup steps.

A known failure preserves the intended expectation beside the observed defect. Executable known-bug blocks use xfail for a reproduced symptom and xpass when it stops reproducing. Cloud accounts, enterprise capabilities and other unavailable prerequisites remain explicit. “Not run” is different from “blocked”.

The maintained map describes application behavior at its pinned revision. Once a fix is verified in the application version under test, remove the obsolete failure note and keep the normal expectation. Dated evidence belongs in the backfill ledger.

Read the full map index and coverage boundaries, the CLI guide, or the earlier illustrated edition. The verification approach adapts Lauren Tan’s (@poteto) pstack verification-skill generators; attribution and adaptation.