Short version: the client can send a tool name, description, JSON Schema parameters, and optional annotations in client_tools. The SDK turns that spec into a normal tool visible to the LLM, emits an ActionEvent when the tool is called, immediately returns an acknowledgement observation, and leaves the real side effect to the client listening on the event stream.
Before this PR, frontend-owned tools needed Python classes on the server: a module defining Action, Observation, and ToolExecutor had to live next to the JavaScript client code, then be imported by the agent server through tool_module_qualnames or --import-modules. That is a lot of server plumbing for UI commands such as opening a file viewer panel.
The new path adds ClientToolSpec and ClientTool in openhands.sdk.tool.client_tool, exposes client_tools on conversation creation, and wires both local and remote conversations so the same JSON spec works whether the SDK runs in-process or through the agent server.
| Piece | Role | Important detail |
|---|---|---|
ClientToolSpec | Request-time JSON model | Requires an object JSON Schema in parameters; annotations are optional and are not assumed read-only by default. |
Action.from_mcp_schema() | Dynamic action class builder | Builds ClientAction_<name> from the JSON Schema so the normal event and LLM-tool paths can be reused. |
ClientTool | ToolDefinition implementation | Has a per-instance name, preserves the original JSON Schema when exporting tool schemas, and uses one shared stateless executor. |
register_client_tools() | Registry bridge | Registers the ClientTool resolver class globally by tool name, but stores each conversation’s full spec in Tool.params["spec"]. |
ClientToolExecutor | No-op server executor | Returns ClientToolObservation with “Tool call dispatched to client.” The server never performs the UI action. |
A client-defined tool is not a separate kind of agent capability once registration finishes. It becomes a regular SDK tool spec injected into agent.tools, so agent initialization resolves it using the existing tool registry.
flowchart TD
C["Frontend / Agent Canvas"] -->|"POST /api/conversations with client_tools"| R["StartConversationRequest"]
R --> V["ClientToolSpec validation: object JSON Schema"]
V --> REG["register_client_tools(specs)"]
REG --> CACHE{"Action type cache: name + schema"}
CACHE -->|"new stable pair"| ACTION["Action.from_mcp_schema -> ClientAction_name"]
CACHE -->|"same name + same schema"| REUSE["Reuse dynamic Action class"]
CACHE -->|"same name + different schema"| ERR["ClientToolSchemaConflictError -> HTTP 400"]
ACTION --> GREG["Global registry: tool name -> ClientTool class"]
REUSE --> GREG
GREG --> TS["Tool(name, params.spec): schema stays per conversation"]
TS --> AG["Inject into agent.tools"]
AG --> INIT["Agent initialization resolves Tool -> ClientTool instance"]
INIT --> LLM["LLM sees a normal tool schema"]
Key boundary: the global registry stores only the resolver class for a tool name. The actual client schema stays with the conversation through Tool.params["spec"], which is why persistence, forks, and resume can recover the exact tool definition.
When the model calls the tool, the SDK emits the same event shape it would emit for any other tool. The only special behavior is execution: the server-side executor is deliberately an acknowledgement, not the real implementation.
sequenceDiagram
participant UI as Client UI
participant RC as RemoteConversation
participant AS as Agent Server
participant LC as LocalConversation
participant A as Agent
participant L as LLM
UI->>RC: Conversation(..., client_tools=[spec])
RC->>AS: POST /api/conversations
AS->>LC: start with registered ClientTool
A->>L: completion request with tool schemas
L-->>A: tool call open_file_viewer(args)
A->>LC: emit ActionEvent(ClientAction_open_file_viewer)
LC->>AS: EventService records and broadcasts action
AS-->>UI: WebSocket ActionEvent(tool name + args)
A->>LC: run ClientToolExecutor
LC-->>A: ClientToolObservation acknowledgement
LC->>AS: EventService records and broadcasts observation
AS-->>UI: WebSocket ObservationEvent(ack)
UI->>UI: perform client-side effect
The important product behavior follows from that sequence: these tools are fire-and-forget. The agent does not block waiting for the browser to return a result. If a future UI tool needs a real client-provided result, that is a different protocol with pause, timeout, and result-submission semantics.
Conversation(..., client_tools=[...]) passes specs into LocalConversation. It registers the tools, appends missing Tool(name=...) entries to the immutable agent via model_copy(update=...), and then initializes normally.
RemoteConversation serializes client_tools into the POST /api/conversations payload. It also registers the dynamic action types locally, so WebSocket and persisted events with ClientAction_* can deserialize on the client side.
ConversationService calls register_client_tools(), injects returned tool specs into the request agent, and persists the original specs because StoredConversation extends StartConversationRequest.
Forking copies stored metadata so a fork keeps its client_tools. On restart, the server reads meta.json and re-registers the specs before event services come back online.
flowchart LR
START["Conversation start"] --> META["meta.json: StoredConversation.client_tools"]
START --> BASE["BASE_STATE: agent.tools params.spec"]
META --> SERVER["Server restart via ConversationService.__aenter__"]
SERVER --> REREG["register_client_tools(stored.client_tools)"]
BASE --> LOCAL["Local resume without supplied client_tools"]
LOCAL --> RECOVER["extract_client_tool_specs(agent.tools)"]
RECOVER --> REREG
META --> REMOTE["Remote reattach by conversation_id"]
REMOTE --> INFO["GET conversation info returns client_tools"]
INFO --> CLIENTREG["ClientTool.from_spec before event sync"]
CLIENTREG --> DESER["Persisted ClientAction_* events deserialize"]
REREG --> RUN["Conversation resumes with tool available"]
Action.from_mcp_schema() creates concrete Pydantic action subclasses. Their kind values are process-global because event deserialization uses the action hierarchy to resolve incoming kind strings. Creating two different classes named ClientAction_open_file_viewer would make Action.resolve_kind ambiguous.
The PR therefore caches generated action types by tool name and deep-copied schema. Re-registering the same name with the same schema is safe and returns the existing class. Reusing a name for a different schema raises ClientToolSchemaConflictError, which the agent-server router maps to a client error instead of letting it become a confusing internal deserialization failure.
| Concern | Implemented design |
|---|---|
| JSON Schema fidelity | ClientTool._get_tool_schema() starts from the original input_schema so enums, nested objects, numeric bounds, and additionalProperties survive provider export. |
| SDK meta fields | The tool overlays SDK-added summary and, when required, security_risk fields onto the original schema rather than rebuilding the whole schema from the generated action model. |
| Annotations | Annotations are optional. If omitted, the SDK does not optimistically mark the tool read-only; clients can explicitly provide MCP-style annotations when appropriate. |
| Server execution | There is no server-side implementation for the real effect. The executor only acknowledges dispatch, so the frontend owns any UI mutation, navigation, or side effect. |
| Backward compatibility | client_tools defaults to an empty list. Existing Python-backed tools and tool_module_qualnames keep working. |
The frontend needs two pieces: send the spec at conversation creation, then listen for matching action events.
{
"client_tools": [
{
"name": "open_file_viewer",
"description": "Open the file viewer panel to display a file",
"parameters": {
"type": "object",
"properties": {
"file_path": {
"type": "string",
"description": "Path to open"
}
},
"required": ["file_path"]
}
}
]
}
if (event.kind === "ActionEvent"
&& event.action.kind === "ClientAction_open_file_viewer") {
openFileViewer(event.action.file_path);
}
Not in this PR: client-to-SDK tool result submission, waiting for the client before the agent continues, result timeouts, retries, or a deprecation of Python-backed custom tools. This PR is intentionally the no-op/acknowledgement layer for client-handled actions.
The implementation keeps the SDK’s usual boundaries: agents remain immutable Pydantic models, conversation state is the mutable source of truth, and dynamic tool behavior is still described declaratively. The only runtime side effect is registering a resolver so the existing tool system can turn a persisted Tool(name, params) back into a ClientTool.
That means Agent Canvas can remove Python-only UI tool modules for fire-and-forget actions, while server-executed tools still use the existing custom tool path. The split is clean: if the browser can handle it from an ActionEvent, use client_tools; if the server must compute a result, ship a real Python tool.