Keep the OpenHands agent-server upstream-shaped. Put durable channel intake and delivery in a small local coordinator that borrows Automation’s table-as-queue mechanics and Hermes’s per-session lane policy.
Adopt a local SQLite-backed Message Work Coordinator outside agent-server.
Channel adapters durably accept input through it and deliver output through it. The coordinator appends messages to agent-server, requests execution, projects deliverable agent events into delivery work, and owns claims, retries, ordering, reconciliation and audit history.
Agent-server remains the source of truth for conversations: durable events, execution, state and event streaming. The coordinator remains the source of truth for external work: whether a platform message has been integrated and whether an agent event has been delivered.
Honest guarantee: intake can become effectively-once after agent-server accepts a deterministic event identity. External delivery is effectively-once only when the platform supports idempotency or reconciliation; an ambiguous send must become delivery_unknown, never a blind retry.
Fig 1 · Conversation truth stays in agent-server; external-work truth stays in the coordinator.
Edge availability: Cloudflare Queue is a spool, not the coordinator
Use Cloudflare Queue in front of the local coordinator for HTTP-reachable ingress.
The public Worker verifies the webhook and enqueues a normalized envelope. A local HTTP-pull adapter consumes it and calls the coordinator; no inbound tunnel is required. GitHub, email and future webhook bridges use this path. Socket-only clients still need a connected adapter somewhere and are not made durable merely by creating a queue.
The Worker authenticates the platform request and writes one envelope with a stable platform source identity.
The local pull adapter computes the canonical lane and calls resolveLane plus acceptIntake.
Only after the SQLite intake transaction commits does the adapter acknowledge the Cloudflare message. If acknowledgment is lost, redelivery hits the coordinator’s unique source key and returns the existing work row.
The coordinator—not Cloudflare Queue—owns lane order, claims, retries after local acceptance, projection, delivery outcomes and audit history.
Operational consequence: retire the Quick Tunnel from webhook delivery. A named tunnel may still serve interactive access, but durable ingress no longer depends on cloudflared staying connected. Cloudflare’s published Workers Free allowance is “10,000 operations/day included” with 24-hour retention; paid retention is configurable up to 14 days. See Cloudflare Queues pricing and HTTP pull consumers.
This is intentionally two durable layers with different facts: Cloudflare Queue records that an edge message has not yet reached the host; the SQLite coordinator records that accepted external work has or has not been integrated and delivered. Neither store impersonates the other.
2One fact, one owner
Fact
Owner
Why
Conversation messages and agent actions
agent-server EventLog
They are conversation history and already survive restart through the SDK’s durable EventLog.
Whether a conversation is executing
agent-server
The coordinator must observe execution state; it must not copy or invent another execution state machine.
External source deduplication
Message Work Coordinator
Only the channel side knows the stable platform message identity.
Canonical chat/thread lane computation
Channel adapter
The adapter knows platform identity rules, thread semantics and aliases such as WhatsApp JID/LID.
Lane-to-conversation correspondence
Message Work Coordinator
The persisted mapping makes routing idempotent and leaves a queryable debugging trail from platform address to agent-server conversation.
Per-chat/thread ordering
Message Work Coordinator
Ordering is a channel delivery concern expressed by the persisted lane_key.
Retry timing and claim recovery
Message Work Coordinator
These are durable-work mechanics borrowed from Automation, independent of agent execution.
Whether one agent event reached one destination
Message Work Coordinator
Agent-server cannot know whether Slack, Discord or WhatsApp accepted a send.
Platform connection, formatting and reconciliation
Channel adapter
The adapter knows platform capabilities; it does not own durable scheduling or the lane directory.
The deletion test is useful here: deleting the coordinator would force deduplication, claims, retries, crash recovery and ordering back into every adapter. That is enough complexity behind one small interface to earn a module.
3Compare the three real systems
source of conversation truth
Agent-server
Appends durable events, starts execution without blocking the request, publishes new events, and persists conversation metadata.
Keep:/events, /run, EventLog, event search/subscription, one owner per conversation.
Do not add: platform claims, retries or delivery state.
source of durable-work mechanics
Automation
automation_runs doubles as a queue: oldest PENDING rows are claimed transactionally, marked RUNNING, launched asynchronously and reconciled by callback/watchdog.
Keep: row-as-work, transactional claim, explicit terminal states, deadline and reconciliation.
Add locally: attempts, backoff and claim deadlines—Automation does not currently provide them.
source of lane policy
Hermes gateway
Builds a deterministic session key, permits one active handler per key, and carefully hands pending input to the next handler without parallel execution.
Keep: deterministic lanes, one active owner, stale-owner recovery, ordered handoff and explicit control handling.
Reject: in-memory pending storage and message merging. Agent-server already accepts each message asynchronously and preserves its event boundary.
Question
Agent-server
Automation
Hermes
Decision
What is durable?
Conversation events
Work row + lifecycle
Session transcript; pending map is memory
Use both durable stores, each for its own fact.
How is work claimed?
Conversation lease + one run promise
DB row lock; SQLite assumes one process
In-memory active-session guard
SQLite compare-and-set claim with expiry.
How is concurrency scoped?
Conversation
Run row
Deterministic session key
One ordered lane per destination conversation.
How does completion arrive?
Durable events + state publication
Callback; watchdog verifies timeout
Background task completion
Project agent events; reconcile expired claims.
What is missing?
External dedup/delivery knowledge
Attempts, retry backoff, platform ordering
Crash-safe pending work
The coordinator fills exactly these gaps.
What the Hermes lessons mean here
Hermes mechanism
What it actually protects
Our landing
Deterministic session key
Every message for the same chat or thread reaches the same concurrency domain.
The channel adapter computes a canonical lane_key, including platform-specific alias normalization. The coordinator persists its correspondence with conversation_id.
One active handler per key
Two agent executions cannot mutate one conversation concurrently.
One claimed intake item per lane; agent-server remains the authority on whether conversation execution is active.
Owner-task identity
An older task’s cleanup cannot clear a newer task’s guard.
Fence every claim with a generation; stale workers cannot settle a claim that has been reassigned.
Stale-guard self-heal
If the guard says “busy” but its task is gone, the next message repairs the split state.
Expired claims are reconciled back to ready or delivery_unknown, depending on whether an external effect may have occurred.
Ordered handoff
The active handler keeps ownership until cleanup finishes, then starts the next pending message.
Only the unresolved lane head is claimable; later messages wait, but each remains a separate durable agent event.
Direct control handling
Stop/reset/approval-like input can reach a blocked execution instead of waiting behind ordinary work.
Control operations use agent-server’s explicit pause/interrupt/control interfaces; they are not hidden inside ordinary intake ordering.
In-memory pending merge
Hermes compresses bursts while one handler is active because its adapter owns scheduling.
Do not borrow. REST and WebSocket already append asynchronous messages separately and request a rerun; preserving IDs, media, senders and reply anchors is more valuable.
Agent-server already schedules the conversation: each REST or WebSocket message is appended immediately. If execution is active, agent-server records a rerun request and processes the newly appended events afterward; the coordinator only guarantees durable acceptance and lane order.
Correction from the first sketch: Automation supplies lifecycle, atomic claim and watchdog patterns. Attempt counts, available_at backoff and claim expiry are requirements for the local coordinator, not features already present in Automation.
4Two work kinds, one durable mechanism
A single agent execution can consume several incoming messages and produce several deliverable events. One end-to-end row would force unrelated failure domains together, so the store has two work kinds behind the same claim/retry mechanism.
Fig 2 · Shared mechanics, separate success conditions. Agent execution state is not duplicated here.
Canonical record
Field
Meaning
Invariant
id, kind
Work identity; intake or delivery
Immutable.
source_key
Stable platform input identity, or agent-event/destination identity
Unique within kind; duplicate accepts return the existing row.
lane_key, sequence
Ordering domain and monotonic position
No later unresolved item overtakes an earlier item in the same lane.
conversation_id, agent_event_id
Join to agent-server truth
Intake uses a deterministic event ID; delivery references the originating event.
state, available_at
Work lifecycle and retry schedule
State changes by compare-and-set.
claim_owner, claim_until
Short ownership of one attempt
Expired claims are recoverable; a stale owner cannot complete a newer claim.
attempts, last_error
Retry evidence
Every failed attempt leaves an audit fact.
external_message_id
Platform acknowledgment for delivery
Set before done when the platform returns one.
payload_json
Normalized input or immutable delivery intent
No mutable process objects or credentials.
Adapter-computed lane key
Borrow Hermes’s deterministic session-key idea, but let each channel adapter compute the canonical key because it owns that platform’s chat, thread and alias semantics:
DMs isolate by stable chat identity; a Slack or Discord thread gets its own lane. Platform aliases—especially WhatsApp JID/LID forms—are canonicalized by the adapter before it returns the key.
Persisted lane directory
The coordinator resolves the adapter’s key to an agent-server conversation and saves the correspondence even though the key is deterministic. That row is both routing state and debugging evidence:
lane_key TEXT PRIMARY KEY
conversation_id TEXT NOT NULL
platform TEXT NOT NULL
account_id TEXT
chat_id TEXT NOT NULL
thread_id TEXT
display_name TEXT
created_at TEXT NOT NULL
last_seen_at TEXT NOT NULL
Repeated resolution of the same lane returns the same conversation_id. Operators can query either side to explain which platform chat created a conversation, why messages split, or whether alias normalization produced an unexpected lane; no version field is introduced yet.
Small coordinator interface
adapter.computeLane(input) → lane descriptor
resolveLane(lane) → lane binding + conversation_id
acceptIntake(binding, input) → existing-or-new work
projectDeliveries(conversation) → number created
claimReady(worker, now) → claimed work | null
settle(claim, outcome) → done | retry_wait | delivery_unknown | failed
reconcile(now) → expired/reconciled count
The storage adapter hides the lane directory, SQLite transactions, lane-head selection, backoff and stale-claim fencing. Channel adapters compute canonical lane identity, normalize input, deliver output and optionally reconcile external effects; they do not persist the mapping themselves.
5Worked trace: two messages arrive during execution
Resolve laneThe WhatsApp adapter canonicalizes the chat address and computes channel:whatsapp:…:chat-7:root. The coordinator finds or creates its persisted conversation_id correspondence and updates last_seen_at.
Accept AMessage A becomes intake sequence 41 in that lane. The unique source key makes a webhook/poll replay a no-op.
Integrate AThe worker appends deterministic agent event uuid5(A) with run=true. Agent-server persists it, starts execution asynchronously, and intake A becomes done.
Accept B + CTwo more messages enter as sequence 42 and 43 while agent-server is active. Both are durable immediately; B is lane head, so C cannot overtake it.
Integrate B, then CThe coordinator appends in lane order. Agent-server’s active execution path records a rerun request rather than starting parallel work.
Project outputThe event projector catches up from its durable cursor, maps each deliverable agent event to one delivery row, then subscribes for live events. A unique event/destination source key makes replay harmless.
DeliverThe channel adapter claims delivery work in lane order. A confirmed platform message ID completes it; an ambiguous effect becomes delivery_unknown and blocks later delivery until reconciled or explicitly resolved.
What this permits: incoming messages can be durably accepted while agent-server is busy. Ordering is established before any HTTP call, while agent-server remains responsible for how those messages affect the next execution.
What this forbids: holding accepted messages only in adapter memory, deriving delivery from “latest response” after a restart, or retrying an external send whose outcome is unknown.
6Crash matrix
Crash or race
Durable fact before crash
Recovery
Result
Duplicate platform input
Existing source_key
acceptIntake returns the same row.
No duplicate intake.
Two first messages resolve one new lane
Unique lane_key
One lane-directory insert wins; both callers use the persisted conversation_id.
One routing correspondence.
Crash after lane mapping, before conversation creation
Lane row with reserved conversation_id
Next resolution ensures that exact conversation exists before accepting intake.
Mapping remains repairable.
After accept, before claim
ready row
Next worker claims it.
No loss.
After claim, before agent-server call
Claim deadline + generation
Expiry returns work to ready; stale worker is fenced.
Safe retry.
Agent event appended; HTTP response lost
Deterministic agent_event_id
Retry same append; agent-server returns existing event.
Requires idempotent append extension.
Event appended; execution request not observed
Event exists in EventLog
Retry /run; active response means already executing, idle response schedules it.
Replay hits uniqueness constraint, then advances cursor.
No duplicate row.
Delivery worker dies before send
Claimed row, no external ID
Claim expires and retries.
No external effect yet.
Platform rejects before accepting
Retryable error
Backoff through retry_wait; preserve lane order.
Bounded retry.
Platform accepts; acknowledgment persists
external_message_id
Set done; replay sees completion.
Effectively-once.
Platform may accept; response lost
No proof either way
Mark delivery_unknown; adapter reconciles when possible, otherwise operator chooses confirm/retry/skip.
Exactly-once is impossible without platform support.
Repeated poison delivery
Attempts + errors
Move to failed after policy limit; lane remains blocked until explicit skip or repair.
No silent overtaking.
Two coordinator processes claim together
SQLite row state + generation
BEGIN IMMEDIATE / compare-and-set update allows one winner.
One active claim.
Machine restarts with queued Hermes-style follow-up
Coordinator row, not an in-memory pending map
Claim after restart.
Hermes UX, durable implementation.
The matrix defines the quality bar. Rows marked “requires” or “impossible without platform support” stay visible; architecture prose must not upgrade them into promises.
7ADR: considered options
Queue inside agent-serverrejectedIt couples platform delivery policy to conversation execution and creates a permanent divergence from the OpenHands contract.
Run Automation unchangedrejectedIts durable mechanics are useful, but automation definitions, tarballs, sandboxes, user/org ownership and cloud callbacks solve a different problem.
Hermes pending mapborrow policyExcellent per-lane behavior and race handling; unsuitable as the durable store because the pending slot and active guards live in process memory.
Retry logic in every adapterrejectedIt repeats crash handling, ordering and reconciliation across Slack, Discord, WhatsApp and future channels.
Local coordinatorchosenA small interface around SQLite localizes durable work while letting agent-server and channel adapters keep their natural responsibilities.
Consequences
The coordinator is a new local module and process responsibility, but not a new network service requirement; it can run in the SmolPaws host process initially.
Every adapter migrates to the same accept/claim/settle interface. Platform code stops polling agent execution directly.
SQLite is the first storage adapter. PostgreSQL may implement the same interface later with FOR UPDATE SKIP LOCKED, matching Automation’s multi-worker shape.
The current agent-server needs one narrow compatibility extension for deterministic message append. Without it, intake remains vulnerable to the append-response-loss window.
Delivery uncertainty becomes explicit product state. This is less comforting than “exactly once,” and more truthful.
8One required agent-server contract delta
The current message request contains role, content and run; agent-server creates a fresh event identity internally and returns only success. A coordinator that crashes after a successful append cannot safely distinguish “append happened” from “append never happened.”
Add idempotent message append without adding queue semantics.
Accept a caller-supplied deterministic event_id or idempotency key, persist the event once, and return the event identity. Repeating the same request returns the existing event; reusing the key with different content is a conflict.
This is a compatible reliability extension at the existing event seam. It does not add external work states, claims, delivery acknowledgments or platform concepts to agent-server.
Projection rule: delivery work is derived from durable terminal-response facts, initially successful finishObservationEvent payloads—the same durable facts agent-server already uses for agent_final_response. Explicit outbound-intent events may be introduced later; the coordinator’s event/destination uniqueness key keeps that evolution behind the projector interface.
9Implementation order
Specify the store through tests. Concurrent lane resolution, persisted lane-to-conversation lookup, duplicate accept, per-lane order, claim fencing, expiry, backoff, projector replay and delivery_unknown outcomes all get deterministic SQLite tests.
Add idempotent event append to agent-server. Prove append-response loss by retrying the same event identity across restart.
Implement intake only. Run a shadow path that accepts and appends one non-critical channel while current delivery remains unchanged.
Add event projection and delivery work. Start with terminal agent responses and a recording adapter; prove projector crash cases before real sends.
Canary WhatsApp. It exercises media, aliases, restart recovery and the most visible duplicate/loss cases. Preserve a rollback switch.
Migrate Discord, Slack, GitHub and email. They become platform adapters around the same coordinator interface rather than new orchestration implementations.
Remove superseded orchestration paths. Only after a soak period shows no unresolved delivery_unknown states, ordering violations or duplicate intake.
No estimates are attached. Each step stops on an executable invariant, not on elapsed time.
10Source map and validation
Claim
Grounding
Agent-server appends messages, schedules execution and requests a rerun when active.
Hermes queues/merges follow-ups while a lane is active and drains them without overlap; the merge is documented for comparison and explicitly not adopted.
Validation performed: code paths and focused tests were read at the pinned commits above. The new design is an ADR, so its crash guarantees remain proposed until the SQLite store and idempotent append tests execute them.
Vocabulary used by this ADR
Agent event: a durable event in a conversation’s EventLog. Intake work: one external input waiting to be integrated into agent-server. Delivery work: one immutable intent to deliver an agent event to one external destination.
Lane: the ordering domain for a chat or thread. Claim: short, fenced ownership of one work attempt. Delivery outcome unknown: an external effect may have happened but cannot yet be proved.