SmolPaws · ADR · message coordination

Durable message work belongs around agent-server

Keep the OpenHands agent-server upstream-shaped. Put durable channel intake and delivery in a small local coordinator that borrows Automation’s table-as-queue mechanics and Hermes’s per-session lane policy.
status: proposed agent-server dcd1bed automation 1ddde99 Hermes 4e6d05c 2026-07-13

1The decision

Adopt a local SQLite-backed Message Work Coordinator outside agent-server. Channel adapters durably accept input through it and deliver output through it. The coordinator appends messages to agent-server, requests execution, projects deliverable agent events into delivery work, and owns claims, retries, ordering, reconciliation and audit history.

Agent-server remains the source of truth for conversations: durable events, execution, state and event streaming. The coordinator remains the source of truth for external work: whether a platform message has been integrated and whether an agent event has been delivered.

Honest guarantee: intake can become effectively-once after agent-server accepts a deterministic event identity. External delivery is effectively-once only when the platform supports idempotency or reconciliation; an ambiguous send must become delivery_unknown, never a blind retry.
Chosen message-work architecture Platform adapters accept input into a durable coordinator. The coordinator appends events and requests execution from agent-server. A projector reads agent events and creates durable delivery work claimed by platform adapters. Channel adaptersnormalize input · send outputSlack · Discord · WhatsApp Message Work Coordinatorsmall interface · durable SQLite implementation Intake worksource identity · lane · append checkpoint Delivery workagent event · destination · send outcome OpenHands agent-serverEventLog · /events · /runevent search + WebSocketpackages/openhands-agent-server Agent event streamdurable catch-up + live subscription accept() append + run project claim + deliver No channel queue is added to agent-server. No platform adapter owns retry loops.
Fig 1 · Conversation truth stays in agent-server; external-work truth stays in the coordinator.

Edge availability: Cloudflare Queue is a spool, not the coordinator

Use Cloudflare Queue in front of the local coordinator for HTTP-reachable ingress. The public Worker verifies the webhook and enqueues a normalized envelope. A local HTTP-pull adapter consumes it and calls the coordinator; no inbound tunnel is required. GitHub, email and future webhook bridges use this path. Socket-only clients still need a connected adapter somewhere and are not made durable merely by creating a queue.
  1. The Worker authenticates the platform request and writes one envelope with a stable platform source identity.
  2. The local pull adapter computes the canonical lane and calls resolveLane plus acceptIntake.
  3. Only after the SQLite intake transaction commits does the adapter acknowledge the Cloudflare message. If acknowledgment is lost, redelivery hits the coordinator’s unique source key and returns the existing work row.
  4. The coordinator—not Cloudflare Queue—owns lane order, claims, retries after local acceptance, projection, delivery outcomes and audit history.
Operational consequence: retire the Quick Tunnel from webhook delivery. A named tunnel may still serve interactive access, but durable ingress no longer depends on cloudflared staying connected. Cloudflare’s published Workers Free allowance is “10,000 operations/day included” with 24-hour retention; paid retention is configurable up to 14 days. See Cloudflare Queues pricing and HTTP pull consumers.

This is intentionally two durable layers with different facts: Cloudflare Queue records that an edge message has not yet reached the host; the SQLite coordinator records that accepted external work has or has not been integrated and delivered. Neither store impersonates the other.

2One fact, one owner

FactOwnerWhy
Conversation messages and agent actionsagent-server EventLogThey are conversation history and already survive restart through the SDK’s durable EventLog.
Whether a conversation is executingagent-serverThe coordinator must observe execution state; it must not copy or invent another execution state machine.
External source deduplicationMessage Work CoordinatorOnly the channel side knows the stable platform message identity.
Canonical chat/thread lane computationChannel adapterThe adapter knows platform identity rules, thread semantics and aliases such as WhatsApp JID/LID.
Lane-to-conversation correspondenceMessage Work CoordinatorThe persisted mapping makes routing idempotent and leaves a queryable debugging trail from platform address to agent-server conversation.
Per-chat/thread orderingMessage Work CoordinatorOrdering is a channel delivery concern expressed by the persisted lane_key.
Retry timing and claim recoveryMessage Work CoordinatorThese are durable-work mechanics borrowed from Automation, independent of agent execution.
Whether one agent event reached one destinationMessage Work CoordinatorAgent-server cannot know whether Slack, Discord or WhatsApp accepted a send.
Platform connection, formatting and reconciliationChannel adapterThe adapter knows platform capabilities; it does not own durable scheduling or the lane directory.

The deletion test is useful here: deleting the coordinator would force deduplication, claims, retries, crash recovery and ordering back into every adapter. That is enough complexity behind one small interface to earn a module.

3Compare the three real systems

source of conversation truth

Agent-server

Appends durable events, starts execution without blocking the request, publishes new events, and persists conversation metadata.

Keep: /events, /run, EventLog, event search/subscription, one owner per conversation.

Do not add: platform claims, retries or delivery state.

source of durable-work mechanics

Automation

automation_runs doubles as a queue: oldest PENDING rows are claimed transactionally, marked RUNNING, launched asynchronously and reconciled by callback/watchdog.

Keep: row-as-work, transactional claim, explicit terminal states, deadline and reconciliation.

Add locally: attempts, backoff and claim deadlines—Automation does not currently provide them.

source of lane policy

Hermes gateway

Builds a deterministic session key, permits one active handler per key, and carefully hands pending input to the next handler without parallel execution.

Keep: deterministic lanes, one active owner, stale-owner recovery, ordered handoff and explicit control handling.

Reject: in-memory pending storage and message merging. Agent-server already accepts each message asynchronously and preserves its event boundary.

QuestionAgent-serverAutomationHermesDecision
What is durable?Conversation eventsWork row + lifecycleSession transcript; pending map is memoryUse both durable stores, each for its own fact.
How is work claimed?Conversation lease + one run promiseDB row lock; SQLite assumes one processIn-memory active-session guardSQLite compare-and-set claim with expiry.
How is concurrency scoped?ConversationRun rowDeterministic session keyOne ordered lane per destination conversation.
How does completion arrive?Durable events + state publicationCallback; watchdog verifies timeoutBackground task completionProject agent events; reconcile expired claims.
What is missing?External dedup/delivery knowledgeAttempts, retry backoff, platform orderingCrash-safe pending workThe coordinator fills exactly these gaps.

What the Hermes lessons mean here

Hermes mechanismWhat it actually protectsOur landing
Deterministic session keyEvery message for the same chat or thread reaches the same concurrency domain.The channel adapter computes a canonical lane_key, including platform-specific alias normalization. The coordinator persists its correspondence with conversation_id.
One active handler per keyTwo agent executions cannot mutate one conversation concurrently.One claimed intake item per lane; agent-server remains the authority on whether conversation execution is active.
Owner-task identityAn older task’s cleanup cannot clear a newer task’s guard.Fence every claim with a generation; stale workers cannot settle a claim that has been reassigned.
Stale-guard self-healIf the guard says “busy” but its task is gone, the next message repairs the split state.Expired claims are reconciled back to ready or delivery_unknown, depending on whether an external effect may have occurred.
Ordered handoffThe active handler keeps ownership until cleanup finishes, then starts the next pending message.Only the unresolved lane head is claimable; later messages wait, but each remains a separate durable agent event.
Direct control handlingStop/reset/approval-like input can reach a blocked execution instead of waiting behind ordinary work.Control operations use agent-server’s explicit pause/interrupt/control interfaces; they are not hidden inside ordinary intake ordering.
In-memory pending mergeHermes compresses bursts while one handler is active because its adapter owns scheduling.Do not borrow. REST and WebSocket already append asynchronous messages separately and request a rerun; preserving IDs, media, senders and reply anchors is more valuable.
Agent-server already schedules the conversation: each REST or WebSocket message is appended immediately. If execution is active, agent-server records a rerun request and processes the newly appended events afterward; the coordinator only guarantees durable acceptance and lane order.
Correction from the first sketch: Automation supplies lifecycle, atomic claim and watchdog patterns. Attempt counts, available_at backoff and claim expiry are requirements for the local coordinator, not features already present in Automation.

4Two work kinds, one durable mechanism

A single agent execution can consume several incoming messages and produce several deliverable events. One end-to-end row would force unrelated failure domains together, so the store has two work kinds behind the same claim/retry mechanism.

Intake and delivery state machines Both work kinds move from ready to claimed and then done. Failed claims can wait for retry. Delivery alone can enter delivery_unknown when an external send may have succeeded. Intake workDone means the deterministic user event exists and execution was requested. ready claimed done retry_wait failed claimcheckpointretryableexhausted Delivery workAmbiguous external effects stop in delivery_unknown; they do not auto-retry. ready claimed done delivery_unknown failed ackunknownresolve A claim deadline returns abandoned claimed work to ready; later work in the same lane waits behind unresolved earlier work.
Fig 2 · Shared mechanics, separate success conditions. Agent execution state is not duplicated here.

Canonical record

FieldMeaningInvariant
id, kindWork identity; intake or deliveryImmutable.
source_keyStable platform input identity, or agent-event/destination identityUnique within kind; duplicate accepts return the existing row.
lane_key, sequenceOrdering domain and monotonic positionNo later unresolved item overtakes an earlier item in the same lane.
conversation_id, agent_event_idJoin to agent-server truthIntake uses a deterministic event ID; delivery references the originating event.
state, available_atWork lifecycle and retry scheduleState changes by compare-and-set.
claim_owner, claim_untilShort ownership of one attemptExpired claims are recoverable; a stale owner cannot complete a newer claim.
attempts, last_errorRetry evidenceEvery failed attempt leaves an audit fact.
external_message_idPlatform acknowledgment for deliverySet before done when the platform returns one.
payload_jsonNormalized input or immutable delivery intentNo mutable process objects or credentials.

Adapter-computed lane key

Borrow Hermes’s deterministic session-key idea, but let each channel adapter compute the canonical key because it owns that platform’s chat, thread and alias semantics:

channel:{platform}:{account}:{chat}:{thread-or-root}

WhatsApp → channel:whatsapp:smolpaws:120363426327189600@g.us:root
Slack    → channel:slack:T06P212QSEA:C06P5NCGSFP:1783869407.798

DMs isolate by stable chat identity; a Slack or Discord thread gets its own lane. Platform aliases—especially WhatsApp JID/LID forms—are canonicalized by the adapter before it returns the key.

Persisted lane directory

The coordinator resolves the adapter’s key to an agent-server conversation and saves the correspondence even though the key is deterministic. That row is both routing state and debugging evidence:

lane_key         TEXT PRIMARY KEY
conversation_id  TEXT NOT NULL
platform         TEXT NOT NULL
account_id       TEXT
chat_id          TEXT NOT NULL
thread_id        TEXT
display_name     TEXT
created_at       TEXT NOT NULL
last_seen_at     TEXT NOT NULL

Repeated resolution of the same lane returns the same conversation_id. Operators can query either side to explain which platform chat created a conversation, why messages split, or whether alias normalization produced an unexpected lane; no version field is introduced yet.

Small coordinator interface

adapter.computeLane(input)        → lane descriptor
resolveLane(lane)                 → lane binding + conversation_id
acceptIntake(binding, input)      → existing-or-new work
projectDeliveries(conversation)   → number created
claimReady(worker, now)           → claimed work | null
settle(claim, outcome)            → done | retry_wait | delivery_unknown | failed
reconcile(now)                    → expired/reconciled count

The storage adapter hides the lane directory, SQLite transactions, lane-head selection, backoff and stale-claim fencing. Channel adapters compute canonical lane identity, normalize input, deliver output and optionally reconcile external effects; they do not persist the mapping themselves.

5Worked trace: two messages arrive during execution

Resolve laneThe WhatsApp adapter canonicalizes the chat address and computes channel:whatsapp:…:chat-7:root. The coordinator finds or creates its persisted conversation_id correspondence and updates last_seen_at.
Accept AMessage A becomes intake sequence 41 in that lane. The unique source key makes a webhook/poll replay a no-op.
Integrate AThe worker appends deterministic agent event uuid5(A) with run=true. Agent-server persists it, starts execution asynchronously, and intake A becomes done.
Accept B + CTwo more messages enter as sequence 42 and 43 while agent-server is active. Both are durable immediately; B is lane head, so C cannot overtake it.
Integrate B, then CThe coordinator appends in lane order. Agent-server’s active execution path records a rerun request rather than starting parallel work.
Project outputThe event projector catches up from its durable cursor, maps each deliverable agent event to one delivery row, then subscribes for live events. A unique event/destination source key makes replay harmless.
DeliverThe channel adapter claims delivery work in lane order. A confirmed platform message ID completes it; an ambiguous effect becomes delivery_unknown and blocks later delivery until reconciled or explicitly resolved.
What this permits: incoming messages can be durably accepted while agent-server is busy. Ordering is established before any HTTP call, while agent-server remains responsible for how those messages affect the next execution.
What this forbids: holding accepted messages only in adapter memory, deriving delivery from “latest response” after a restart, or retrying an external send whose outcome is unknown.

6Crash matrix

Crash or raceDurable fact before crashRecoveryResult
Duplicate platform inputExisting source_keyacceptIntake returns the same row.No duplicate intake.
Two first messages resolve one new laneUnique lane_keyOne lane-directory insert wins; both callers use the persisted conversation_id.One routing correspondence.
Crash after lane mapping, before conversation creationLane row with reserved conversation_idNext resolution ensures that exact conversation exists before accepting intake.Mapping remains repairable.
After accept, before claimready rowNext worker claims it.No loss.
After claim, before agent-server callClaim deadline + generationExpiry returns work to ready; stale worker is fenced.Safe retry.
Agent event appended; HTTP response lostDeterministic agent_event_idRetry same append; agent-server returns existing event.Requires idempotent append extension.
Event appended; execution request not observedEvent exists in EventLogRetry /run; active response means already executing, idle response schedules it.Event is not duplicated.
Agent-server dies during executionConversation EventLog + coordinator intake checkpointRestore conversation, observe non-terminal state, request execution again.Conversation history survives.
Second message arrives while activeNext lane sequenceAppend in sequence; agent-server records a rerun request.No parallel lane execution.
Projector dies before delivery insertAgent event + old durable cursorReplay event; unique event/destination key inserts once.No missing delivery work.
Projector dies after insert, before cursorDelivery row + old cursorReplay hits uniqueness constraint, then advances cursor.No duplicate row.
Delivery worker dies before sendClaimed row, no external IDClaim expires and retries.No external effect yet.
Platform rejects before acceptingRetryable errorBackoff through retry_wait; preserve lane order.Bounded retry.
Platform accepts; acknowledgment persistsexternal_message_idSet done; replay sees completion.Effectively-once.
Platform may accept; response lostNo proof either wayMark delivery_unknown; adapter reconciles when possible, otherwise operator chooses confirm/retry/skip.Exactly-once is impossible without platform support.
Repeated poison deliveryAttempts + errorsMove to failed after policy limit; lane remains blocked until explicit skip or repair.No silent overtaking.
Two coordinator processes claim togetherSQLite row state + generationBEGIN IMMEDIATE / compare-and-set update allows one winner.One active claim.
Machine restarts with queued Hermes-style follow-upCoordinator row, not an in-memory pending mapClaim after restart.Hermes UX, durable implementation.
The matrix defines the quality bar. Rows marked “requires” or “impossible without platform support” stay visible; architecture prose must not upgrade them into promises.

7ADR: considered options

Queue inside agent-serverrejectedIt couples platform delivery policy to conversation execution and creates a permanent divergence from the OpenHands contract.
Run Automation unchangedrejectedIts durable mechanics are useful, but automation definitions, tarballs, sandboxes, user/org ownership and cloud callbacks solve a different problem.
Hermes pending mapborrow policyExcellent per-lane behavior and race handling; unsuitable as the durable store because the pending slot and active guards live in process memory.
Retry logic in every adapterrejectedIt repeats crash handling, ordering and reconciliation across Slack, Discord, WhatsApp and future channels.
Local coordinatorchosenA small interface around SQLite localizes durable work while letting agent-server and channel adapters keep their natural responsibilities.

Consequences

  • The coordinator is a new local module and process responsibility, but not a new network service requirement; it can run in the SmolPaws host process initially.
  • Every adapter migrates to the same accept/claim/settle interface. Platform code stops polling agent execution directly.
  • SQLite is the first storage adapter. PostgreSQL may implement the same interface later with FOR UPDATE SKIP LOCKED, matching Automation’s multi-worker shape.
  • The current agent-server needs one narrow compatibility extension for deterministic message append. Without it, intake remains vulnerable to the append-response-loss window.
  • Delivery uncertainty becomes explicit product state. This is less comforting than “exactly once,” and more truthful.

8One required agent-server contract delta

The current message request contains role, content and run; agent-server creates a fresh event identity internally and returns only success. A coordinator that crashes after a successful append cannot safely distinguish “append happened” from “append never happened.”

Add idempotent message append without adding queue semantics. Accept a caller-supplied deterministic event_id or idempotency key, persist the event once, and return the event identity. Repeating the same request returns the existing event; reusing the key with different content is a conflict.
POST /api/conversations/{id}/events
{
  "event_id": "uuid-v5(platform + source_message_id)",
  "role": "user",
  "content": [...],
  "run": true
}
→ { "success": true, "event_id": "...", "created": false }

This is a compatible reliability extension at the existing event seam. It does not add external work states, claims, delivery acknowledgments or platform concepts to agent-server.

Projection rule: delivery work is derived from durable terminal-response facts, initially successful finish ObservationEvent payloads—the same durable facts agent-server already uses for agent_final_response. Explicit outbound-intent events may be introduced later; the coordinator’s event/destination uniqueness key keeps that evolution behind the projector interface.

9Implementation order

  1. Specify the store through tests. Concurrent lane resolution, persisted lane-to-conversation lookup, duplicate accept, per-lane order, claim fencing, expiry, backoff, projector replay and delivery_unknown outcomes all get deterministic SQLite tests.
  2. Add idempotent event append to agent-server. Prove append-response loss by retrying the same event identity across restart.
  3. Implement intake only. Run a shadow path that accepts and appends one non-critical channel while current delivery remains unchanged.
  4. Add event projection and delivery work. Start with terminal agent responses and a recording adapter; prove projector crash cases before real sends.
  5. Canary WhatsApp. It exercises media, aliases, restart recovery and the most visible duplicate/loss cases. Preserve a rollback switch.
  6. Migrate Discord, Slack, GitHub and email. They become platform adapters around the same coordinator interface rather than new orchestration implementations.
  7. Remove superseded orchestration paths. Only after a soak period shows no unresolved delivery_unknown states, ordering violations or duplicate intake.

No estimates are attached. Each step stops on an executable invariant, not on elapsed time.

10Source map and validation

ClaimGrounding
Agent-server appends messages, schedules execution and requests a rerun when active.eventService.ts:101–142
Agent-server publishes newly durable events and a state update after execution.eventService.ts:238–264
Conversation ownership is already guarded by an expiring lease.conversationService.ts:335–362
The current message request has no caller event identity.models.ts:39–48
Agent-server already derives its final response from durable assistant messages or successful finish observations.eventService.ts:206–220
Automation’s run row doubles as its queue and has explicit lifecycle states.models.py:36–45 · models.py:121–200
Automation claims oldest pending rows, uses DB locking where available and launches asynchronously.dispatcher.py:94–122 · dispatcher.py:347–400
Automation completion races settle through a guarded state update.router.py:325–390 · watchdog.py:225–275
Hermes holds one active task and one pending map entry per deterministic session key.base.py:1843–1867 · session.py:617–698
Hermes queues/merges follow-ups while a lane is active and drains them without overlap; the merge is documented for comparison and explicitly not adopted.base.py:3919–4110 · base.py:4490–4532
Cloudflare Queue can spool edge input on Workers Free: 10,000 operations/day included, 24-hour retention.Queues pricing
A pull consumer explicitly pulls and acknowledges messages from infrastructure outside Workers.HTTP pull consumers
Validation performed: code paths and focused tests were read at the pinned commits above. The new design is an ADR, so its crash guarantees remain proposed until the SQLite store and idempotent append tests execute them.
Vocabulary used by this ADR

Agent event: a durable event in a conversation’s EventLog. Intake work: one external input waiting to be integrated into agent-server. Delivery work: one immutable intent to deliver an agent event to one external destination.

Lane: the ordering domain for a chat or thread. Claim: short, fenced ownership of one work attempt. Delivery outcome unknown: an external effect may have happened but cannot yet be proved.