Short version: the service uses the database as both catalog and durable queue. Cron polling, webhook ingress, and manual dispatch all create AutomationRun rows in PENDING. A dispatcher claims those rows, starts bash in an execution backend, then returns immediately. Completion is reconciled by an SDK callback, with a watchdog that verifies stale runs if the callback is missed.
OpenHands Automations is not the agent runtime. It owns the when and the packaging of what to run. The actual work happens in an agent server: either a new OpenHands Cloud sandbox per run or a configured local/self-hosted agent server.
flowchart LR
User["User / UI / API client"] --> FE["Automations frontend SPA"]
User --> API["FastAPI routers under /api/automation/v1"]
FE --> API
subgraph Service["Automation service"]
API --> DB[("PostgreSQL or SQLite")]
API --> Store["FileStore: GCS, S3, or local"]
Scheduler["scheduler_loop"] --> DB
Events["event_router"] --> DB
Dispatcher["dispatcher_loop"] --> DB
Dispatcher --> Exec["execute_in_context"]
Watchdog["watchdog_loop"] --> DB
end
Store --> Dispatcher
Exec --> Backend{"ExecutionBackend"}
Backend --> Cloud["CloudSandboxBackend: create sandbox"]
Backend --> Local["LocalAgentServerBackend: reuse agent server"]
Cloud --> Agent["OpenHands agent server"]
Local --> Agent
Agent --> Callback["POST /v1/runs/{run_id}/complete"]
Callback --> API
| Piece | Files | Role |
|---|---|---|
| Application shell | app.py, config.py, db.py | Creates the FastAPI app, shared HTTP client, async database engine/session factory, routers, static frontend hosting, and background tasks. |
| Automation API | router.py | CRUD, manual dispatch, run listing, tarball download, completion callback, and cancellation. All user-facing operations are owner/org scoped. |
| Preset API | preset_router.py | Builds SDK tarballs from prompt or plugin requests, writes them to storage, then creates normal automation records. |
| Uploads | uploads.py, storage/* | Streams user tarballs with a size limit into the configured file store and exposes completed uploads as oh-internal://uploads/{id}. |
| Event ingress | event_router.py, webhook_router.py, utils/webhook.py | Verifies HMAC signatures, parses built-in and custom webhook payloads, matches event trigger patterns and filters, then creates pending runs. |
| Scheduling | scheduler.py, utils/cron.py | Polls enabled cron automations, rotates batches fairly via last_polled_at, and creates pending runs when a schedule is due. |
| Dispatch | dispatcher.py, execution.py, backends/* | Claims pending runs, resolves execution backend, prepares tarball and environment, starts the bash command, and records sandbox/command ids for later verification. |
| Reconciliation | watchdog.py, utils/agent_server.py, utils/sandbox.py | Scans stale running rows, checks the actual bash result in the execution backend, marks missed callbacks terminal, and cleans up resources. |
The important design choice is that no separate queue system is required. The automation_runs table is the durable queue, while automations is the catalog of definitions and tarball_uploads points to executable bytes.
erDiagram
AUTOMATION ||--o{ AUTOMATION_RUN : creates
AUTOMATION {
uuid id
uuid user_id
uuid org_id
string name
json trigger
text tarball_path
text entrypoint
int timeout
bool enabled
datetime last_polled_at
datetime last_triggered_at
}
AUTOMATION_RUN {
uuid id
uuid automation_id
enum status
json event_payload
string sandbox_id
string bash_command_id
string conversation_id
datetime timeout_at
text error_detail
}
TARBALL_UPLOAD {
uuid id
uuid user_id
uuid org_id
enum status
text storage_path
bigint size_bytes
}
CUSTOM_WEBHOOK {
uuid id
uuid org_id
string source
string webhook_secret
string signature_header
string event_key_expr
bool enabled
}
Queue invariant: once a run row exists in PENDING, it is eligible for the dispatcher. Cron, webhook, and manual entry points deliberately converge on that same state instead of having three execution paths.
There are two creation shapes. Custom automations upload a tarball and create a record pointing at it. Presets generate the tarball for the user: prompt automations include main.py, prompt.txt, setup.sh, and optional repo config; plugin automations add plugin or experiment config.
sequenceDiagram
participant Client
participant API as "preset_router / uploads"
participant Store as FileStore
participant DB as Database
alt Prompt or plugin preset
Client->>API: POST /v1/preset/prompt or /plugin
API->>API: generate SDK tarball
else Custom tarball
Client->>API: POST /v1/uploads raw tar bytes
end
API->>DB: create TarballUpload(UPLOADING)
API->>Store: stream tarball bytes
Store-->>API: size_bytes
API->>DB: mark upload COMPLETED
API->>DB: create Automation(trigger, tarball_path, entrypoint)
API-->>Client: AutomationResponse
Generates SDK boilerplate that loads the prompt, optional repo list, LLM settings, secrets, and MCP configuration, then runs a normal OpenHands SDK conversation and posts the completion callback.
Packages plugin source declarations or A/B experiment variants into JSON config files. The runtime script fetches plugins and invokes the same prompt path with those capabilities loaded.
Lets a caller provide their own SDK script and entrypoint. The service validates and stores the tarball, but dispatch treats it like any other automation definition.
For preset prompt automations, changing the prompt regenerates the stored tarball because the runnable prompt is baked into prompt.txt; the database prompt column is metadata.
Triggers only decide whether to create a run. They do not execute code directly.
flowchart TD
Cron["scheduler_loop polls enabled cron rows"] --> Due{"is_automation_due?"}
Due -->|yes| Pending["create_pending_run"]
Due -->|no| Polled["update last_polled_at only"]
Webhook["POST /v1/events/{org_id}/{source}"] --> Sig{"valid HMAC signature?"}
Sig -->|no| Reject["401 / 404"]
Sig -->|yes| Parse["parse_event / event_key_expr"]
Parse --> Match["matches_trigger: source, on pattern, JMESPath filter"]
Match -->|matched| Pending
Match -->|not matched| NoRun["received, zero runs"]
Manual["POST /v1/{automation_id}/dispatch"] --> Pending
Pending --> Run[("AutomationRun status=PENDING")]
PostgreSQL deployments use FOR UPDATE SKIP LOCKED in the scheduler and dispatcher so multiple service workers can avoid claiming the same rows. SQLite mode skips those locks and assumes a single-process local deployment.
The dispatcher is intentionally not a blocking job runner. It claims a batch, changes each row to RUNNING with a precomputed timeout_at, and launches an async background task that starts the command in the execution backend.
sequenceDiagram
participant D as Dispatcher
participant DB as Database
participant B as ExecutionBackend
participant S as Storage
participant A as Agent server
participant Script as Automation code
participant API as Completion API
D->>DB: select oldest PENDING rows
D->>DB: mark RUNNING and set timeout_at
D->>B: get_execution_context
alt Cloud mode
B->>B: mint per-user API key
B->>B: create sandbox and wait RUNNING
else Local mode
B->>B: return configured agent server URL and key
end
D->>S: read internal tarball bytes
D->>A: upload tarball or curl external URL
D->>A: start bash with exports, extraction, setup, entrypoint
A-->>D: bash command id
D->>DB: persist sandbox id and bash command id
D-->>D: return without waiting for process exit
Script->>API: post run completion callback
API->>DB: RUNNING to COMPLETED or FAILED
ExecutionBackend is the seam that keeps the dispatcher and watchdog mode-agnostic. Both modes expose the same verbs: acquire a context, build environment variables, verify a stale run, and clean up after verification.
| Concern | Cloud sandbox backend | Local agent-server backend |
|---|---|---|
| Context | Creates a fresh sandbox via OpenHands Cloud, polls until RUNNING, and reads the exposed AGENT_SERVER URL plus session key. | Returns the configured persistent agent-server URL and API key. |
| Auth | Mints a per-user API key from the service key and retries once on 401/403 by refreshing it. | Uses the configured local agent-server API key and optional callback API key. |
| Workspace | Extracts into /workspace/project in the fresh sandbox. | Extracts under {workspace_base}/automation-runs/{run_id} for run isolation. |
| Cleanup | Deletes the sandbox after completion or failed verification unless keep_alive is set. | No-ops; the agent server is managed externally. |
The service and user code communicate through environment variables. The dispatcher builds these after acquiring the backend context, then execute_in_context() exports them before setup.sh and the entrypoint.
| Variable | Meaning |
|---|---|
OPENHANDS_API_KEY | Per-user key for OpenHands Cloud and SDK calls in Cloud mode. |
OPENHANDS_CLOUD_API_URL | Cloud API base URL. |
AGENT_SERVER_URL | Local-mode URL as seen from inside the bash chain. |
SESSION_API_KEY | Agent-server session key for files, bash, settings, and SDK workspace calls. |
SANDBOX_ID | Cloud sandbox id when available. |
AUTOMATION_CALLBACK_URL | Run completion endpoint on the automation service. |
AUTOMATION_RUN_ID | The run id the callback should report. |
AUTOMATION_API_URL | Automation service base URL. |
AUTOMATION_EVENT_PAYLOAD | JSON object with trigger type, full trigger config, automation metadata, event payload when present, and model when configured. |
AUTOMATION_MODEL | Resolved model profile name for the run, when configured. |
Callbacks, watchdog scans, and cancellation can race. The code resolves that by making terminal updates conditional: update only rows that are still in the expected non-terminal state.
stateDiagram-v2
[*] --> PENDING: cron / event / manual
PENDING --> RUNNING: dispatcher claims
PENDING --> CANCELLED: user cancels before claim
RUNNING --> COMPLETED: SDK callback status COMPLETED
RUNNING --> FAILED: SDK callback FAILED or dispatch/runtime error
RUNNING --> CANCELLED: user cancels running run
RUNNING --> COMPLETED: watchdog verifies exit_code 0
RUNNING --> FAILED: watchdog verifies error or timeout
PENDING --> SKIPPED: concurrency limit before execution
COMPLETED --> [*]
FAILED --> [*]
CANCELLED --> [*]
SKIPPED --> [*]
Race boundary: the completion endpoint and watchdog both use UPDATE ... WHERE status = 'RUNNING' and check rowcount. If another actor has already made the run terminal, the loser gets a conflict or simply skips the row; it does not overwrite the final result.
After start_bash_command returns, the dispatcher stores the agent-server bash_command_id. The watchdog passes that id back to the verifier so it reads the output for this automation run, not the latest unrelated bash activity on a shared local agent server or a busy sandbox.
flowchart LR
Run["RUNNING run past timeout_at"] --> Verify["backend.verify_run(run_id)"]
Verify --> Cmd["query agent-server bash history by bash_command_id"]
Cmd --> Outcome{"verified?"}
Outcome -->|exit_code 0| Done["mark COMPLETED: callback missed"]
Outcome -->|nonzero| Fail["mark FAILED with stderr/stdout excerpt"]
Outcome -->|none or killed| Timeout["mark FAILED: timed out"]
Outcome -->|cannot verify| Cleanup["backend cleanup unless keep_alive"]
Cleanup --> Fail2["mark FAILED: no callback / verification error"]
AutomationRun(PENDING), which keeps dispatch logic small.ExecutionBackend.