← Home

OpenHands Deferred Init Runtime Boundary architecture note

Notes from PR #3844, PR #400, and PR #3287, which introduced deferred init. The question: how should warm-pool agent-servers accept per-user runtime config without scattering mutable state across closures, globals, middleware, and service singletons?

Short answer: deferred init wants a standardized runtime boundary. Register routes once, but make every request and WebSocket resolve a typed AgentServerRuntime from app.state. Build the ready runtime off to the side, enter its services, then atomically swap it in. Avoid per-route config closures, module-level services, and conditional auth registration.

current merged path

PR #3287 added a dormant warm-pool mode. The server boots, starts stateless side services, mounts all routers, and waits for POST /api/init. That init call merges a narrow runtime payload into the dormant Config, starts ConversationService, stores the new config and service on app.state, then flips InitService.state to ready.

PR #3844 lesson

Auth was the first concrete failure. Dependencies captured startup config, so a key delivered by /api/init was not honored. Also, if there were no session keys at startup, the auth dependency was never registered. PR #3844 fixed this by making auth dependencies read request.app.state.config on each request and by registering them unconditionally.

1. Current flow

The current design separates route shape from runtime readiness. Routes exist immediately, but most /api/* routes include require_initialized, which returns 503 while the init service is dormant or initializing. The init endpoint uses X-Init-API-Key and the dormant server's bootstrap secret_key; the session key delivered later becomes the normal API credential.

sequenceDiagram
  participant Pod as Warm pod process
  participant App as FastAPI app
  participant Init as InitService
  participant Orchestrator
  participant Runtime as ConversationService
  participant Client

  Pod->>App: create_app(dormant Config)
  App->>App: mount routers and middleware
  App->>Init: attach only when deferred_init=True
  App-->>Orchestrator: /ready is ready, /api/* mostly 503
  Orchestrator->>Init: POST /api/init + X-Init-API-Key
  Init->>Init: merge InitRequest into base Config
  Init->>Runtime: build and enter ConversationService(new_config)
  Init->>App: app.state.config = new_config
  Init->>App: app.state.conversation_service = service
  Init-->>Orchestrator: state=ready
  Client->>App: /api/* + X-Session-API-Key
  App->>App: auth reads app.state.config at request time
  App->>Runtime: handle conversation/tool/workspace request
    
Boot

One FastAPI app is created from a dormant config. Its root path, static files, middleware, route dependencies, and some module globals are established at import/startup time.

Init

InitService merges the runtime payload, mutates process env, starts the conversation service, updates app.state, and marks the server ready.

Serve

HTTP auth now checks the latest config. Handlers use a mixture of app.state, default singleton accessors, and module-level service references.

Current risks

PR #3844 fixed the known session-key bug, but the shape that produced it is still easy to repeat. The risks are less about one bad dependency and more about the lack of one canonical place where request-time runtime state is resolved.

Captured startup config

Any dependency factory or closure that receives Config during route registration can silently ignore the runtime config delivered by /api/init.

Conditional registration

If routes decide at startup whether auth or guards are needed, a dormant server that starts without keys may never protect the routes after keys arrive.

Multiple sources of truth

The effective runtime can live in base_config, app.state.config, conversation-service singletons, route-module globals, and middleware copies.

WebSocket parity drift

HTTP dependencies can be corrected while WebSocket modules still use import-time service objects or default config helpers unless they share the same resolver.

Startup-only app shape

Values like root_path, static file mounts, and CORS middleware are established when the app is created. If they are allowed in InitRequest, they need dynamic handling or they are only partially effective.

Process-wide env mutation

InitRequest.env updates os.environ. That is acceptable for one runtime per process, but it is not scoped, reversible, or naturally test-isolated.

2. PR #400-style flow

PR #400 was not the deferred-init PR. Its goal was to move agent-server toward a FastAPI factory pattern: create the app from an explicit Config, store config and services on app.state, and make routers/dependencies resolve services from that app state instead of relying on imported globals. That is the right direction for deferred init because it makes the app instance the ownership boundary.

flowchart TD
  subgraph Factory[App factory]
    C["Config input"] --> A["create_app(config)"]
    A --> R["register routers once"]
    A --> S["app.state.runtime = DormantRuntime"]
  end

  subgraph Provider[Request-time provider]
    G["get_runtime(request or websocket)"]
    G --> Auth["auth policy"]
    G --> Ready["require_ready_runtime"]
    G --> Services["conversation, bash, workspace services"]
  end

  subgraph Init[Deferred initialization]
    I["POST /api/init"] --> B["build ReadyRuntime off to the side"]
    B --> E["enter services"]
    E --> Swap["atomic app.state.runtime swap"]
  end

  R --> G
  S --> G
  Swap --> G
    

A deferred-init version of that pattern should go one step further than PR #400's original sketch: the thing stored on app.state should not be just config plus several independent service attributes. It should be a typed runtime object with a single state transition.

@dataclass(frozen=True)
class DormantRuntime:
    bootstrap_config: Config
    state: Literal["dormant", "initializing"]

@dataclass(frozen=True)
class ReadyRuntime:
    config: Config
    conversation_service: ConversationService
    bash_event_service: BashEventService
    state: Literal["ready"] = "ready"

AgentServerRuntime = DormantRuntime | ReadyRuntime

Standardized pattern

App shape is static; runtime is explicit. The factory owns route registration and static server-shape decisions. A runtime provider owns all request-time config, auth, and services. Deferred init replaces the runtime object only after the new runtime is fully built and entered.

Current vs. runtime-context design

ConcernCurrent merged flowRuntime-context flow
Route registrationRoutes are registered once. PR #3844 removed session-key conditionals for HTTP auth.Routes are registered once, always with the same typed dependencies. No route decides auth based on startup keys.
Config lookupSome auth paths read app.state.config; other code may still use default accessors or import-time objects.Every route, WebSocket, and service dependency uses get_runtime(...). Fallbacks to process defaults are not part of request handling.
ReadinessInitService.state gates most /api/* routes. Other surfaces need to remember to participate.require_ready_runtime returns a ReadyRuntime or 503. Any handler needing services must request this dependency.
InitializationInit mutates env, resets a singleton, enters service, then updates multiple app.state fields.Init constructs and enters a complete ready runtime locally, then swaps one pointer. Failure cleans up the local candidate and leaves the old runtime intact.
TestingTests can miss stale closures unless they mutate app.state.config after route registration.Unit tests target the provider: dormant returns 503, ready returns typed services, swap changes auth and services for HTTP and WebSocket together.
Design pressureEach new router must remember which global helpers are unsafe in deferred mode.The safe path is the default. A router cannot get runtime services without going through the provider.

Recommended boundary split

The deepest simplification is to stop treating all config fields as equally mutable. Some fields define the ASGI app's shape; others define the user runtime. Mixing them is how partial application bugs appear.

flowchart LR
  Static[Server-shape config
root_path, static files, middleware strategy, enabled routers] --> Factory[create_app] Runtime[Runtime config
session keys, secret, persistence paths, webhooks, user env] --> Init[POST /api/init] Factory --> App[FastAPI app] Init --> Context[AgentServerRuntime] Context --> Req[HTTP + WebSocket dependencies] App --> Req
Static app config

If changing a value would require rebuilding middleware, remounting routes, or changing root path behavior, prefer setting it before create_app. If it must vary per user, implement it as dynamic middleware that reads the runtime provider.

Runtime config

If a value affects auth, storage locations, user webhooks, secret encryption, or tools used after init, it belongs in the ready runtime and should be visible through one provider.

State transition

Use a candidate runtime: build, enter, validate, then swap. Do not publish partially initialized pieces across several app.state attributes.

What I would do next

  1. Introduce a typed runtime provider. Keep it small: get_runtime, get_ready_runtime, HTTP auth, WebSocket auth, and service accessors. Make it the only approved request-time path to config-backed services.
  2. Move WebSockets onto the same provider. This closes the most obvious parity gap left by fixing HTTP auth: auth and event/bash services should come from the same ready runtime as REST handlers.
  3. Classify init fields. Decide which InitRequest fields are truly runtime mutable. For web_url and CORS, either make the middleware/root-path behavior dynamic or remove them from deferred-init payload expectations.
  4. Retire default singleton use inside request handlers. Default factories can remain for the normal app bootstrap path, but routers should not call them after the app exists.
  5. Add mutation tests around the boundary. Tests should update or swap the runtime after route registration and assert old keys/services stop working and new keys/services work across REST, workspace cookie auth, OpenAI routes, and WebSockets.

Design bias: make deferred init boring. The warm-pool controller should deliver one payload; the server should either remain dormant or publish one ready runtime. Every handler should ask the same question: “what runtime owns this request?” If the answer is standardized, stale closures and missing guards become much harder to write.

Source trail