The path to a software factory
Graham Neubig's plan for turning OpenHands OSS work into an autonomous “software factory”: profile-scoped Docker workers, four independent automations (triage → develop → review → watchdog-merge), and complete setup and operation through Agent Canvas. This page mirrors and lightly formats that document so it is easier to peruse.
Source doc updated 2026-09-14 15:27 UTC. 13 PRs remain to ship the factory described below; SDK #4931 and #4966 and Extensions #581 have merged. Companion note on this site: What we take from pstack, which maps verification-as-infrastructure onto Graham's four factory bottlenecks.
factory-state links resolve only inside the working deployment.1Required outcome
A user starts with a fresh Agent Canvas, configures an LM and credentials through the UI, and adds four separate automations. Opening an application issue then starts this cycle:
The implementation must satisfy these requirements:
- Each automation uses a saved agent profile to select its model, tools, and allowed secrets.
- The same automation bundle works in local and Docker workspaces. The factory runs its workers in separate Docker sandboxes with bounded resources.
- Factory dispatch and Docker workspace operations use the SDK and TypeScript runtime clients.
- The reviewer can read code and publish findings/statuses without pushing code. The watchdog uses the developer token, as requested, and merges only accepted, tested, current heads.
- The operator only opens application issues; automations implement, review, test, revise, and merge the application code.
A PR is included below when removing its capability blocks setup or operation of this factory in the current implementation. Broader architecture follow-ups remain tracked in a separate classification record; they do not block factory delivery. Inclusion does not mean every line of that PR is indispensable. The running demo uses integrated preview builds; these changes still need upstream releases for a normal fresh install.
2Existing machinery
The SDK already provides agents, conversations, workspaces, profiles, secret storage, and clients. Automation already provides schedules, triggers, dispatch, run history, and timeout recovery. Extensions already provides implementation, review, and QA workflows. Canvas already provides model configuration, the profile library, and the automation catalog. The required changes connect and extend that machinery.
secret_refs and captures the allowed names for new conversations, enforcing that scope during launch, credential updates, and resume. SDK #4966 reuses the existing runtime service routers under conversation-scoped paths, adds the common lifecycle contract, and lets the existing TypeScript runtime clients select a conversationId. Both are in SDK main; downstream releases still need to include them. Extensions #581 extracts the request, pagination, repository, and per-repository execution code already shared by GitHub workflows.3Required changes
A. Profile-scoped workers and Docker isolation
The server must resolve the selected profile, retain its secret boundary, and run each worker through a conversation-scoped workspace. Shared contracts precede the Docker implementation.
| PR | Why it is required |
|---|---|
| SDK #5017 | Delivers only selected secrets to the worker and shell entrypoint, reusing the secret registry and masking. Without it, script-based automations cannot reliably receive their profile-selected GitHub token. |
| SDK #3403 | Implements the Docker runtime: provisioning, scoped credentials, forwarding, recovery, and cleanup. It provides the bounded-resource sandbox isolation the factory workers run inside. |
Profiles select repository-scoped fine-grained PATs. Triage updates issues; developers publish code; reviewers read code and publish reviews/acceptance statuses; the watchdog shares the developer PAT for guarded merges. The merge gate reads Actions and commit statuses. Any required external Check Runs must be enforced by GitHub protection without a watchdog bypass.
Older conversations without a captured secret scope retain legacy behavior; the new boundary applies to newly launched scoped conversations and their subsequent updates/resumes.
B. Four independent automations
Automation owns profile selection and dispatch. The four extensions own GitHub workflow decisions and reuse the canonical implementation, review, and QA machinery. Workers attach to the conversation provided by the service through the SDK.
| PR | Why it is required |
|---|---|
| SDK #5010 | Completes the Python RemoteWorkspace support omitted when #4966 added scoped routes and TypeScript routing. It also adds explicit profile-based conversation creation and read-only attachment, durable command start/poll, restricted runtime credential retrieval, and cleanup through the existing conversation and workspace classes. |
| Automation #449 | Dispatches bundles through the shared conversation runtime, admits bounded concurrent workers, detects completion, and releases resources. It provides the common local/Docker execution path. |
| Automation #453 | Saves each automation's selected profile and snapshots it on queued runs. Without it, the four roles cannot reliably select different capabilities and credentials through profiles. |
| Extensions #570 | Adds only independent triage: acceptance criteria, priority, dependency checks, readiness, and suppression of repeated unchanged triage. |
| Extensions #571 | Adapts canonical issue-to-PR delivery to the provided conversation, developer lanes, review feedback, and base updates. It implements ready issues and revises submitted PRs. |
| Extensions #572 | Extends only the existing reviewer to run readable review, independent tests, and canonical QA, then publishes acceptance for the exact tested commit. Its linked review and QA resources use the existing skill installer; it adds no catalog tooling. |
| Extensions #573 | Adds only the periodic watchdog that merges current, independently accepted heads with passing required statuses and Actions. It closes the issue-to-merge loop. |
Canvas packs each automation from the generated catalog's embedded file contents, including the shared GitHub helper and reviewer QA resources. These files travel with each bundle; workers need no external helper directory or extension-side profile loader.
SDK #5010 extends the existing conversation and workspace classes. Conversation creation and attachment are explicit methods: create submits a typed request; attach reads an existing conversation and fails if it is missing. They share the existing connection/event machinery. The constructor continues to require an agent. Automation, extensions, and docs have been migrated.
C. Complete setup and operation through Canvas
The UI must save profile selections, accept the canonical reviewer bundle, and address Docker conversations for files, commands, Git, and logs.
| PR | Why it is required |
|---|---|
| Canvas #17237 | Adds the profile secret-scope picker, gated by SDK enforcement support. It lets the user configure role-specific secret access entirely through the UI, including credentials entered while creating an ACP profile. |
| Canvas #17396 | Adds profile selection to automation creation/editing and preserves it in requests. It also requires a profile for bundles that need one, preventing a saved automation that cannot attach to its conversation. |
| Canvas #17403 | Accepts the extensions plugin-resource contract. Without it, the canonical reviewer bundle cannot open its direct setup form. |
| Canvas #17387 | Uses SDK runtime clients and the server's single conversation_runtime signal for files, commands, Git, and logs. This directs Canvas operations to the selected Docker conversation rather than the host workspace. |
The completed split audit records the cleanup and validation: runtime routing belongs in #4966, Docker compatibility belongs in #3403, and merged profile definitions are no longer repeated in #5017. The four catalogs require a profile and carry their own workflow resources.
4Review and release order
Start with independent SDK roots #5010 and #5017. SDK #4931 and #4966 are already merged. Then follow the native stacks:
| Stack | Review order |
|---|---|
| SDK | #5017 → #3403; both now include merged #4931 and #4966 |
| Automation | #449 → #453 |
| Extensions | Shared foundation #581 is merged. The former GitHub stack has been unstacked; #570, #571, #572, #573 are independent main-based PRs and can be reviewed or merged in any order. |
| Consumer PR | Prerequisite outside its native stack | Gate and reason |
|---|---|---|
| SDK #5010 | SDK #4966 release | Additive client code can be reviewed/merged separately; the merged runtime contract must also be included in the supported server release before these methods ship to consumers. |
| Automation #449 | SDK #5010 merged & released, with the #4966 contract | Its integration pin follows the current #5010 head and must become a released SDK dependency before merge. Docker operation additionally requires #3403; scoped factory credentials require #4931/#5017. |
| Automation #453 | SDK #5010 release; #4931/#5017 for scoped credentials | Inherits #449's merge/release gates. Selecting a profile does not itself enforce its secret boundary. |
| Extensions #570–#573 | No cross-repository merge prerequisite | They can merge independently now. Activating the three agent-backed workers needs Automation dispatch and SDK #5010. Running all four with the intended credential boundary additionally needs Automation #453, SDK #4931/#5017, Canvas #17396; Docker isolation needs #3403. |
| Canvas #17396 | Automation #453 merged & available first | UI profile selection needs the backend to persist and dispatch agent_profile_id. |
| Canvas #17387 | Merged SDK #4966 TS client released & pinned first | Uses the conversation-scoped runtime operations. Actual Docker workspace operation additionally requires SDK #3403. |
| Canvas #17237 | SDK #4931 to enable the feature | Can merge first: the picker is capability-gated; unavailable until the server supports profile secret enforcement; shell delivery also needs #5017. |
| Canvas #17403 | Extensions #572 for the factory reviewer use case | Can merge first: a backward-compatible validator expansion. The reviewer bundle needs it installed, rather than the validator needing the reviewer merged first. |
Release the Python SDK and TypeScript client capabilities before deploying the Automation and Canvas consumers that call them. The four Extensions roles can merge independently; enable their profile-backed catalog entries only on a compatible runtime. Replace temporary SDK source pins and update the Canvas package pins to those releases. SDK #3403 remains last in its stack: Docker implements the runtime and secret contracts established by its predecessors.
Current review state
- Extensions #570–#573: all four target
main, have clean CI, and have exact-headall-hands-botapprovals. They are the closest remaining PRs to merge. - SDK #5010 and #5017: CI is clean; both remain draft and require approval. SDK #3403 is clean and approved, but remains stacked on #5017.
- Automation #449 and #453: both approved, but their stack is behind current
main. Refreshing it must also reconcile the profile migration with024_add_run_sandbox_cleanup_due_at.py, which landed after the stack was created. - Canvas #17403: CI clean but remains draft and requires approval. Canvas #17237 and #17396 still fail the PR-description gate because their HUMAN testing notes are empty. Canvas #17387 additionally waits for a released TypeScript client containing #4966 before its build can pass.
5Validation and remaining acceptance
The operator only opened application issues. Automations wrote, reviewed, tested, revised, accepted, and merged the application code.
| Validation | Evidence and limit |
|---|---|
| Box clone | Final acceptance PR #8 merged after independent review and QA. Four application PRs completed. Box acceptance complete |
| Airbnb clone | 28 application PRs merged; final acceptance complete and repeatedly validated after the four-extension split. Independent QA found a blank-amenities crash in PR #50; the developer fixed it and the watchdog merged the corrected head. The first clean post-refactor run triaged issue #51, delivered PR #52, and auto-merged commit 15b7a1e. A later run on current independent Extensions heads triaged issue #55, delivered PR #56, published a readable exact-head review and before/after browser QA, set both factory statuses to success, and watchdog-merged commit a2f66a9 as 66b1ce5. |
| UI-only configuration | An empty isolated Canvas was configured through the UI with an LM, secrets, four profiles, and four automations (animated setup walkthrough and individual screenshots in the working deployment's evidence store). |
| Setup regressions | Fresh before/after Canvas recordings demonstrate a new ACP credential remaining selectable after deselection and retaining exactly its selected profile grant, and a profile-required automation rejecting an empty selection then saving the selected profile. Triage opens directly from the catalog with no GitHub MCP configured. Covers configuration/persistence; makes no new model-execution claim. |
| Workspace parity | The same bundle completed real model tasks locally and in Docker through explicit create/attach (30-second GIF). Missing attachment returns 404 without creating a conversation. Scoped file/Git operations, detached commands and automatic release passed. |
| Credential-scope correction | Live Canvas before/after recording proves excluded credentials are blocked on local ACP launch/resume, using a disclosed deterministic ACP subprocess. Explicit updates and Docker transport have separate regression coverage. |
The active UI-created factory runs at http://127.0.0.1:9104, with four separate schedules targeting Airbnb. Admission is limited to three Docker workers, each bounded to 2 GB RAM, one CPU, and 256 processes. The host memory guard pauses scheduling and stops workers below 2 GiB available memory. Automation serializes ticks from the same definition, so a long reviewer run does not create overlapping reviewers.
Mirror of Graham Neubig's document, reproduced for reference. Some links (evidence indexes, deployment records, factory-state paths) resolve only inside the working deployment and are described in text here rather than linked.