Skip to content
usine

Workers and the Gateway

Worker kinds in usine.yaml, preflight, the Hetzner Gateway VM, how each kind of Worker gets a GitHub token for its requests, and the cloud-init files.

A Worker is the machine where an Attempt’s checkout is and where its children run. The Pi parent always runs in the orchestrator. The operating recipes (creating machines, logging in to the harnesses, the SSH configuration, deleting) are in infra/hetzner/README.md and infra/hetzner/gateway/README.md; this page explains what the code does.

Worker kinds in usine.yaml

Section titled “Worker kinds in usine.yaml”

Worker is a union of three classes (factory.ts:150). Without a workers key the factory has one Worker named local (factory.ts:214).

kind Fields Executor Children run GitHub credential on the machine
local name, max_concurrent_attempts (default 1) local.ts as processes on the orchestrator’s machine token file in a private temporary directory
ssh name, destination, root, optional identity_file, port, max_concurrent_attempts, gateway ssh/ssh.ts one SSH command each token file under <root>/workers/<name>/credentials/
ssh with gateway gateway.domain, gateway.admin_url, gateway.admin_token_env, optional gateway.llm_providers (only anthropic) the same SSH executor one SSH command each; the operator’s SSH configuration routes it through the Gateway with ProxyJump none; a caller token in the SSH session’s environment
exe.dev name, max_concurrent_attempts exe-dev.ts one SSH command each on the VM none; exe.dev’s edge adds it

The two ssh forms, as they appear in the Hetzner READMEs:

usine.yaml
workers:
  - name: eu-one
    kind: ssh
    destination: usine@203.0.113.7
    identity_file: ~/.ssh/usine_worker
    root: /home/usine/.usine
    max_concurrent_attempts: 2

usine up uses one executor for all Workers. workerEngineLayer (worker-engine.ts:13) picks the SSH executor when every Worker is ssh and the local executor otherwise. stepExecutorLayer then refuses any Worker whose kind differs from the executor’s with UnsupportedWorkerError (scheduler.ts:76). A mix of kinds, or any exe.dev Worker, therefore stops startup. The exe.dev executor also needs githubIntegrations options that usine.yaml can’t set (exe-dev.ts:34), so it runs only from the integration test (pnpm --filter @usine/orchestrator test:exe-dev). #228 tracks both.

flowchart TD
  workers["Workers in usine.yaml"] --> all{"every Worker<br/>has kind ssh?"}
  all -->|yes| ssh["SSH executor"]
  all -->|no| local["local executor"]
  ssh --> same{"every Worker has<br/>the executor's kind?"}
  local --> same
  same -->|no| refused(["UnsupportedWorkerError:<br/>usine up stops"])
  same -->|yes| preflight(["preflight every Worker"])

What the executor does for each kind

Section titled “What the executor does for each kind”

local

Section titled “local”

State is under <instance>/.usine/workers/<worker>/: repos/ for bare caches, attempts/ for worktrees, turbo/ for TURBO_CACHE_DIR (local.ts:103, local.ts:290). Credentials go to a usine-credentials-* directory in the OS temporary folder, mode 700, with the token in token and gh’s hosts.yml and config.yml next to it (local/credentials.ts:22). Git reads the token through a credential helper and gh through GH_CONFIG_DIR (credential-environment.ts:15), so a renewed file takes effect without restarting a child. The executor removes every issued token from Git output before the output goes into an error or a log (local.ts:133).

ssh without a Gateway

Section titled “ssh without a Gateway”

Every command is ssh -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=4 [-p port] [-i key -o IdentitiesOnly=yes] <destination> -- <quoted command> (commands.ts:87, ssh/ssh.ts:22, child-host.ts:85). The SSH client inherits only HOME, LANG, LC_ALL, PATH, SHELL, SSH_AGENT_PID, SSH_AUTH_SOCK, TMPDIR, and USER (commands.ts:70). The Worker directory on the machine is <root>/workers/<name>.

A shell script writes the token to <root>/workers/<name>/credentials/<attempt> from standard input, so the token doesn’t appear on any command line (ssh/credentials.ts:25). The directories are mode 700 and the files 600. The first Attempt of each Worker in a process removes every credential directory that an earlier process left. All Attempts on one machine run as the same user and can read each other’s tokens (ADR 0026, #231).

ssh behind a Gateway

Section titled “ssh behind a Gateway”

gatewayAttachmentsFor (attachments.ts:58) gives the Attempt repository access with no token on the machine:

  1. Write gh’s configuration to <worker directory>/gh: http_unix_socket: /run/usine-gateway/http.sock and the host git.<domain>. Neither file contains a token (gateway/environment.ts:42).
  2. Create the Attempt’s token and send POST /v1/attachments to the Gateway’s admin API. The response contains the attachment id and a caller token.
  3. Renew the same attachment id with PUT before the token expires, and store it again every two minutes (attachments.ts:35), because the Gateway keeps attachments in memory and loses them when it restarts. The caller token stays the same.
  4. DELETE the attachment when the Attempt’s scope closes.
  5. Once per process and Worker, detach the attachments that an earlier process left for that Worker (attachments.ts:64).

The children’s environment has two parts (gateway/environment.ts:88). The exported part is in each command line and contains no secret. The sent part goes through OpenSSH SendEnv (commands.ts:101), so the caller token doesn’t appear on any command line; the Worker’s sshd must accept each sent variable name with AcceptEnv.

Variable Value Part
GIT_CONFIG_COUNT 3 exported
GIT_CONFIG_KEY_0 http.http://git.<domain>/.extraHeader exported
GIT_CONFIG_VALUE_0 x-usine-gateway-token: <caller token> sent
GIT_CONFIG_KEY_1, GIT_CONFIG_KEY_2 url.http://git.<domain>/.insteadOf exported
GIT_CONFIG_VALUE_1, GIT_CONFIG_VALUE_2 https://github.com/, https://git.<domain>/ exported
GH_HOST git.<domain> exported
GH_CONFIG_DIR <worker directory>/gh exported
GH_PROMPT_DISABLED 1 exported
HTTPS_PROXY, HTTP_PROXY, https_proxy, http_proxy http://<domain>:3128 exported
NO_PROXY, no_proxy localhost,127.0.0.1,::1,<domain> exported
GH_ENTERPRISE_TOKEN, GH_TOKEN the caller token sent
ANTHROPIC_BASE_URL http://llm.<domain>/anthropic, only when llm_providers contains anthropic exported
ANTHROPIC_AUTH_TOKEN the caller token, only when llm_providers contains anthropic sent

The clone URL is http://git.<domain>/<owner>/<repo>.git (gateway/environment.ts:33).

Shared by every remote kind

Section titled “Shared by every remote kind”

remoteExecutorFor (remote.ts:44) contains the checkout, salvage, recovery, and reattach code that SSH and exe.dev share. Each Claude Code or Codex child is one SSH command with a pid file <worktree>.child-<n>.pid (child-host.ts:96). CHILD_KILL_SCRIPT sends TERM, waits up to five seconds, then sends KILL, before the worktree is kept or removed (child-host.ts:106).

Preflight

Section titled “Preflight”

stepExecutorLayer runs Executor.preflight for every configured Worker with every rostered brain while the runtime layer is built (scheduler.ts:96). A failure stops usine up. prepare repeats the check for the Attempt’s brain only (local.ts:331, remote.ts:236). A missing capability found there fails the Attempt with AgentFailure; a transport failure or a timeout is a retryable InfraFailure (scheduler.ts:51).

flowchart TD
  kind{"Worker kind"} -->|local| lgit["git --version"]
  lgit --> lbrain["each brain, with the<br/>child's environment"]
  kind -->|ssh| connect["ssh destination true"]
  connect --> root["root is a<br/>writable directory"]
  root --> sgit["git --version"]
  sgit --> gw{"gateway?"}
  gw -->|no| sbrain["each brain over SSH"]
  gw -->|yes| admin["the admin API accepts<br/>the admin token"]
  admin --> sent["sshd accepts the<br/>sent variables"]
  sent --> curl["curl to git.domain returns<br/>missing-caller-token"]
  curl --> gbrain["each brain behind<br/>the Gateway"]
  lbrain --> ok(["Worker preflight verified"])
  sbrain --> ok
  gbrain --> ok

Behind a Gateway, the check for each brain depends on where its harness gets its model credential (gateway/preflight.ts:162):

flowchart TD
  harness{"brain's harness"} -->|"claude, with anthropic<br/>in llm_providers"| binary["claude --version only"]
  harness -->|"claude without it,<br/>or codex"| login["login check, as on a<br/>Worker without a Gateway"]
  login --> hosts["each login host responds<br/>through the forward proxy"]
  harness -->|pi| refused["refused: Pi children<br/>do not run remotely"]
Worker Checks Code
local git --version; each brain where children run, with the child’s environment local.ts:256, agents/src/preflight/preflight.ts
ssh SSH connects (true); root is a writable directory; git --version; each brain over SSH; Pi is refused on a remote machine ssh/preflight.ts:31
ssh with a Gateway The checks above, plus: the admin API accepts the admin token; sshd accepts the variables that contain the caller token; curl http://git.<domain>/ from the machine is refused because it has no caller token; for each harness that uses its login on the machine, the login’s hosts respond through the forward proxy; only the Claude Code binary is checked when the Gateway has the Anthropic key gateway/preflight.ts:47, gateway/preflight.ts:162

Local and direct SSH preflight don’t check the GitHub credential. Only the exe.dev preflight raises the credentials capability (exe-dev/preflight.ts:123, #278).

The Hetzner Gateway VM

Section titled “The Hetzner Gateway VM”

infra/hetzner/gateway/gateway.sh create creates a private network (10.42.0.0/16), the Gateway VM with a public address, and a Hetzner firewall that allows SSH from the orchestrator’s address. On first boot, gateway-cloud-init.yaml installs the parts below and writes an admin token and a caller key, generated with openssl rand -hex 32, to /etc/usine-gateway/secrets.env.

The orchestrator connects to the Worker and the admin API through the Gateway’s sshd. The diagrams use the defaults of gateway.sh: the domain gw.internal and the Gateway’s private address 10.42.0.2.

flowchart LR
  subgraph orchestrator["Orchestrator machine"]
    executor["executor"]
    forward["local forward<br/>127.0.0.1:7070"]
  end
  subgraph gateway["Gateway VM"]
    sshd["sshd :22"]
    adminapi["admin API<br/>127.0.0.1:7070"]
  end
  subgraph private["Private network 10.42.0.0/16"]
    worker["Worker VM<br/>sshd :22"]
  end
  executor -->|"attach, renew, detach,<br/>list: bearer admin token"| forward
  forward -->|"ssh -L, kept open<br/>by the operator"| sshd
  sshd --> adminapi
  executor -->|"ssh -J, caller token<br/>in SendEnv"| sshd
  sshd -->|"SSH into the private<br/>network only"| worker

A Worker has no public address and no default route. All its outbound traffic goes through the Gateway:

flowchart LR
  subgraph worker["Worker VM"]
    git["git"]
    gh["gh"]
    socket["/run/usine-gateway/<br/>http.sock"]
    claude["Claude Code, with<br/>anthropic in llm_providers"]
    tools["package managers,<br/>harness logins, apt"]
  end
  subgraph gateway["Gateway VM 10.42.0.2"]
    dns["dnsmasq :53"]
    reverse["reverse proxy :80"]
    egress["forward proxy :3128"]
  end
  gh --> socket
  socket -->|"systemd socket proxy"| reverse
  git -->|"git.gw.internal"| reverse
  claude -->|"llm.gw.internal"| reverse
  tools -->|"HTTPS_PROXY, HTTP_PROXY,<br/>apt's proxy setting"| egress
  worker -.->|"name lookups"| dns
  reverse -->|"HTTPS, installation token"| github["github.com<br/>api.github.com<br/>uploads.github.com"]
  reverse -->|"HTTPS, the Gateway's key"| llm["Anthropic, OpenAI"]
  egress -->|"allowlisted hosts only"| hosts["package hosts,<br/>login hosts"]
Part Listens on Does Code
Reverse proxy (usine-gateway) port 80 on the private address (the binary’s default is 127.0.0.1:80) git.<domain> forwards to github.com, api.github.com (paths under /api/), and uploads.github.com with the attached installation token; llm.<domain> forwards to Anthropic or OpenAI with the Gateway’s own credential. Every request needs a caller token whose attachment is active; a request for any other host gets 404 proxy.ts, github.ts:15
Forward proxy port 3128 on the private address CONNECT to port 443 of an allowlisted host only, after checking that the TLS ClientHello’s server name is that host and that the name resolves to public addresses only; the proxy forwards plain HTTP with caller tokens removed egress-proxy.ts:241, client-hello.ts:88
Admin API 127.0.0.1:7070 POST /v1/attachments, PUT /v1/attachments/:id, DELETE /v1/attachments/:id, GET /v1/attachments, bearer admin token api.ts:92
dnsmasq port 53 on the private address Answers <domain> and its subdomains with the Gateway’s address and refuses other names; local=/<domain>/ answers AAAA with no address, for Codex’s musl resolver (#260) gateway-cloud-init.yaml:42
nftables Input: SSH, plus DNS, 80, and 3128 from the private network. No forwarding. Outbound SSH only into the private network gateway-nftables.conf
sshd port 22 Bastion with AllowTcpForwarding local, for ssh -J to Workers and the admin API forward gateway-cloud-init.yaml:66

gateway.sh replaces the @@NAME@@ placeholders before it creates the VM. These tabs show the service, DNS, and SSH settings from gateway-cloud-init.yaml, and the firewall rules:

infra/hetzner/gateway/gateway-cloud-init.yaml
write_files:
  # ...
  - path: /etc/usine-gateway/settings.env
    permissions: "0644"
    content: |
      USINE_GATEWAY_DOMAIN=@@DOMAIN@@
      USINE_GATEWAY_PROXY_HOST=@@GATEWAY_IP@@
      USINE_GATEWAY_PROXY_PORT=80
      USINE_GATEWAY_EGRESS_HOST=@@GATEWAY_IP@@
      USINE_GATEWAY_EGRESS_PORT=3128
      USINE_GATEWAY_ADMIN_HOST=127.0.0.1
      USINE_GATEWAY_ADMIN_PORT=7070
      USINE_GATEWAY_EGRESS_ALLOWLIST=/etc/usine-gateway/egress-allowlist

The admin API is an Effect HttpApi that the executor’s client and the Gateway’s server share:

packages/gateway/src/admin/api.ts
export const GatewayAdminApi = HttpApi.make("GatewayAdminApi").add(
  HttpApiGroup.make("attachments")
    .add(
      HttpApiEndpoint.post("attach", "/v1/attachments", {
        payload: AttachRequest.annotate(strict),
        success: Attached,
        error: AttachmentAlreadyExpired.annotate({ httpApiStatus: 422 }),
      }),
      HttpApiEndpoint.put("renew", "/v1/attachments/:id", {
        params: { id: AttachmentId },
        payload: AttachRequest.annotate(strict),
        success: Attached,
        error: [
          AttachmentAlreadyExpired.annotate({ httpApiStatus: 422 }),
          AttachmentConflict.annotate({ httpApiStatus: 409 }),
        ],
      }),
      HttpApiEndpoint.delete("detach", "/v1/attachments/:id", {
        params: { id: AttachmentId },
        success: Detached,
      }),
      HttpApiEndpoint.get("list", "/v1/attachments", { success: AttachmentList }),
    )
    .middleware(AdminAuthorization),
);

The caller token is HMAC-signed with the caller key and contains its attachment id, Worker, and repository (caller.ts:93). The reverse proxy reads it from the x-usine-gateway-token header, an Authorization header, or x-api-key, and removes it before it forwards the request. The proxy refuses a token from a detached attachment, even when a new attachment exists for the same Worker and repository. Refusals have a reason in the x-usine-gateway-refusal header (proxy.ts:33):

Reason Status
unknown-host, unsupported-path 404
missing-caller-token, invalid-caller-token 401
stale-caller-token, attachment-expired, repository-mismatch 403
provider-not-configured 503

The forward proxy has its own reasons: host-not-allowed, port-not-allowed, address-literal, private-address, not-tls, server-name-missing, and server-name-mismatch (egress.ts:59). The service reads its settings once at start (node/environment.ts:64): USINE_GATEWAY_DOMAIN, USINE_GATEWAY_ADMIN_TOKEN, USINE_GATEWAY_CALLER_KEY, the allowlist file (default /etc/usine-gateway/egress-allowlist), and optionally ANTHROPIC_API_KEY or USINE_GATEWAY_ANTHROPIC_SUBSCRIPTION_TOKEN (not both) and OPENAI_API_KEY. It logs one JSON object per line to journald (node/main.ts:7). ADR 0027 explains why usine has its own proxy instead of a TLS-intercepting one.

Credential flow

Section titled “Credential flow”

Only the orchestrator has the GitHub App’s private key, which it reads from the environment variable USINE_GITHUB_PRIVATE_KEY or from .usine/github-app.json. For each Attempt, CredentialProvider.issue (orchestrator/src/credentials.ts:79) calls GitHubApp.workerToken (github-app.ts:294), which sends POST /app/installations/:id/access_tokens with one repository id and receives a token that expires after at most one hour. renewableCredential issues a new token when less than five minutes remain, and checks every 30 seconds (renewable-credential.ts:11, renewable-credential.ts:69). The next steps depend on the Worker kind:

sequenceDiagram
  participant O as Orchestrator
  participant D as Temporary directory
  participant C as git or gh child
  participant G as GitHub
  O->>G: create a one-repository token
  G-->>O: token, valid at most one hour
  O->>D: write token, hosts.yml, config.yml
  C->>D: read the token through the credential helper or GH_CONFIG_DIR
  C->>G: HTTPS with the token
  loop every 30 seconds
    O->>G: create a new token when less than 5 minutes remain
    O->>D: rename the new files into place
  end
  Note over O,D: the directory is deleted when the Attempt's scope closes

The orchestrator reads the Gateway’s admin token once at startup, from the environment variable named by admin_token_env. If the variable is unset, startup stops with a gateway capability error (orchestrator/src/ssh.ts:27). The orchestrator doesn’t open the SSH forward to the admin API itself (#249).

Harness logins on the Worker

Section titled “Harness logins on the Worker”

The Claude Code and Codex logins belong to the operator, who creates them on the Worker as the usine user with claude auth login and codex login --device-auth. Behind a Gateway, the logins call their providers through the forward proxy, so the allowlist includes their hosts:

infra/hetzner/gateway/egress-allowlist
# Claude Code with a Claude subscription: the API, and the OAuth token
# endpoint that claude auth login and token refreshes call.
api.anthropic.com
platform.claude.com


# Codex with a ChatGPT plan: the ChatGPT API, and the device-code login and
# token refreshes.
chatgpt.com
auth.openai.com

With llm_providers: [anthropic], Claude Code children call llm.<domain>/anthropic instead, and the Gateway sends its own key. The Gateway’s use of a Claude subscription token for this isn’t verified against Anthropic (#234), and the Gateway has no ChatGPT login (#235). ADR 0027, amendment of 2026-10-15, explains why the logins stay on the Worker.

cloud-init files

Section titled “cloud-init files”
  • Directoryinfra/hetzner/
    • cloud-init.yaml the Worker machine
    • create-worker.sh
    • Directorygateway/
      • egress-allowlist
      • gateway-cloud-init.yaml the Gateway VM
      • gateway-nftables.conf
      • gateway.sh
      • worker-cloud-init.yaml added for a Worker behind the Gateway
      • worker-nftables.conf
File Adds
infra/hetzner/cloud-init.yaml Debian 13 with the usine user, key-only SSH (AllowUsers usine). Packages build-essential, bubblewrap for Codex’s sandbox (#269), ca-certificates, chromium, curl, git, gnupg, jq. A 2 GB swap file and vm.overcommit_memory = 1 for the 6 GiB virtual buffer of the lint gates (#270). /etc/environment gets DISABLE_AUTOUPDATER=1 and PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH=/usr/bin/chromium. A setup script installs gh, Node.js 24, pnpm 11.20.0, Claude Code 2.1.296, Codex 0.153.4, and creates /home/usine/.usine.
infra/hetzner/gateway/worker-cloud-init.yaml Appended for a Worker behind the Gateway: the Gateway as the only resolver, apt through the forward proxy, Worker nftables (gateway only), /etc/environment with GH_HOST, the proxy variables, NO_PROXY, and CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1, a systemd socket that forwards /run/usine-gateway/http.sock to the Gateway’s port 80, and AcceptEnv GIT_CONFIG_VALUE_0 GH_ENTERPRISE_TOKEN GH_TOKEN ANTHROPIC_AUTH_TOKEN.
infra/hetzner/gateway/gateway-cloud-init.yaml The Gateway VM: nftables, dnsmasq, the bastion sshd settings, Node.js, the usine-gateway systemd service, the secrets file.
infra/hetzner/create-worker.sh Creates a server from cloud-init.yaml, labels it usine=worker, attaches a firewall that allows SSH from one address.
infra/hetzner/gateway/gateway.sh create, delete, install <tarballs>, and worker-user-data, which renders the two Worker files as one MIME cloud-init.

The parts of worker-cloud-init.yaml that every SSH session reads:

infra/hetzner/gateway/worker-cloud-init.yaml
write_files:
  # ...
  # Read by every SSH session, the harness logins included, and by the setup script of ../cloud-init.yaml.
  - path: /etc/environment
    append: true
    content: |
      GH_HOST=git.@@DOMAIN@@
      HTTPS_PROXY=http://@@GATEWAY_IP@@:3128
      HTTP_PROXY=http://@@GATEWAY_IP@@:3128
      https_proxy=http://@@GATEWAY_IP@@:3128
      http_proxy=http://@@GATEWAY_IP@@:3128
      NO_PROXY=@@DOMAIN@@
      no_proxy=@@DOMAIN@@
      CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
  # ...
  # The executor sends each Attempt's caller token in these variables through the SSH connection, so it is in no command line.
  - path: /etc/ssh/sshd_config.d/40-usine-gateway.conf
    permissions: "0644"
    content: |
      AcceptEnv GIT_CONFIG_VALUE_0 GH_ENTERPRISE_TOKEN GH_TOKEN ANTHROPIC_AUTH_TOKEN

These proxy variables use the Gateway’s address and apply to SSH sessions such as the harness logins. An Attempt’s children also get the executor’s exported values, which override them and use the domain: http://<domain>:3128.

gateway.sh writes the Gateway’s private address into the Worker nftables file and the domain into the Worker cloud-init; the defaults are 10.42.0.2 and gw.internal (gateway.sh:19):

infra/hetzner/gateway/gateway.sh
NAME=${NAME:-usine-gateway}
LOCATION=${LOCATION:-fsn1}
SERVER_TYPE=${SERVER_TYPE:-cx23}
IMAGE=${IMAGE:-debian-13}
NETWORK_ZONE=${NETWORK_ZONE:-eu-central}
NETWORK_CIDR=${NETWORK_CIDR:-10.42.0.0/16}
SUBNET_CIDR=${SUBNET_CIDR:-10.42.0.0/24}
GATEWAY_IP=${GATEWAY_IP:-10.42.0.2}
DOMAIN=${DOMAIN:-gw.internal}
LABEL=${LABEL:-usine=gateway}
GATEWAY_PACKAGE=${GATEWAY_PACKAGE:-@usine/gateway}

A Worker behind the Gateway has no time source (#254), and you must apply infrastructure changes to running machines by hand (#264).

GuidesArchitectureReferenceGitHub

Guides

Run a factoryWrite a LoopWrite an AgentOperate RunsBounded review and fix LoopsAdd a Worker

Architecture

Architecture overviewRun lifecycleHow an Attempt runsWorkers and the GatewayStorage and configurationOperator surfacesThe dogfood instance's LoopsGaps

Reference

Overview