EmberVM Architecture · Docs

How EmberVM works: the current state and the decided future, readable end to end. Assumes Kubernetes and Linux virtualization; EmberVM terms are in the section 2 vocabulary. Claims carry four flags:

Built: current behaviour, and the default for any unflagged claim. If reality disagrees with this document, one of them is a bug to fix.
Planned: designed, not yet or only partly landed; scheduled work carries its tracking issue.
Decided direction: agreed design, not an implementation claim.
Accepted risk: an eyes-open trade, stated where it applies.
Not planned: a historical proposal that is not an active delivery promise. Its linked issue records the current decision.

model-checked means the named TLA+ spec in projects/embervm/specs/ satisfies the stated property under TLC in the build. It is not an implementation claim: adoption.tla's invariants are checked against dev's live trace by a hand-written checker, and TLC trace validation is Staged (#6415).

1. What EmberVM is

Self-hosted Firecracker orchestration: a private Lambda equivalent on metal you own. An organization sizes (or elastically bounds) a Firecracker nodepool; EmberVM provides placement, fairness, isolation, metering, and lifecycle so internal workloads get Lambda-shaped submit / scale-to-zero / warm-serve behaviour without a hosted FaaS product, without a Kubernetes object per invocation, and without etcd in the execution path.

EmberVM deploys on any Kubernetes with KVM-capable nodes, a local scratch disk per FC-labelled Kubernetes node, and an S3-compatible object store; section 11 states the platform contract. The reference deployment documentation in this monorepo covers the always-on GKE hub and a checked-in home overlay whose concrete shape and deployment caveats live in deploy/README.md.

Goals: private Lambda ergonomics (HTTP invoke, zip or image source, caps, quotas, internal chargeback); honest scheduling on finite capacity; isolation by default; the control plane off the hit path; one Helm-installed system. That system comprises an Elixir control plane, fixed-size brick pods running noded, node and edge Envoy tiers, and a Workload CRD; the op-log database and the S3-compatible store are operator-provided.

Non-goals: a hosted multi-tenant cloud; "agent platform as the product" (agents are dogfood consumers); every workload class as an equal pillar (task, serving, session are the core; stateful and composite are optional advanced classes); pretending capacity is infinite (queue depth, saturation signals, and admission control are the product surface instead).

Go protobuf sources

proto/embervm/node/v1/node.proto defines the control-to-node wire contract. Bazel's node_go_src generates Go build inputs under its generated/ output directory using the pinned compiler and plugins; nodev1 consumes those outputs. The two checked-in files, node.pb.go and node_grpc.pb.go, are copies for ordinary Go tooling. CI copies them after parallel formatting with update_go_stubs, uses the existing branch autoformat writeback, and compares both with check_go_stubs on every head. The Elixir generator stays build-time only. Protocol and runtime behaviour are unchanged.

Why. A fresh checkout needs the Go package for root build, vet and tidy. Keeping the pinned Bazel outputs as build inputs preserves reproducibility; the drift check covers the copies used outside Bazel (#7066).

Capability matrix

Capability	Status	Limitation / tracking
Task execution	Built	Fresh VM per invocation
Zip lane	Built	Runtime base and handler shim
Sessions (bank/relight + workspace)	Built	Retention TTLs apply; no workspace size budget is planned unless the evidence gate in #5074 is met
Serving	Built	Control-plane lifecycle on misses
Stateful	Built	Node-local authoritative volume
Composite	Built	No current consumer
Node-local activator	Planned	Partly landed
Brick autoscale	Configured at rung `full`; operational validation outstanding	Scale-down acceptance remains on #6059
Memory admission	Built: reserved declared claims, no overcommit; Staged per environment (the chart default and both deployments still run `observed`)	Arming per stage after the chart deploys: #6965; live acceptance (no brick OOMKill under claude-runtime load) outstanding
Firecracker jailer	Built, Staged: disarmed everywhere until embervm-dev arms it	Dev arming, then production after a dev soak: #6965, #5255; conformance S7 gates dev promotion on it once armed
S3 archive-at-bank	Decided direction	Archive at bank commit
Transport auth CP-to-noded	Built (bearer + ingress policy)	SPIFFE mTLS is Built, default off: noded's second listener (`noded.spiffe`, #5757) and the control plane's file-based dial (`controlPlane.spiffe`, #5758) both render inert until flipped; phase 2 of #5706 in flight
Guest identity (JWT-SVID)	Decided direction	Per-principal SVID delivered over vsock, phase 3 of #5706
Encryption at rest	Built	Per-principal mutable artifacts (#4691), enabled per environment by values; Account-scoped immutable rootfs chunks are Not planned (ADR 028, #4182)
Cells / multi-cell	Not planned	No cell seams exist; #3855 and #4753 closed without the unmerged PR #6069
Standalone packaging	Built	Quickstart merged (#3858, PR #6070); not yet validated on a clean KVM host
Website snapshotter (task guest)	Built	Headless Chromium screenshot over MCP (ADR embervm/035), #4994

Why. The original single-node RPC path had no durable task record, ownership, managed retry, cross-node placement, or fairness (ADR embervm/001). Argo Workflows was rejected because per-job Kubernetes objects move high-churn state into etcd and add pod startup to every short task; a bare pull queue was rejected because it discards placement and snapshot locality. A BEAM control plane over a Firecracker data plane was chosen with an accepted Elixir toolchain cost and a smaller contributor ecosystem (ADR embervm/001).

Hard multi-tenancy (virtual control planes) is deferred pending real demand. Internal rung IDs R0-R9 map onto the capabilities above.

2. System overview

Vocabulary

<details> <summary><b>EmberVM terms</b>, used throughout without redefinition</summary>

Kubernetes node: a KVM-capable machine.
brick: a fixed-size capacity pod scheduled onto a Kubernetes node, several per node.
noded: the Go daemon on each brick that owns Firecracker and node-local I/O.
op-log: the durable ordered journal of lifecycle and enforcement actions.
bank / relight: suspend a VM into a warm snapshot, then restore that snapshot.
primed / banked / cold / running: ready pristine, snapshotted, fresh-boot, or actively executing instance states. Cold and warm describe boot mechanics only; task isolation remains no reuse, including after snapshot restore.
generation: the volume incarnation number used to prove state provenance and pair a stateful volume with a memory snapshot.
principal: the identity boundary a workload runs as.
dial-home: noded registration and status sent to the control plane.
warmth: a reusable boot artifact, such as a base or banked snapshot, that can make the next start faster but is never correctness-critical.
bundle: the published set of artifacts needed to relight a workload.
blessing: the durable control-plane act that authorizes a volume generation.
wake grant: a bounded authorization for an activator to advance a workload wake during control-plane absence.

The diagrams use vsock (Firecracker's host-guest socket), DNAT (destination network address translation), xDS (the Envoy configuration protocol), BEAM (the Erlang virtual machine), ETS (Erlang Term Storage), and CNPG (CloudNativePG).

graph TB
    subgraph cp ["Control plane (Elixir/OTP, 'ember')"]
        API["HTTP API<br/>/v1/workloads /v1/usage"]
        DISP["Admission + placement<br/>(dispatcher, class managers)"]
        POOL["Warm-pool refill<br/>(PoolManager)"]
        XDS["Serving endpoint publication<br/>(EndpointPublisher, sole xDS writer)"]
        OPLOG[("op-log<br/>Postgres in reference deployment,<br/>SQLite-WAL in chart/dev defaults")]
        ETS[("ETS hot set<br/>rebuilt on start")]
    end

    subgraph node ["brick (pod on a Kubernetes node)"]
        NODED["noded (Go)<br/>Firecracker driver, vsock,<br/>tap/DNAT, volumes, activator"]
        ENVOY["node Envoy<br/>(serving relay)"]
        PROXY["egress proxy"]
        VM1["task / session VM<br/>vsock only, no NIC"]
        VM2["serving VM<br/>tap NIC"]
        VOL[("vol.img on node NVMe<br/>stateful, authoritative")]
        SCRATCH[("/var/lib/embervm/scratch<br/>warmth cache: bases,<br/>banked snapshots")]
    end

    S3[("S3-compatible object store<br/>artifact export / archive")]
    K8S["Kubernetes<br/>Workload CRD (definitions),<br/>brick pods, PriorityClasses"]

    CALLER["task/session caller"] --> API
    API --> DISP -->|gRPC| NODED
    NODED -->|vsock| VM1
    EDGE["Gateway API HTTPRoute"] --> ENVOY
    XDS --> ENVOY
    ENVOY -->|DNAT into tap| VM2
    VM1 --> PROXY
    VM2 --> PROXY
    NODED ==>|dial-home NodeStatus:<br/>authoritative state| DISP
    NODED <-->|Export/Restore/Evict| S3
    NODED --- VOL & SCRATCH
    K8S -.->|watch definitions| API

Diagram edges: solid = request path; thick = facts; dotted = definitions. The edge Envoy tier is the Gateway API ingress in front of node Envoy.

Division of labour (each responsibility placed where it is cheapest to make correct):

Component	Owns	Never does
Control plane (Elixir)	Coordination facts: queue, placement, fairness, backpressure, lifecycle decisions, xDS publication, op-log appends	Carry steady-state payloads; sit on the serving hit path
noded (Go, one per brick)	Payload and OS work: Firecracker process supervision, vsock relay, tap/DNAT, volume attach, snapshot byte movement, node-local wake	Make fleet-wide decisions; join the BEAM cluster (it speaks a narrow gRPC surface instead)
Envoy (edge + node tier)	The serving request path, health-based ejection, rate limits	Lifecycle
Kubernetes	Low-churn `Workload` definitions, brick pod scheduling, PriorityClasses	Per-invocation objects, execution state
Object store (S3-compatible)	Artifact export/archive, off-node durability	Anything on a hot path (read only on deliberate restore or local miss)

Bricks dial home: on start and on a jittered interval each noded POSTs its identity {node, pod_uid, address, boot_id} to /v1/nodes/register; the control plane adopts it keyed by (node, pod_uid) and streams WatchNode. The control plane never lists-and-watches daemon pods. Growing the fleet is labelling a node (homelab.io/firecracker=true), not a values edit.

Decided direction (#4696): enrollment consolidates on a single chart-configurable key, embervm.jomcgi.dev/node, used as both the enrollment label and the key of the optional hard taint; the separate serving label is retired, since every enrolled node carries the serving relay.

How one invocation works

A serving hit (the steady state), where the control plane is not involved at all:

sequenceDiagram
    participant C as Caller
    participant E as node Envoy
    participant VM as Serving VM
    C->>E: edge HTTPRoute
    E->>VM: kernel DNAT into tap NIC
    VM-->>C: response

A wake (a serving or stateful miss) is normally coordinated by the control plane, which restores the VM and republishes the endpoint through its EndpointPublisher:

sequenceDiagram
    participant C as Caller
    participant E as node Envoy
    participant A as Fallback activator
    participant CP as Control plane
    participant VM as Guest VM
    C->>E: request (workload scaled to zero)
    E->>A: fallback endpoint, request parks
    A->>CP: wake request
    CP->>VM: single-flighted restore or cold boot
    CP-->>E: real endpoint published via xDS
    E->>VM: parked bytes splice through
    VM-->>C: response

The node-local activator (Planned, partially landed) runs the same wake node-side during control-plane absence: parked bytes splice via stable node-local DNAT with no mid-wake xDS republish.

A task (the all-miss case by policy), a fresh VM per invocation, async by default:

sequenceDiagram
    participant C as Caller
    participant CP as Control plane
    participant N as noded
    participant VM as Guest VM
    C->>CP: task POST /v1/workloads/:name/tasks
    CP->>CP: admission + quota, op-log append
    CP-->>C: 202 + task_id (?wait=true parks instead)
    CP->>N: gRPC assign (primed VM)
    N->>VM: payload over vsock
    VM-->>N: result
    N-->>CP: result, VM destroyed after one task

The platform contract is in section 11; the security posture summary is in section 10.

Why. Per-dispatch metering and placement made control-plane work grow with the number of sandboxes, while session endpoints in xDS would make configuration grow with session count (ADR embervm/020). Keeping central filter, score, and bind placement at dispatch rate was rejected as a scheduler throughput ceiling, and a global session directory was rejected because it adds a lookup to every request. The control plane now owns fleet facts and precomputed assignment while bricks own runtime facts, accepting reconcile-time metering gaps and bounded stale-state risk during partitions (ADR embervm/014, ADR embervm/018, ADR embervm/020).

3. Workload classes

Definitions are Kubernetes Workload CRs (schema-validated, GitOps-synced); kubectl get workloads reads back snapshotRef, observedGeneration, and readiness. Task invocation is POST /v1/workloads/:name/tasks. Session invocation is POST /v1/workloads/:name/sessions, followed by POST /v1/sessions/:id/invoke. source is a oneOf ladder: image (bring an OCI image, no SDK; contract is "listen on the declared port, answer a health path") and zip (runtime base + Lambda-compatible handler(event, context) shim; archives are fetched and unpacked inside the disposable guest).

Class	Semantics	Network	State	First consumer
task	Fresh VM per invocation from a pristine base, destroyed after one task. Dispatch is assignment-only from a primed pool	vsock only, no NIC	none	code sandboxes, zip functions
session	Bank/relight sandbox: idle snapshot to disk, restore on next invoke, principal-bound lineage	vsock only, no NIC	memory snapshot (+ workspace tier)	agent sandboxes
serving	Long-lived warm HTTP endpoint; Envoy routes hits, CP only on miss/wake	tap NIC	none durable	tenant web APIs, og-image
stateful	Scale-to-zero singleton datastore; L4 wake-on-connect; volume owns data, snapshot owns warmth	L4 via node Envoy	`vol.img` on node NVMe (authoritative)	demo-postgres
composite	Multi-VM group, private per-group /24, all-or-none bundle-set bank/relight; warmth only, no member volumes	per-group bridge	group snapshot set	no current consumer

Banked bundle rootfs identity

Every newly banked session, serving, stateful, and composite member bundle writes bundle.json schema v1 before publishing its snapfile completeness marker. The provenance is rootfs_identity, the canonical ext4 filesystem UUID read from the exact rootfs resource recorded in jail-resources.json. This is the identity a restored guest kernel has cached, so an OCI digest or a byte hash is not a substitute.

Compatibility is intentionally asymmetric. No bundle.json means legacy v0 and is always grandfathered, including stateful bundles and every other bankable class. A present v1 stamp is valid only when its version and identity are well formed, the persisted rootfs path is readable, and its current ext4 UUID matches. Malformed or unsupported metadata, missing v1 provenance, an unreadable rootfs, and a UUID mismatch are invalid-v1 signals. With EMBERVM_NODED_ENFORCE_BUNDLE_ROOTFS_IDENTITY=false, the daemon logs those signals and continues the pre-v1 restore path. With the gate true, it refuses an invalid v1 bundle before launching Firecracker. The chart, production, and dev values keep the gate false for the initial write-and-observe rollout.

A serving base built from an image registers its rootfs as the serving-images entry with no handler path, and a fresh boot runs the image's own entrypoint as the HTTP server; only zip-lane boots attach a handler drive (Built, ADR embervm/038, noded/server/activator.go).

Open direction:

Not planned, isolated high-throughput lane: each request gets a fresh VM, with admission dependent on reusable implementation and evidence from #6132; #3864 closed without it and records the reopen gate.
Decided direction, persistence as a declared property: workloads declare persistence flags; memory and filesystem persistence decouple from class.

Why. Per-request tenant isolation (one fresh VM per request, destroyed after the response, re-primed locally with no control-plane hop) is the fast execution path the session and task classes do not offer, and it is retained, not dropped. It is not built as a second stack. The next bounded integration must first use #6132's delivered implementation and evidence to identify reusable task-library, pool, HTTP, admission and cleanup behavior. Fresh execution state per request is the invariant; restoring many guests from one clean base satisfies it, reusing a prior tenant's mutated state never does. The closed #3865 and #3866 decisions record the integration and routing boundaries, not requirements for an independent library, pool, router or xDS path. Existing Kubernetes Service and readiness routing is assessed first, and a retry never replays tenant work after an uncertain result. The per-brick fail-closed quota lease remains withdrawn and #3867 stays closed.

Why. Classes originally bundled persistence policy, leaving no class for a filesystem-backed workload without a memory snapshot (ADR embervm/027). A new class for each persistence shape was rejected because persistence is orthogonal to class scheduling and network semantics; composite groups were rejected as the default application graph because joint lifecycle and private networking couple otherwise independent components (ADR embervm/022, ADR embervm/027). The historical mediated-bindings alternative is not an active programme: #5813 closed as not planned, while the shipped composite class remains available for multi-kernel groups.

The native EmberVM API, Workload CRD and Helm chart remain the supported interfaces. The Kubernetes Agent Sandbox adapter (#5806) and generic Kubernetes/Helm adapter (#5807) are not planned. Their separately dropped migration and binding proposals (#5808, #5809, #5810, #5811, #5812, and #5813) do not remove any shipped interface or workload.

Composite member spans expose ember.was_relight and carry ember.clock_delta_ms = -1 because no numeric delta is reported. The sentinel means unknown, never a successful clock measurement, and a relight-success boolean does not establish measured clock correctness. The numeric measurement and new alert-data bridge are not planned (#3953); historical alert placeholders are not live coverage.

4. Lifecycle

Stateful bank/relight and the interruptible bank

stateDiagram-v2
    [*] --> serving: wake (cold boot or relight)
    serving --> banking: idle window elapses, 0 active connections
    banking --> banked: atomic bank (pause, snapshot, destroy)
    banking --> checkpointed: interruptible bank opt-in:<br/>pause + snapshot to temp
    checkpointed --> serving: connection parked -> ABORT<br/>(bless gen+1, delete temp, resume: HOT)
    checkpointed --> banked: nobody waiting -> COMMIT<br/>(publish bundle, destroy)
    banked --> serving: wake -> relight<br/>(only if bundle_generation == volume_generation)
    banked --> serving: pairing mismatch -> cold boot from volume<br/>(slower, never incorrect)

This internal stateful-bank checkpoint is not a user-facing session checkpoint, fork or rollback API. That programme is not planned (#5700). The separately approved exact-turn interrupt relay and stop control remains bounded work under #4321; it does not create a snapshot or rollback capability.

Why. Duplicate Stop for the same dispatch replays the successful 202 outcome after turn completion on every runtime, until a successor starts. A double click must not become a stale-dispatch failure just because one runtime exits faster.

Facts that make this safe:

Bank only starts at zero active connections; a long-lived connection is never severed by an idle bank (rolls are the exception, bounded by the drain contract: a graceful roll gets the platform's own 110 s budget, a GCE Spot preemption about 30 s, two numbers never to be conflated).
Resume requires an exact (memory snapshot, volume generation) pair; mismatch discards warmth and cold-boots from the durable volume. Model-checked (bank_relight.tla), including the monotonic floor: a lagging node report can never regress the stored pair key, so a stale report cannot legalise a stale snapshot.
The interruptible bank (spec.stateful.interruptibleBank: true, off by default; armed in the reference deployment for demo-postgres only) makes steady-state wakes always hot or warm, never cold. The three cold exceptions: genuine first boot, explicit operator reset, and the max-lifetime forced roll.
Only a snapshot taken immediately before teardown may be kept. Any snapshot whose VM resumes is discarded (the abort orders bless-generation, delete-temp, resume, so a surviving temp is always refused by pairing).

Direction	Status	Tracking
On any future `pair_broken` recurrence, use the existing bounded metadata and log owners to capture the paired volume and snapshot generations together and retain control-plane and noded logs spanning the event before any wake, repair, lease change, export or data mutation	Standing watch; the 2026-09-06 trigger is unproven and no new defect is confirmed	#5858

Why. The known lagging-generation-report mechanism was fixed on 2026-07-20 PDT by PR #3770, merged at 2026-07-21 06:54:11 UTC, about six weeks before the #5856 rollout. Its monotonic pair-key floor is present in the source deployed for that rollout, so it cannot explain the later eviction. The retained observations begin after the eviction and do not establish which other trigger occurred. Recording the evidence gap and the smallest future observation is therefore the resolution, not a speculative code change.

2026-09-06 demo-postgres warm-pair evidence

The direct runtime evidence below is retained in #5858: its issue body and 2026-09-07 follow-up. The follow-up names temporary coordinator evidence files, but they are not repository artifacts. The CP and noded logs from before the first retained read at 2026-09-06 20:20:24 UTC, including the eviction window, are permanently unavailable. Times without seconds in the issue are preserved at their reported precision.

UTC	Direct evidence
2026-09-06 20:11:13	#5856 merged as `3878200fa2c7f802949a2e8e5d0149f5557aa2b3`; the issue records the Ember 0.70.6 rollout as the event context, but does not provide a rollout-start timestamp.
2026-09-06 20:15:33.743	Predecessor `stf-01M1VR6TPHM7RCDN4H14SHTVAG`, stored snapshot generation 534569, was recorded `evicted` with reason `pair_broken`.
2026-09-06 20:20:22.956	Successor `stf-01M1W624SCMHZX2386EBK1GS2F` was created and later observed healthy.
2026-09-06 20:20:24	The first of four bounded CP and noded log reads began. They contained no warning or fault match, but none covered the 20:15 eviction or earlier rollout activity.
2026-09-06 20:25:14.936	The successor was banked with no live VM. Complete diagnostics at 20:36, 20:39 and 20:48 found its volume/snapshot pair valid; #5858 does not supply the paired generation value for those diagnostics.
2026-09-07 00:30:12 and 00:30:23	Noded logged blessing-lease exhaustion, then an observe-only UNBLESSED asynchronous export warning for volume generation 534623.
2026-09-07 01:03 to 01:06	The read-only follow-up found unchanged pod identities, image IDs and restart counts, and found the banked successor's snapshot generation 534629 paired with the volume generation.

The follow-up pins the historical deployed source to 3878200fa2c7f802949a2e8e5d0149f5557aa2b3. The source reviewed for this record is 7da0236d174ceeaaa22e17807ce2aa92072f2b00 on 2026-09-23. Both contain the monotonic upsert_volume generation floor from eb0e6d7f16959440f2f9332b3592bd40c8be4444. At both pins, an invalid warm pair found by a wake causes immediate instance-bundle eviction with pair_broken; the ordinary anchored path then plans a cold boot from the existing volume. The eager sweeper instead requires three consecutive broken observations. If an anchor is confirmed gone, recovery must exist before eviction; otherwise the broken bundle is retained and the wake is refused. The eviction transition does not delete the volume. These source facts establish the transition semantics, not the historical trigger.

Three trigger candidates remain distinct:

Candidate	What the retained evidence establishes
Expected forward volume activity	Possible, but the predecessor snapshot and later volume generations were not collected together. Their difference does not establish when or why the volume advanced.
Stale or rebuilt generation facts	The specific lagging-report regression is ruled out by the deployed monotonic floor. No contemporaneous CP rebuild, raw generation report or paired generation sample survives to decide another stale or rebuilt-fact path.
A new regression	Not demonstrated. The `pair_broken` state and later valid successor pairing are compatible with normal invalid-warmth eviction as well as a defect, so the transition result alone cannot choose between them.

Generations 534623 and 534629, and the 2026-09-07 lease and export warnings, all postdate the eviction. At the pinned source those warnings describe the node-local activator's lease-exhaustion self-bump and observe-only asynchronous export path. They explain the later warnings, but do not establish causality for the earlier eviction. The last bounded observation establishes only pairing consistency at that time. It does not verify application data, prove a successful data recovery, or establish continuous stability. The older snapshot_unrestorable entry remains historical evidence of a different failure; its timing and artifacts are not supplied by #5858, and it is not evidence for this incident. The closed #4077 and #4146 reports likewise remain separate.

Generation blessing and quarantine

The control plane is the adjudicator of volume generations, with exactly three legitimate issuance shapes:

CP-issued pre-dispatch: blessed durably before a wake/attach (op-log-before-dispatch). The default.
Checkpoint-abort auto-heal: noded's resolve-timeout auto-abort self-bumps by exactly +1 on the same vm_id. A durable checkpoint_dispatched{workload, vm_id, generation} record lets a restarted CP prove the +1 was its own checkpoint and bless it instead of quarantining. Anything unproven stays quarantined; the break-glass runbook is docs/runbooks/embervm-stateful-generation-quarantine.md.
Activator self-advance (Built, Fork A): during control-plane absence the volume's anchor brick (the brick holding its authoritative volume) self-bumps the generation on wake. On CP return the fenced-writer anchor-adoption rule trusts a self-bump from the current anchor and backfills it; anything it cannot prove is quarantined. A first form of Fork B is Built: every wake grants the anchor a durable bounded blessing lease (the blessing_lease_granted op: start and lease_end generations, reloaded on boot) that pre-authorises the advance. Decided direction (Fork B remainder): lease expiry, moving toward a steady-state lease with brick-owned idle-bank.

The advance changes who may issue a generation, never who may write a volume (invariant 6).

That fence is also what makes anchor-loss recovery safe (ADR 040). When a brick is confirmed gone and the volume has an export, the control plane restores it onto a live peer and re-anchors there, writing node_id alone: the blessing remains the sole generation authority, so a stale copy on a node that later returns carries an older generation and loses its pair check rather than competing for the write. The anchor pin from standing decision 11 is unchanged for a volume that exists on exactly one disk, which is still the case the fence was built for.

Wake path and the node-local activator

A request to a scaled-to-zero workload lands on a fallback endpoint, parks, and triggers a single-flighted wake; the real endpoint is then published and bytes splice. The activator (L7 serving, L4 stateful/composite) belongs in noded rather than the CP pod so that a CP Recreate roll cannot black-hole cold wakes: stable DNAT inside noded's own netns, NodeStatus advertises the noded pod IP and activator port as the activator endpoint, EndpointPublisher renders it as the fallback, and instance ids minted node-side carry origin: ACTIVATOR for the CP to adopt and backfill on reconcile. The brick activator only wakes workloads whose nodeLocalWake is set (the CRD default is false) and answers 503 for the rest, so EndpointPublisher renders a brick's advertised activator as the fallback only for those workloads and keeps the CP activator for every other serving workload; routing a CP-only workload to the brick would 503 its first request forever and the CP would never see a miss. Planned: partially landed, soak ongoing.

Each published stateful endpoint shares its EDS assignment with a per-workload L4 activator fallback at the next Envoy priority. Serving workload clusters instead reuse one shared L7 activator endpoint, with the injected x-ember-workload header disambiguating the workload. Active TCP health checks eject unreachable live endpoints, so traffic falls through to the activator and returns to the live priority when it recovers. A control-plane StartServing call carries a deadline past noded's readiness budget (EMBERVM_NODED_BOOT_READY_TIMEOUT, 180 s in production, plus a 30 s RPC margin) and a StartStateful call one derived from the workload's own wake bound (wakeTimeoutSeconds plus margin, plus the same 30 s), as StartGroupMember already did: the grpc-elixir default of 10 s cancelled any slower boot mid-gate and the daemon reaped the healthy VM, while a deadline far past the stateful wake bound would leave the daemon booting a VM whose callers were already failed. Every stateful wake outcome carries the token of the wake that spawned it, so a worker that outlives its wake_timeout cannot complete a later wake with its VM; its exact StartStateful instance gets an immediate DESTROY retried on reconcile (orphan destroy skips those ids), and the entry stops fencing placement (with an error log) once the owning instance has been continuously absent (missing from NodeCapacity or unresolvable) for the destroying escape interval. The fence is bounded but the retry is not: an escaped entry stays pending and keeps attempting the exact-instance DESTROY every reconcile, so an owner that returns with the stale VM still alive is torn down, and orphan destroy keeps skipping the id. Serving drain-before-bank bookkeeping (the drain timer, the draining map, the bank worker's reply address) is sweeper-process state over ServingStore rows that ServingManager adoption deliberately leaves alone, so every sweep re-adopts the rows it is not tracking: a drained row gets its window re-armed and a banking row is settled from the node's report (VM live: bank again; VM gone and an unclaimed snapshot reported: adopt it; neither: terminal). A sweeper restart therefore cannot strand a healthy VM out of the fan-out. On the stateful L4 path, a connection that reaches the activator withdraws the stale resident health fact before starting the ordinary rate-limited wake, unless it is within the five-second publish cooldown or the registry's retained node inventory reports the exact VM with healthy: true and last_probe_unix_ms no older than one two-second status interval. In either guarded case, the activator treats the failover as a transient and splices to the published endpoint.

For serving, Envoy's default overprovisioning factor makes the priority shift fractional, so one unhealthy host out of two can still send some traffic to the activator. The node-local serving straggler lookup only splices to an endpoint whose probe is healthy and no older than its configured probe cadence plus the bounded probe timeout; otherwise, and only when no bank or active splice owns the entry, it withdraws and tears down that endpoint before following the ordinary bounded wake path (#5818). A bank-owned or actively spliced entry is left for its operation, and the witnessed-connect withdrawal above remains stateful-only.

Why Fork B stays unbuilt. ADR embervm/018 split brick-owned lifecycle into Fork A (node-local wake, shipped) and Fork B (brick-authoritative idle banking, generation advancement under a wake grant, checkpoint COMMIT on the node). Fork A answers the availability question that motivated the split: a request during a control-plane gap reaches a ready VM. Fork B would move generation authority off the control plane, the seam ADR embervm/011 made single-writer on purpose, and no workload has yet needed idle bank or relight through a sustained control-plane gap, nor has control-plane sweep load been measured as a problem. Fork B therefore waits for one of those two facts and starts with one bounded transition (ownership, authority, persistence, recovery stated), not the seven-phase plan in the ADR history. The one-way evidence sequence is the completed handoff model (#4701), then the current two-waker model (#4702), then runtime evidence before the separately gated retirement decision in #4013. No earlier step authorizes deletion, and the sequence does not imply Fork B. The two-waker model is projects/embervm/specs/dual_wake.tla, model-checked under TLC within its stated bounds. It establishes neither implementation conformance nor authority to retire either wake path.

Base snapshot device compatibility

Base capture writes jail-resources.json before publishing the bundle. That producer-owned file records the backing resources embedded in the Firecracker snapshot and therefore establishes whether the captured device table contains the non-root volume drive. The same file is included by recursive artifact export, restored into staging, checked against store metadata, and read again when noded rebuilds its base registry after restart.

A volume request patches and resumes a base only when this captured metadata proves that volume exists and the receiving node's WarmRestoreWithVolume safety gate is armed. The gate therefore controls both placeholder-volume capture and restore: a disarmed node cold-boots even if it hydrates or adopts a placeholder-bearing base captured by an armed sibling. A current rootfs-only base is cold-booted with the requested volume, so Firecracker is never asked to patch a nonexistent device. A bundle without jail-resources.json predates this contract and is unknown, not rootfs-only: historical bundles can have either device shape. Such a legacy bundle remains warm-restorable for volume-less work. A volume request takes the same safe cold-boot path, while the node-stable placeholder backing file remains available so a legacy placeholder-bearing bundle can still load for volume-less work. Malformed or internally conflicting metadata is refused without deleting the bundle.

The base identity does not include the device set. Publication, store export, and hydration therefore fence same-ref replacements unless both known device sets match. Hydration downloads to an isolated directory and publishes the complete bundle atomically, so a failed or incompatible download cannot partly rewrite an existing base. For legacy store markers, the downloaded bundle file is the available compatibility evidence; absent metadata remains explicitly unknown and is never guessed from the brick's current configuration.

Parked workspaces can rejoin across nodes

Cross-node rejoin (#6663) is unconditional. Drain workers synchronously export the unattached workspace through the owning instance dial, falling back to the draining pod's dial or the pre-park session dial when draining has removed its capacity fact. Export retries are bounded to three attempts.

SessionManager keeps the export fingerprint with the session's volume node, lineage and attachment epoch in memory. Parking and a rejoin success advance the epoch and clear evidence. A worker that issued prime also clears evidence unless prime returned a recognised pressure denial. Pre-prime failures retain evidence for a pressure-wait retry; terminal transitions remove both maps. A CP restart drops all evidence and restores the node pin.

When pinned placement returns capacity and the volume node has no base-ready brick for that workload, valid evidence permits another fitting brick that reports store reachability. Targets without a local lineage copy are preferred. Restore and prime use that same instance dial. Restore must echo the expected fingerprint; NotFound, a mismatch or an old daemon's empty echo fails before prime. The durable session_rejoined projection moves volume_node_id to the landed node. Departure preserves a proved parked session only while another base-ready, store-reachable target exists. Stale copies on other nodes, of live rows after a move and of terminal rows, are deleted without exporting over the current store copy. A relighting row is skipped, since its rejoin target holds the copy while volume_node_id still names the source.

noded also fences workspace exports on content lineage. Beside each lineage it records the store fingerprint the local copy descends from (written on a fingerprint-verified restore and on every successful or checksum-equal export). Store.Export refuses a workspace whose store content differs from both the local content and that recorded base with ErrStaleGeneration: the sync RPC answers skipped and the async path logs, keeps the copy and marks it unexportable so a drain does not wait on it. A copy with no recorded base predates the marker and fails open.

Why. A draining brick disappears from capacity before its channel expires, and an asynchronous ArchiveVolume ACK cannot prove the store has current content. In-memory evidence limits that proof to one unattached content epoch. The restore fingerprint guard checks it on the receiving daemon before a VM can attach the workspace. A workspace key always exports at generation 0, so the generation fence cannot order copies and a node that a lineage moved off would otherwise upload its old copy over newer content on its next drain. The lineage fence closes that, and the sweep removes the stale copy. Follow-ups are persisted evidence and noded-reported export state.

Sessions: the durability ladder

Tier	Window	Artifact	Pinning
Live	up to `maxLifetimeSeconds` (21600s in the agent lanes by values; CRD default 86400s, no platform clamp)	running VM	node-resident
Warm bank	7 days from last bank	memory snapshot in S3	CPU-vendor + base-generation
Durable workspace	7 days from last use	zstd content-addressed file set	none

Why. The historical durable-workspace proposal set a 30-day window, but PR #4319 implemented a narrower seven-day default through sessionWorkspaceTtlMs. Parked sessions expire earlier at the control-plane layer, so seven days bounds the S3 storage tier.

Resume is one interface with four verbs: cold boot; base-snapshot restore; warm (memory) restore; base + workspace hydration. The CP picks the cheapest unexpired artifact. Two different windows apply: the same banked session is resumable for its warm-bank TTL (an hour in the agent lane, a deploy value, and bounded strictly below the warm-bank GC TTL: the chart render and the CP's Workload admission both reject a bankedTtlSeconds greater than or equal to warmthS3Gc.sessionTtlMs, because a banked snapshot has no expiry hold in that GC), after which resume takes the session-expiry 410 path; the durable workspace lineage is adoptable for 7 days, so a new session generation inherits the prior workspace rather than starting blank. Decided direction: capture decouples from bank (close-triggered for no-memory-snapshot workloads). Artifact GC keeps only the newest artifact. latest + N remains the planned retention direction. The CRD exposes no filesystem retention setting: the former persistence.filesystem.retention field had no runtime reader and was removed with its chart declarations in #7053. A workspace size budget is not planned until #5074's gate identifies a supported workload whose storage growth, snapshot cost or banking delay is not adequately controlled by existing limits. That gate does not authorize a cap, warning, bank refusal, retention change or data deletion.

Repo-backed sessions hydrate their workspace with a direct HTTPS clone through the egress lane. The node-local git mirror sidecar (gitMirror) is Built but off in every deployment and has never served a hydration: the shim's mirror attempt is denied at the egress proxy and falls through to the direct clone.

Why. Repo-backed Claude, Codex and Pi workers use their native AGENTS.md loaders from the hydrated checkout. Root and ancestor guidance loads at startup; workers read scoped guidance when their assigned work enters that subtree. Pi keeps context loading enabled alongside its factory system prompt, while extension, skill and prompt-template discovery stay disabled. This keeps shared repository guidance versioned with the checkout. Factory prompts and pinned contracts still own roles, scope, tool limits and sequencing. The shim does not copy instructions into CLAUDE.md or inject a second copy into the prompt. (see: /projects/embervm/runtimes/claude/shim.py, /AGENTS.md)

Why. The Claude runtime ships a stdlib ci-log helper (#6442) so factory workers share commit-status lookup, bounded log traversal and redaction. It uses existing GitHub access and the credential-free BuildBuddy log RPC; guests receive no new CI credentials. Fresh exact-commit status determines running versus terminal, while chunk coverage and test counts remain separate evidence. Private bounded caches reuse only redacted terminal logs. Image publication does not establish guest uptake or measured reductions in repeated CI calls; the next retro window supplies that operational acceptance.

Banking always writes a Full snapshot. The dormant dirty-page diff path and its merge-at-bank reflink helper were removed in 2026-11: its prerequisite, XFS reflink scratch (#5699), was closed as not planned.

maxLifetimeSeconds is a version-convergence bound, not a data lifetime: it stops a session riding a stale base forever and is never raised to buy continuity. Continuity comes from adoption: lineage is decoupled from session generation, so a later generation inherits the prior lineage's workspace.

Parked sessions count as disk, not against concurrency.cap: a parked session holds zero RAM, concurrency.cap bounds running VMs only, and wake does not re-check it; placement's memory admission and the per-principal wake-rate limit protect the receiving node.

The S3 artifact GC uses an 8-hour TTL for stateful warmth and 7-day TTLs for session memory, serving snapshots, session workspaces, and group sets. Its recheck-then-delete of a workspace prefix and a restore_lineage create of the same terminal lineage are mutually exclusive (Built, Embervm.LineageFence, #6736): the GC claims the lineage before its recheck and releases it after the delete, the SessionManager claims it before placing the restoring create and releases it once the heir row exists, each claim one atomic ETS insert, so a restore admitted inside the window is denied with the retryable lineage_restore_in_flight reason and a restore in flight holds the prefix for that sweep. The destructive GC gate stays off by default regardless.

Historical cessation evidence. The management-authenticated, owner-scoped GET /v1/sessions/:id/cessation-evidence reads an exact original dispatch after its session projection has been pruned. It accepts only a retained original creation, one invocation start, positive idled parking/parked commits, and idle_ttl eviction. Guest, lineage, principal, workload, original node/VM, generation and timestamps must agree. Creation must be above the compaction marker; complete bounded creation history must contain no successor. Unknown transitions, missing fields, malformed payloads and exhausted bounds refuse. The limits are 256 target events/1 MiB, 32,768 creation events/16 MiB, and 64 KiB per payload. Neither raw payloads nor credential hashes leave the verifier.

The backend reads marker, history and durable ownership in one read-only snapshot. The serialized session store and manager additionally refuse all hot lineage holders, even terminal ones, registry owners, pending restore authority and unresolved residency aliases. A surviving terminal holder could otherwise authorize a later restore. Successful reads carry a fresh observation timestamp and a stable hash of the exact positive history, independent of unrelated new sessions. They neither recreate a session nor determine its invocation result.

Why. Seven-day projection pruning can remove a stopped guest while its thirty-day journal still contains positive cessation evidence. A missing row alone cannot justify recovery. This proof is explicitly scoped to the deployed single-writer control plane (replicas: 1, strategy: Recreate), not an unimplemented multi-writer authority. A crashed manager's orphan Restore/Prime RPC may finish resource work, but cannot register, Assign or dispatch through its dead owner. The restart regression covers that distinction; the proof does not assert that every orphan RPC has stopped. Consumers must retain their own freshness, exact ownership, audited authorization and unknown-outcome fences.

Retained-session recovery fencing. The private factory recovery workflow uses management-authenticated POST /v1/sessions/:id/recovery-fence, restricted to the retained principal without a session-token or admin substitution. Preview binds a request key to generation zero, one exact dispatch and its interrupted turn marker, positive idled parking, and idle_ttl or node_gone eviction. Complete bounded original/creation history and fresh durable/hot ownership checks must exclude every successor, residency alias and dispatch-capable pending restore; tracked handoff exports also refuse. Missing evidence fails closed.

Apply repeats those checks under the serialized manager and compares the exact preview digest in one backend transaction. It appends session_recovery_fenced and writes only immutable sessions.recovery_fence_json; original state, timestamps, interruption and usage remain unchanged. An identical request can read back a fresh proof after a lost response. A changed key, identity, history, claim or digest cannot reuse the fence as recovery authority. Freshness and the fenced flag are excluded from the stable preview digest.

Why. A retained terminal row remains eligible for workspace restoration, so proof of prior physical cessation alone cannot safely authorize retry. Restore admission now reads the durable fence on every restoring create, including after an uncertain apply, and fails closed on unavailable storage. The field survives restart; after ordinary projection pruning, the missing durable holder refuses as unknown_lineage even if ETS still has the old row. The fence never fabricates an invocation result or recreates a missing native receipt. It shares the existing single-writer deployment assumption. Orphan non-dispatching export RPCs may finish uploading bytes, but cannot restore the fenced lineage or authorize execution; the proof makes no assertion that all resource RPCs have ceased.

Why. A control-plane restart during an interruptible bank could leave a benign generation advance indistinguishable from an unauthorized one, forcing quarantine (ADR embervm/017). A live-volume timer snapshot was rejected because blocks can be captured inconsistently, and keeping wake only in the control-plane pod was rejected because a control-plane roll could black-hole cold wakes (ADR embervm/018, ADR embervm/025). Durable checkpoint provenance and a node-local activator preserve the fail-closed generation rule, accepting a narrow quarantine window when provenance was never recorded and best-effort metering during a control-plane gap.

The control plane does not retry a session invoke, and the guest classifies its own failures. Unlike a task, a session invoke's guest error is the guest's answer, not a VM failure: run_invoke issues one Assign and returns whatever the guest said, and the router proxies that status and body verbatim. Embervm.Retry and a workload's invocation.retry are task-lane only, so nothing in EmberVM retries a session turn. That leaves the guest 422 body as the only place a transient cause can be declared, and the shim is the only layer that can tell one from a permanent one: it raises TransientTurnError and adds retryable: true to the body, and the caller owns the backoff.

Why. A guest 422 is a catch-all: the shim's except Exception maps a provider stream disconnect, a missing config file, a missing binary, and an exhausted usage limit to one status with one body shape. Over the 14 days to 2026-09-16 that was 40 turns, of which 16 were a codex config load failing, 8 a missing muse binary, 11 an unreachable model catalog, and exactly one the stream disconnect a retry clears, so a blanket retry would have doubled the loud failures rather than fixed them. That ratio is also why the transient set is limited to an ALREADY-ESTABLISHED leg dying, never a bare transport generic: a permanent egress fault (a host missing from egressTo, a dead catalog entry denying its host, a broker grant failing closed) emits "transport error" and "error sending request" on every attempt, so matching those would spend a whole ladder to arrive at the same failure. Classifying in the control plane was rejected because it would have to parse guest payloads and still could not separate these causes; retrying in the control plane was rejected because the shim has to classify first either way. The cost of NOT retrying is not one wasted boot: a 422 marks the factory attempt uncertain, and with factoryStopSupervisionEnabled false that parks the run holding a parallel slot and its cost reservation, so the cheap same-session retry replaces an expensive fresh-VM node attempt. The accepted cost is that a codex failure raised after turn/start may already have run tool calls under danger-full-access, and the retry re-delivers the same message to the same live thread, so a partially effectful turn can replay up to the ladder's bound. A header-based control-plane retry for callers outside the monolith transport closed not planned (#6185). The closing note says the monolith ladder covers the only caller that meaningfully hits transient 422s.

Why snapshot anywhere uses the between-turns safe point (#6256). A provider stream can run for twelve hours, so waiting for it makes routine rollouts take hours and cannot fit a Spot notice. Every drain edge now fences session invoke admission and interrupts the exact active dispatch using the interrupt primitive. An operator stop surface can be added on top of that primitive when there is a caller. The shim stops the provider, preserves its native transcript and partial output in the workspace, syncs the session volume, and returns terminal_reason: interrupted_for_drain. Only that ended turn permits banking; a signaling acknowledgment or elapsed timer never proves a safe snapshot. The flush window is bounded by EMBERVM_SESSION_DRAIN_FLUSH_MS (default 60000) and the remaining drain deadline less the bank budget. The shim reserves another two seconds for sync and response delivery, with a one-second minimum provider wait. An idle guest banks immediately. Deadline expiry is a failure to complete evacuation, never permission to snapshot a running provider.

The control plane journals the interrupted turn sequence and CLI identity before banking. On the next invoke after relight, on any eligible brick, it prepends a continuation hint to the caller's prompt: the previous turn was interrupted for a drain and the transcript and workspace are intact. Factory retains the pending logical turn, graph attempt and reservation, and records the interrupted prefix and its cost. It emits no escalation card or terminal Discord notice. The next physical turn resumes the guest's own state; it does not replay the original prompt. Filesystem-only sessions use the same flushed safe point for park/archive and rejoin. The memory-enabled lane uses the existing memory bank/export path.

Direction	Status	Tracking
Snapshot anywhere through exact interrupt, bounded flush, bank/export and next-turn continuation	Built; one path for SIGTERM and Spot, sharing the shim handler with user stop	#6256

A direct Registry or ETS shortcut for already-running session invokes is also not planned without evidence. Reconsider it only if the measured queue-wait or stall gate in #4015 is met; the closed broad cleanup programme is not a prerequisite.

Durable capacity waits for existing sessions

Issue #7210 adds two default-off gates: Ember's session.asyncResidency.enabled and factory's agents.sessions.asyncResidencyEnabled. Existing synchronous clients retain their current behavior. New-session capability bootstrap remains separate.

Factory owns logical work, prompts, dependencies, review, quota, costs and attempts. Ember stores demand for resident capacity using its existing op-log and Postgres/SQLite projections. The session capability authenticates POST /v1/sessions/:id/residency, paginated GETs and DELETE of an operation. POST supplies Idempotency-Key, generation, deadline, task_deadline and first_wait_at, with times in Unix milliseconds. A replay returns the original operation. A different fingerprint conflicts. Neither tokens nor prompts enter the operation record. The deadline is bounded by both task and session expiry.

queued owns no additional VM reservation. admitting references one exact brick restore claim; interests for the same session share it. ready records the current generation, VM and brick incarnation. cancelling retains an unresolved restore while dropping demand. consumed records the winning invoke handoff; execution completion still belongs to the existing dispatch record. Cancellation of consumed demand returns already_consumed. Other terminal states are cancelled, expired and failed. Cancellation never destroys a shared session.

An opted-in invoke carries X-Ember-Residency-Operation, X-Ember-Residency-Epoch and the existing exact dispatch ID. SessionStore writes the consumed operation and session_invoke_started in the same transaction before SessionAssign. Cancellation, deadline and the current resident tuple are checked at that boundary. A bank winner returns exact not_dispatched evidence under the reserved X-Ember-Invoke-Outcome header. Guest responses cannot supply that origin marker. A lost handoff response stays uncertain; a consumed operation never authorizes replay, including a crash before the RPC.

Why. A capacity wait can outlive an HTTP caller or a process. The durable pending turn retains the prompt and logical permit while the residency operation retains placement demand. Factory polls without claiming an executor, advancing a dispatch counter, preparing a receipt or starting its execution clock. It rechecks controls and shared admission before POST. Only a proven unused POST can unwind provisional receipt and retry bookkeeping. Unknown outcomes retain existing receipt recovery and cost reservations.

Admission requires a fresh brick report with reserved declared-memory admission and a positive budget. #6965 owns enabling that prerequisite. Wakes use the existing scheduler and final noded admission, charging the greater of declared and snapshot memory plus existing overhead. The finite restore cap defaults to two; queue caps default to 1024 globally and 128 per principal. A bounded scan of the capped queue rotates principals and preserves FIFO age while bypassing requests that do not fit. no_fitting_class, capacity and safe_memory_unavailable are distinct observations. Existing wake-rate limits, lineage and workload limits remain in effect. No memory request, CPU ratio or jailer setting changes here.

A lost restore keeps its exact claim without a lease timeout. Restart rebuilds claims before considering queued work. A known returned VM can be adopted from its exact inventory; an anonymous Prime with a lost response is never guessed or repeated. It stays fenced until existing exact-incarnation cessation evidence resolves it. Resident reporting or committed checkpoint plus confirmed teardown acknowledges the claim durably before it is released. The existing idle-bank lifecycle manages VMs after demand is cancelled.

Status is available on the capability-scoped operation and under /v1/nodes.async_residency: state/reason counts, oldest queued age, unknown restores, restore concurrency/limit, declared resident/in-flight MiB, restore attempts and consumed handoffs. [:embervm, :residency, :decision] emits bounded state/reason labels and wait/restore durations. Structured logs carry operation and session IDs; those IDs are excluded from metric labels. Factory emits factory.capacity_wait.transitions and factory.capacity_wait.invocation_starts and persists its wait reason/deadline on the pending turn. Existing session lifecycle telemetry supplies park/relight events. Terminal dedupe projections and their owning capability records are retained conservatively: this slice has no evidence that factory has dropped every reference, so automatic deletion of those projections is disabled. An indexed active/unresolved working set excludes settled history from periodic reconciliation, admission and claim scans; key lookup and queue sequence allocation do not scan terminal records. The existing bounded compactor continues for unrelated data. Retention changes need reference-release evidence before pruning.

Enable and rollback order for a separately authorized canary:

Publish and verify chart/image provenance and Kargo's live revision. Satisfy #6965's safe-memory prerequisites. Apply the additive factory migration and deploy readers/reconcilers with both producers disabled.
Enable Ember admission in dev with finite caps, then enable the factory producer. A first authoritative unsupported response permits legacy fallback. An ambiguous enqueue or an accepted operation never falls back. Older control planes without a typed unsupported response remain fenced until upgraded. There is no production activation in #7210 preparation.
Roll back the factory producer first, then stop new Ember admissions. Keep operation readers, cancellation and reconciliation running until durable backlog and unknown restores resolve. Replays of accepted keys still work with admission disabled. Do not roll back to a binary that cannot read the operation projection while any accepted work remains.

The canary uses the same fixed workload before and after, with more logical sessions than resident slots. Record completed work per hour and native completed turns; resident GiB-hours and resident-memory-seconds per completed turn; queue latency distributions; restore duration, failures and churn; physical denials; and declared resident plus in-flight peak memory. Include failed and unfinished work in the denominator and report external Spot/preemption events separately. Missing telemetry or no traffic is UNKNOWN. Stop on duplicate dispatch, lost lineage, workload-induced OOM, unplanned eviction or a resident budget breach. Queueing can trade responsiveness for utilization; it supplies no promised throughput or cost improvement. #6781 retains its separate measured experiment.

Cooperative active parking is a separate step

QoS, execution substrate and lifecycle are independent dimensions. Asynchronous work can be high priority and can be unsafe to park. #6256 owns the existing interrupt/flush/bank continuation; #6564 owns exact-turn Stop validation. A follow-on runtime contract should reuse session, lineage and exact-dispatch identities: request yield with a reason and deadline; receive not_safe or an acknowledgement naming a flushed checkpoint; reclaim only after actual VM cessation; resume that identity. Cancellation remains a separate operation. This issue implements durable demand for existing idle-session wakes.

The completion contract needs separate evidence at each boundary:

Boundary	Required evidence and owner
Turn settled	Runtime confirms provider output, tools and follow-ups for the exact dispatch have reconciled. A quiet socket or heartbeat is insufficient.
Checkpoint safe	Runtime acknowledges a flushed checkpoint and its session/lineage/dispatch identity, or returns `not_safe`.
Residency released	Ember confirms the exact VM has ceased and its physical accounting has transferred or released. Lowering a reservation alone reclaims no memory.
Logical run complete	Factory confirms task dependencies, final artifacts and review outcomes have settled. A completed turn may still require another turn.

#6256's dependent contract needs tests for tool/follow-up activity after provider output, unsafe yield, checkpoint acknowledgement lost around a crash, delayed VM exit, cancellation during yield, exact-identity resume and final task completion after multiple turns. A density experiment must show actual resident memory falling after cessation and fitting queued work progressing within the same resident-plus-restore budget. Queue acceptance and an idle heartbeat prove none of these completion boundaries. Guest ballooning or another way to shrink a running VM requires separately supported backend work and measurement; this slice assumes neither ballooning nor reclaim from low observed RSS.

The follow-on must preserve current CPU, RAM, disk, process and output limits. Separate active execution leases from queued/parked time, keep cumulative accounting across resumes, distinguish semantic progress from heartbeats and bound repeated park/restore churn. Missing a safe-yield deadline leads to controlled containment and an uncertain outcome, never blind replay. Numerical limits, CPU sizing (#6966), security policy, jailer activation (#6965) and the human-wait delivery-slot work (#7163/#7180) remain with their owners.

5. The invariants

These are the rules everything else rests on. Every design change is judged against them.

The hit/miss invariant. A task requires synchronous control-plane admission and assignment. A serving or stateful miss normally requires the control plane; the node-local activator (Planned, partially landed) additionally runs a bounded node-side wake during control-plane absence. Steady-state hits never touch the control plane.
Facts through the control plane, payloads never. Snapshot bytes move node-to-store via noded; serving traffic moves through Envoy; the control plane carries facts. The two deliberate exceptions are lifecycle-rate: the parked first request after a serving miss, and task dispatch/results.
No mutable VM or snapshot lineage ever crosses a principal. Memory snapshots, workspaces, volumes, and other mutable artifacts never deduplicate across principals (erasure would become a cross-tenant reference-counting problem). ADR 028 defines the narrow exception for immutable, reconstructable rootfs chunks: private chunks may deduplicate within one Account, explicitly published platform chunks may deduplicate globally, and principal-scoped mount authorization remains mandatory. The workload-class table in section 3 carries the per-class network and state boundaries.
Fail closed on enforcement, fail open on warmth, and metering is not enforcement. Containment (concurrency caps, fair queues, admission, the node-side pressure predicate, node-confirmed destruction) fails closed. Warmth (a missing snapshot, an unreachable store) fails open to a cold boot: slower, never incorrect. Metering is counting, not enforcement, and fails open by design: a brick out of contact keeps running and keeps counting, and unreconciled spend is written off. Cutting off a principal is an admission action (stop minting tokens, 402 at the edge), not a metering one. The quota gate is model-checked (quota.tla): budgets are opt-in, and for a principal with a budget both the submit gate and the dispatch gate fail closed when the usage cache is unreadable.
The node agent is authoritative for instance runtime state. What a brick reports over dial-home about its VMs, taps, and volumes is the truth; control-plane tables are a reconciled cache. The registration route binds that authority to the pod UID and node name claims in the brick's bound ServiceAccount token, so a brick can establish only its own stream. The one carve-out is destruction: an instance is recorded destroyed only after the owning node confirms teardown (unconditional since 2026-11; armed in production and dev from 2026-08-22, #4758), and reconciliation is fail-closed toward destruction (an unrecognised node VM is an orphan to destroy, unless it carries the origin: ACTIVATOR marker, in which case it is adopted and backfilled). Model-checked (adoption.tla) against control-plane and node crashes between any two steps: the dispatch restart wedge, forget-before-kill resurrection, and reap-would-wipe-fleet bug classes are excluded by invariant. The carve-out has its own carve-out: a destroy whose owning node has LEFT the fleet completes as destroyed with terminal_reason: node_gone, both at request time and from the reconcile sweep, so a session stuck in destroying self-heals. A node counts as gone only when it publishes no capacity facts for the session's dial AND the node registry holds no runtime instance for it, or holds only a tombstone. Health is deliberately not part of the test. An exact stop is included, and terminalizes with its stop_completion left null rather than a synthesised tuple, so a caller holding the stop precondition can see that departure produced no proof and decide for itself whether that is acceptable. The same registry-departure proof terminalizes dormant session warmth that cannot wake: a parked row follows its volume_node_id, a banked row follows its node_id, and an expired owner moves either row to evicted. A stale conflicting secondary ID cannot override that state-specific owner. The row remains non-terminal only when a healthy peer positively reports the exact workspace volume or snapshot it needs, or when a banked row has an exact snapshot reference plus an eligible peer that can restore it from reachable object storage. Those are the recovery targets the established wake paths can actually use. Archival and node cleanup remain best effort, and a transient terminal-op append failure is retried without restamping a successful terminal row, so neither can hold irrecoverable warmth indefinitely.

Why. A replaced brick can never send the confirmation the gate waits for, so before #6004 the session sat in destroying forever and held every downstream reservation bound to it, which is a worse failure than completing a destroy whose VM demonstrably died with its node; departure from the registry is the only sound signal for that, because the registry ages an instance to unknown and retracts its capacity row after five seconds of silence, so any "unhealthy for longer than a threshold" test would also fire on a healthy brick with a brief status gap under a destroy that had been retrying for minutes, and record destroyed for a VM still running.
Single-writer for stateful is a physical fact, not a protocol. The storage section explains the raw sparse file, one-writable-attach rule, and generation-as-provenance detail; failover remains a deliberate operator action and the partition window is bounded by the brick silence timeout.
Durable-before-observed for the journal. Every lifecycle and enforcement action is an ordered op-log append; callers await the durable commit (one transaction per append today; group commit is a Decided direction, not built). The op-log doubles as the audit record.
Kubernetes arbitrates pods; EmberVM arbitrates VMs; the pod is the ABI. No scheduler or autoscaler code is imported and no node provisioning is rebuilt. The capacity section's three-layer table shows the levers, and a Pending brick is the scale-up signal or the fleet-full page.
Classes are reuse semantics; substrates are lanes. New execution technologies (Hyperlight, wasm) enter as lanes under existing classes, never as new classes.
Users express posture, never mechanics. The surface is a small set of declared scalars in user-facing units, each under a platform ceiling with a non-escalation rule. Priority rungs, disruption budgets, grace periods, and CPU are platform-derived; users do not select those mechanics. The scalar details are carried by the storage and control-plane sections.

Why. Lifecycle appends remain synchronous: the durable write succeeds before ETS advances or ownership is acknowledged. The unarmed async lifecycle implementation was removed under #7059, superseding embervm/014 decision 2. Worker-authoritative state and node-confirmed destruction remain unchanged. Forward-only projection guards preserve terminal and generation protections for late records and historical replay. Vocabulary-only conformance remains insufficient because a modeled action can exist in an enum and never be emitted (ADR embervm/034).

6. Control plane internals

The control plane holds sparse facts and derives dense things on demand; it never stores or witnesses anything that scales with the fleet.

State model: hot working set in ETS (rebuilt on start, healed by adoption from node reports); durable book-of-record in the op-log behind the Embervm.OpLog behaviour. The reference deployment configures Postgres on the shared CNPG cluster; the chart default and isolated dev deployment use SQLite-WAL. SQLite remains the zero-dependency single-node backend and the reference deployment's rollback path. Embervm.Application.op_log_mod/0 selects between them purely on EMBERVM_OPLOG_DSN being set, so the rendered pod spec names the runtime-selected backend (see: /projects/embervm/deploy/values.yaml, /projects/embervm/dev/deploy/values.yaml, and /projects/embervm/chart/templates/deployment.yaml). Either adapter creates its own schema on boot, so there is nothing to migrate. The dispatch path never reads the durable store.
Retention: result TTLs enforced at read time; terminal tasks pruned past 7 days; the ops journal prefix-compacted past a 30-day horizon behind a durable compacted_through_seq marker. Long-horizon audit lives in the observability stack.
Adoption: noded reports primed_vm_ids, session VMs, checkpoint-pending VMs, and banked artifacts on every NodeStatus; the dispatcher and managers reconcile on boot and every sweep. Adoption is ETS-only except the stateful bundle heal, which appends a stateful_banked op marked adopted: true. The protocols are model-checked: the specs in projects/embervm/specs/ run under TLC in the build, so a spec violation is a red build, and vocabulary.exs keeps them honest against the code's enums, though not against whether a modeled op kind is ever appended (#4756) or the gate it needs is armed (#4758). Built, direct-checker tier: the debug-gated SpecTrace (#4770) records adoption.tla-shaped events in embervm-dev (specTrace.enabled; production keeps it off), Embervm.SpecTrace.Checker evaluates nine of that spec's invariants over them with pass, fail or vacuous plus coverage, and GET /v1/conformance serves the result. The dev conformance runner (projects/embervm/conformance) drives task, session and destroy scenarios against dev, reads that endpoint as scenario S4, and its /verdict gates the Kargo dev-to-prod promotion on the hub. The checker re-implements the invariants by hand; it does not ask TLC whether a trace is a behaviour of the model. Why. inventory_reconciled compares the node's live_vms, its count of every live VM, with the control plane's primed pool. A session VM, or a task VM a worker has already claimed, on a node with an empty pool made it fail on every dev suite, so no chart could promote (#6422). Each checkpoint now records, per node, the VMs live for another reason: the node's own session, serving, stateful and group-member lists (each disjoint from its primed pool) plus claimed task VMs. The checker flags only the live VMs left over. It keeps live_vms rather than switching to primed_count because a node that stops reporting its pool zeroes primed_count too, which would hide the wedge the invariant exists to catch (#4838). A session's VM is not brief: it stays a primed task VM on the node until the session's first operation adopts it, and a session destroyed unused never leaves the task registry. The live dev lane failed S4 on every suite until checkpoints also counted, per node, the primed ids the control plane holds as durable session claims. Bare session reservations are deliberately not counted: they have no expiry, so one leaked by a failed create must still read red. Brief windows can still read red at a single checkpoint: a primed VM mid-teardown, a session VM between reservation and commit, a finished task VM the node has not yet dropped, and task VMs orphaned by a control-plane restart. Staged: the TLC tier, trace validation of adoption.tla against dev SpecTrace windows (#6415), ships default-off behind conformance.s6.enabled (adoption_trace.tla plus the S6 runner feeding /verdict, provable in CI on fixture windows with no live cluster). The runner image carries the pinned JRE and TLC under /opt/tla; turning the key on renders the S6_TLC_* env, a /tmp emptyDir and JVM-sized memory (conformance.s6 in the chart values). Live embervm-dev validation remains open on the issue. The broader ADR embervm/034 harness beyond that is not planned (#4761, #4763 closed).
Cells are not an active programme: no cell_id, workload-to-cell assignment or per-cell dial-home address exists. The implementation proposal in PR #6069 closed unmerged with #3855 and #4753. Multiple nodes or clusters alone do not require independently operated cells. This specific disposition does not close #3853 or any other scale-out item; each keeps its own evidence and decision.
Registry survives restarts: noded persists its last-synced registry to NVMe marked stale; a restarting noded with an absent CP serves warm workloads from cache. No dependency's brief absence may turn a warm node into a dead one.
Bounded node dials: every node dial carries a 3 s TCP connect timeout, and channel dials run in callers so they never block the NodeChannel process (#5124).
Staged OTLP capacity metrics: the endpoint-less default has no metric reader. When OTEL_EXPORTER_OTLP_ENDPOINT is present, the control plane adds a 60 s periodic reader backed by otel_exporter_metrics_otlp from the pinned official experimental metrics API and SDK 0.6.0 packages. Release boot fails immediately if that exporter module is missing, and the Linux release-boot smoke exercises this configured path without requiring a live collector. Production delivery and stability checks remain outstanding on #6258.
Health surface tiers (historical embervm/031): #4338 built a detector for sustained artifact-export failure streaks and warmth-GC manifest age. It was never armed. #7058 removes the detector, its HTTP route and its monolith consumer. The store trust probe, readiness and liveness remain. Why. Joe's 2026-10-28 decision removes the unused detector and corrects the claim that its latch protected health. Artifact export, warmth GC and their manifests retain their existing behavior.

Decided direction:

Op-log restructuring: payloads separate from facts and ops are time-partitioned, making principal-scoped erasure an indexed delete.
Admission-only control plane: the CP admits and mints encrypted session-routing tokens; peers redistribute under pressure and metering fails open.
Resource model: memMib is the only declared dial, CPU is derived from it, and GB-seconds is the accounting unit.

Why. The original journal retained request payloads without a bound, and a full durable volume would stop submissions because lifecycle writes fail closed (ADR embervm/002). A larger unbounded volume was rejected because it delays the same outage; central placement and metering on every dispatch were rejected as control-plane throughput ceilings (ADR embervm/002, ADR embervm/020). Projections, prefix compaction, group commit, and forecast-time assignment keep durable work off dispatch. Separating payloads from journal facts adds a second write and an orphan-reclamation path, an accepted consequence of shorter payload retention and principal-scoped erasure (ADR embervm/019).

7. Capacity, bricks, and Kubernetes

A brick is the capacity unit everywhere: a fixed-size noded Deployment pod in a T-shirt size class (2gi through 16gi; small classes pack tasks densely, the largest hold 1-2 serving/session VMs), with honest requests (memory requests at roughly 75% of the limit, moving to equal the limit wherever reserved admission is armed, CPU request-only, see below), sized roughly 4-8x the largest VM of its class, a handful per node. The daemon is budget-agnostic: it reads its ceiling from its own cgroup, so a size class is a resources block, not a code fork, and a brick's size is fixed for its lifetime.

Layer	Owner	EmberVM's lever
VM to brick slot	EmberVM control plane	per-brick admission (`mem_admittable_mib` net of in-flight claims); pack-to-empty scoring; best-fit placement across classes; retry on refusal
Brick pod to node	kube-scheduler	pod shape only
Node provisioning	Karpenter or equivalent (cloud) / none (fixed fleet)	brick count vector from the single-writer controller; a Pending brick is the signal

Why. The reserve constants are provisional and measured only once: daemonReserveMib (256 in the chart, 512 in the binary default when the chart does not inject it), vmOverheadMib (64, a cushion charged only under reserved), and the roughly 3 MiB per guest the overhead script reported on the first hub samples. Stats() is read once at release, so there is no recurring per-VM RSS series and no separation of daemon working set from guest RSS from shared page cache. That is deliberate: a sampler is cheap to write and expensive to keep honest (page-cache first-toucher misattribution, cgroup v2 no-internal-process rules, cardinality), and no sizing or incident decision has yet needed a number the existing facilities cannot supply. Recurring per-VM telemetry (#4773) is added when a named decision needs it, scoped to that decision, and never changes the reserve, admission or banking policy in the same change. The B5 shadow ledger (#6056) is the next measurement that exists by design.

On a fixed fleet a Pending brick is the fleet-full signal: the controller flags :fleet_full, the dispatcher refuses placement (503), and a human is paged, rather than overcommitting.

Admission is reserved (#4185; noded.admissionModel: reserved). Staged: the chart default and both deployments still run observed until #6965 flips each stage after the chart reaches it, because a value flipped before its template deploys would run the old daemon in the new mode. A brick admits a boot only when the declared memMib of every live and in-flight VM, plus vmOverheadMib per VM, fits its budget (memory.max minus daemonReserveMib), and the observed guard also holds: cgroup headroom net of reclaimable page cache, less boots in flight, less memRejectFloorMib. noded evaluates that predicate in one function under one mutex, against an atomic ticket ledger: every boot verb holds a ticket from admission until its driver boot call returns, so concurrent boots cannot each pass against the same headroom. BuildBase holds its ticket for the whole build, because a build guest never joins the live map, but the observed guard stops subtracting it once the build guest boots, because the cgroup headroom then charges that guest. A restore (a Prime, or a session, serving, stateful or group-member relight) is charged the larger of its declared memMib and the size recorded with the snapshot it restores, because the live map counts the restored VM at the snapshot's size. A refusal is RESOURCE_EXHAUSTED with a stable token (pressure:mem, pressure:taps, slots), logged and counted in NodeStatus.admission_refusals. Every NodeStatus carries mem_admittable_mib, the largest memMib the brick would admit for one more VM at that instant, computed by the same function; it is the one memory predicate the control plane trusts.

The control plane keeps its own in-flight ledger (Embervm.Scheduler.Inflight). Every new-VM placement claims what the chosen brick will charge (its declared memory, plus vmOverheadMib under reserved) before the boot RPC and releases it on a refusal. After a successful boot it keeps the claim until the brick's next report under reserved, which charges the landed VM, and for a 5 second hold under observed, which does not charge a guest until it faults its memory in. Scheduler.mem_eligible?/2 admits a need only when it fits mem_admittable_mib net of the claims that report does not count yet. noded reports separately the in-flight boots it admitted for a control-plane RPC still open (mem_inflight_placed_mib), and the control plane nets its unsettled claims against that figure only, so a node-local activator wake cannot hide one of its claims. A classed brick reporting a zero budget is not placeable. Two placers racing between reports can still pick the same brick; the brick's own admission refuses the loser, and with dispatcher.placementRetry (armed with reserved admission per stage, #6965) task, serving and session placement moves on to the next candidate instead of failing the attempt, recording demand once. A serving relight is pinned to the brick holding its snapshot, so when that brick cannot fit the guest (by its report net of pending claims, or by refusing the relight) the same wake boots cold on another brick and the snapshot stays banked; only that cold placement records demand or fails the wake.

vmOverheadMib and the jailer's ceilings are different numbers. The overhead is what admission charges each VM for host memory beyond guest RAM. The jailer's per-VM leaf caps one VM at its memMib plus 192 MiB and charges nothing; the shared embervm-vms parent caps all guests together at the brick limit minus daemonReserveMib, with memory.oom.group 0 there and 1 on each leaf, so an aggregate overrun kills one guest and the container lives. With the jailer disarmed, as in production today, the container limit is the only ceiling and the kubelet's memory.oom.group=1 on it takes noded and every guest down together.

Brick memory requests move to equal their limits with the same values change (#6965). A reserved brick fills its budget with declared memory, so requests below the limit would let kube-scheduler place more brick limit on a node than it holds, and the kubelet would evict co-located bricks together.

Why. On 2026-10-10 the hub's 8gi brick (budget 7936 MiB) ran the observed model and admitted five to seven claude-runtime guests at 3072 MiB each, 15 to 21 GiB declared. A restored guest is charged only for pages it has touched, so for minutes after a restore it costs a few hundred MiB of headroom, and creates scored on one stale report herded onto the fullest brick. As the guests warmed up the container crossed its limit and was OOMKilled whole, noded and every guest, three times in about 70 minutes. Reserved admission fits two such guests per 8gi brick and one per 4gi, so the hub's 8gi ceiling rises from 3 to 6 bricks with the flip (#6965), and demand past a brick now shows up as denials that scale bricks up. An overcommit ratio was rejected while production runs without the jailer, because without the embervm-vms cap an overcommit failure still kills the whole brick. Rollback is noded.admissionModel: observed.

Memory admission across bricks is model-checked (admission.tla): noded's reserved model admits a boot atomically against the declared memory of its live VMs, its in-flight tickets and the per-VM overhead, and the control plane places on each brick's last, possibly stale, mem_admittable_mib net of its own pending claims and retries a RESOURCE_EXHAUSTED refusal. TLC shows no brick commits or holds more than its budget as lazily restored guests fault their RAM in, every ticket is released once, every claim drains, and every request an empty brick would admit is eventually placed, with or without the frontier walk. Its negative modes re-find the 2026-10-10 OOMKills (observed admission), a check-then-act double admit, and a single terminal attempt stranding a request. The result covers the reserved configuration, which each stage arms after this chart reaches it (#6965); the specs README records what the model abstracts away.

Built: every brick and the serving relay run a disposable-shaped priority class (homelab-disposable, or homelab-preemptible on GKE, see below), so guests are the first to yield under node memory pressure; per-workload arbitration happens only in CP dispatch. QoS is Burstable: CPU carries no limit (request-only, sized as a CFS fair-share weight rather than a reservation). Memory requests sat at roughly 75% of limit (PR #5519, merged 2026-09-01, ADR embervm/039) because guest RAM is demand-faulted; the chart default keeps that for the observed model, and the stages that arm reserved admission request the full limit (#6965) because it fills a brick to it. The limit itself still sets usable_mib.

Decided direction (ADR embervm/039, from the 2026-09-01 capacity incidents): overcommit blast radius is a ladder, guest (Firecracker jailer per-VM cgroup, memory.oom.group) before brick (kubelet eviction of the lowest-priority pod) before node; and unsatisfiable demand (no class fits, or class and pool headroom are both exhausted) becomes one signaled capacity condition on /health and Workload status, held with hysteresis, rather than a retry at request frequency (today only the :fleet_full 503 exists). Pressure shedding is Built (#5521): a brick sheds its least-recently-active idle guests via the existing bank paths once its observed memory crosses statefulSweeper.pressureBanking.highWaterFraction (0.85), with hysteresis back to the low-water mark (0.70) before shedding stops. Production Git values enable the lane. This is configuration evidence, not a new observation of a live rollout. homelab-preemptible (-9, platform chart) replaces homelab-disposable (-1000) on GKE bricks specifically so a Pending brick stays visible to the cluster autoscaler's scale-up (above its -10 expendable-pods cutoff) while remaining first-evicted; this reopens, and only partially answers, the mass-VM-death preemption concern ADR embervm/016 raised against below-default brick priority (scheduler preemption deletes rather than drains). The Firecracker jailer that backs the guest rung of the ladder is Built and staged (#5255; section 10 has the containment detail): embervm-dev arms it with a values-only change once this chart reaches dev, and production after a dev soak (#6965).

Why. Host memory usage cannot say whether a guest is reclaiming hard or has already killed its application: a guest OOM leaves the Firecracker process alive, a host cgroup OOM kills the VMM. Exit classification is shipped as observation only (#5805), with four reasons: positive per-VM cgroup OOM evidence first (host_cgroup_oom), observed VMM liveness plus a SIGKILL outcome for host_requested, a panic marker in the bounded serial tail for guest_kernel_panic, otherwise unclassified. Guest feedback is Shipped-but-suspended behind default-off EMBERVM_NODED_GUEST_MEMORY_FEEDBACK_ENABLED and noded.guestMemoryFeedback.enabled. After a rootfs rebake, k3s guests serve the additive memory_status command with available memory, supported PSI and OOM evidence; other runtimes report unsupported. The host handle owns each activation and every request carries a fresh host nonce. A bank cancels and joins its in-flight request before pausing. Restore starts pending with a new counter baseline and discards pre-restore observations. Only a positive within-activation oom_kill delta is guest_oom; evidence becomes stale after three poll intervals. Missing or stale data is neither healthy nor proof of OOM. Host-observed capacity stays authoritative: a guest report can never raise capacity, bypass a quota, or relax idle-only banking, in-flight protection, backoff or hysteresis (#5521). This change adds no admission or banking input. Any future automatic action needs its own flag and issue, gated separately from telemetry rollout. The control plane reports fixed-cardinality instance diagnostics and logs positive counter deltas. Exit and guest OOM counters are cumulative since daemon start; new instances, boots and backwards counters establish fresh baselines. The daemon guest OOM total saturates at the uint64 limit. A listener unavailable during guest boot is retried; only an explicit unsupported-command response suppresses further probes for that activation.

Decided direction: PriorityClass ranking of brick pools by lane with sacrificial balloon bricks for burst headroom, and remaining-node-lifetime as a placement input so a terminating node takes only work that fits its horizon. Disruption splits workloads into preemptible posture (task, isolated lane) and durable posture (session, stateful: continuity via banked-state durability, not node pinning).

Brick discovery and scale: bricks are the only source of node capacity, so the control plane picks the brick mix from placement demand rather than running one fixed daemon per node; discovery is dial-home, never a Service or a per-node DaemonSet. Desired per-class counts are a values knob reconciled by Embervm.BrickController through /scale, because ArgoCD ignores /spec/replicas fleet-wide and git-declared replicas would not sync; instance identity is the kubelet pod UID. Brick autoscale is configured at rung full on the observe -> up -> full ladder: denial-driven scale-up acts, clamped to the chart's maxReplicas, and drain-aware scale-down may act after its idle and fully-exported victim rails pass. Because placement is best-fit across classes, a denial scales the smallest class whose empty brick would admit the need and that still has replica headroom (never one at an operative ceiling of 0), and demand for which every fitting class is at its bound latches the ceiling_exhausted capacity condition with or without a ceilingBound. This is staged repository configuration, not proof of a rollout or live behavior. Post-rollout operational acceptance remains outstanding on #6059. The current brick mix is deployment state and lives in the fleet section.

Why. Workspace archives acknowledge durable export through each node's session_volumes[].exported fact. ArchiveVolume returns after enqueueing the transfer, so its RPC acknowledgement cannot authorize scale-down. The optional bricks.autoscale.archiveAckGate rail fails closed on a missing or false flag and requests archives before choosing a victim. A fact whose workspace scan is incomplete (session_volumes_complete false) is likewise never a safe victim and requests no archive, since an empty list from a failed scan must never read as safe. The volume inventory is node-shared but the exported and attached flags are per brick process, so each request is addressed to the victim instance (never the node name or first-in-table owner) and its ledger is keyed {instance_id, lineage_id}; only the victim's own export flips the victim's fact. The request ignores the workload persistence flag, like workspace retirement, so a disarmed flag cannot strand a victim behind a silent success. A blocked victim stays alive; timeoutMs: 180000 bounds the quiet wait to the drain_node default deadline, then logs an error once (Logger.error) and keeps the archive_pending reason on the controller's decision span every tick. It never forces removal. The [:embervm, :brick, :archive_ack_timeout] and [:embervm, :session, :archive_result] telemetry events are emitted but no handler or metrics exporter consumes them, so they are not an exported signal: the one-time error log and the span reason are the observable ones. The timeout is a reversible alarm threshold. Restarted controllers re-derive eligibility from node facts, independently of their request ledger. The gate lands default-off for staging. Enabling it in deploy values and live acceptance remain on #6533.

Why. Joe chose one brick per node on 2026-10-10 (#6533). The single switch bricks.autoscale.archiveAckGate.enabled also renders required pod anti-affinity on kubernetes.io/hostname across every class and node floor. Its release-scoped selector includes noded-brick and the interim noded DaemonSet: both pod shapes mount the same host scratch and derive the same node-shared VolumeRoot. No size-class or floor label narrows that selector. The existing floor nodeSelector remains intact. Kubernetes' inter-pod affinity filter counts terminating pods still bound to the node, so with maxSurge: 0 a replacement waits until the old pod leaves. Required anti-affinity is non-retroactive: already co-located pods remain until rolled or removed.

The controller therefore holds the whole candidate node whenever another registered instance has the same node ID and a different Brick.dial_id, even if the victim reports every volume exported. Missing node identity, unavailable registry coverage or an absent victim registration also holds. facts_fun reads only dispatchable NodeCapacity rows; the gate additionally reads every node_runtime entry through NodeRegistry.status, including draining, starting, unknown and not-yet-reporting instances. A held candidate gets no archive request, stays archive_pending under the keep-and-alarm timeout and ranks behind an archivable single-instance candidate. When its sibling deregisters, normal per-process export acknowledgement resumes. The earlier per-workload sibling filter is removed because the whole-node hold covers it.

Archive candidates must also belong to the elastic class Deployment being shrunk. The pod's controlling ReplicaSet name must equal that Deployment's name plus its pod-template-hash; a same-class floor or another release's pod cannot authorize deleting an elastic replica. Missing ownership or a failed pod list withholds scale-down. The directing lookup checks ownership again before annotating. Floor facts still participate in the registered-instance node hold. When that Deployment has pods from multiple ReplicaSets, the gate holds all candidates without archive requests, under the same keep-and-alarm timeout. The directing lookup checks this again. A Deployment scale-down during a partial roll may reduce a different ReplicaSet from the approved pod's; deletion cost only chooses the victim within one ReplicaSet. Once the old ReplicaSet's pods leave, normal acknowledgement resumes. Upstream also ranks unassigned pods, non-Running phases and NotReady pods ahead of cost, so the directing lookup additionally holds unless every other owned, assigned, non-terminating pod is Running and Ready, and holds when another owned pod still carries the victim cost from a prior annotate whose scale write failed. This hold does not stop Kubernetes from performing its independently configured rollout.

Joe accepted losing brick bin-packing and the extra node spend. If the sum of class maxReplicas plus floor replicas exceeds eligible FC nodes, pods can stay Pending. A same-release DaemonSet also consumes those eligible slots. Pending pods have no dispatchable capacity row: desired replicas above registered count trigger fleet-full after the configured dwell (default 5 minutes); while flagged, denial-driven scale-up waits (fleet_full_wait). Floor Deployments remain outside the controller's scalable class pool. Capacity planning must account for floors and remove the interim DaemonSet during a staged transition. Multiple bricks per node would need a separate flock-epoch decision, outside this change.

Residual windows remain between registry/fact reads, asynchronous export and ReplicaSet deletion. An unregistered process, a registration expired while its pod still runs, or a different release sharing the host path is outside this release-scoped placement rule and registered-instance hold. An acknowledged export may become stale before deletion; there is no node-wide lock or epoch. The existing preemption drain remains the backstop for placement races. A controller restart resets the quiet timeout and reconstructs eligibility from current reports. A sibling's readiness can also flip between the victim-directing pod list and the ReplicaSet's own ranking at /scale. Live acceptance still needs placement across classes/floors, zero-surge roll completion, co-location and archive-failure hold/alarm drills, restart mid-drain, acknowledged export then shrink, cross-node export/rejoin and scale-down proceeding after a Spot preemption leaves a Failed brick pod.

Catalog-derived class floors apply only to the scalable, unpinned class Deployment. Same-class nodeFloors are additive topology guarantees and are not credited against that derived count: the controller never scales their separate pinned Deployments, and a topology guarantee cannot silently become the spare capacity of the demand-driven pool. This is deliberately conservative when both exist, but keeps the nodeFloors contract orthogonal and untouched.

Built, hub dev canary staged (ADR embervm/042, Accepted 2026-09-05): the maxReplicas ceiling itself becomes denial-driven, not only the replica count clamped inside it. A class stuck at a ceiling of 0 today needs a human to raise it by hand (the values-gke.yaml edits behind #5503 and #5504); instead, a new ceilingBound values field names the operator's outer bound per class (absent still means never, preserving today's explicit-zeros contract), and sustained denial pressure for a class below its bound raises the ceiling one step, unblocking the existing scale-up decision on the next tick. Scale-to-zero is symmetric, on a longer idle window than the replica-level one, and reuses the same drain-aware victim selection. maxReplicas in git becomes a bootstrap floor read only at first render, the same ownership shift #5498 already made for replica counts. Unsatisfiable-forever demand (no class fits at any bound, or a class pinned at its bound and still denying) rides the existing decision-6 capacity signal (ADR embervm/039) with two new reason codes rather than a second alerting path. This does not reopen in-place pod resize, which ADR embervm/013 section 7 and ADR embervm/039's own alternatives already declined to reopen. Implementation is tracked on #5505.

The ownership boundary is explicit. maxReplicas is the bootstrap ceiling and the catalog-derived and manual floors still apply inside it. A positive per-class ceilingBound transfers the ongoing ceiling to the controller; a missing or zero bound keeps the old fixed-clamp behavior. Chart defaults and production leave that transfer disabled. The hub embervm-dev overlay stages only 4gi with ceilingBound: {4gi: 1}, bootstrap maxReplicas: {4gi: 0} and mode full. Its 2gi min and max both remain 1 to preserve the conformance brick. Only needs above the 2gi admission capacity and fitting 4gi attribute denials to the canary; small-workload saturation on 2gi does not open it. Live acceptance remains pending on #5505. On restart the controller reconstructs a raised operative ceiling from the live Deployment replica count, never from an unreadable scale response. Pinned nodeFloors remain separate Deployments and are neither counted as scalable replicas nor consumed as ceiling headroom.

Sustained workload-attributed denials raise an authorized operative ceiling by one. Replica growth is a separate decision on a later reconciliation tick. A raised ceiling retires by one only after the scalable Deployment has remained at zero replicas for bricks.autoscale.ceilingIdleMs (default 3,600,000 ms); any denial or nonzero replica count resets that boundary. Replica scale-down therefore completes first through the existing draining, live-VM, exported-warmth, pod-resolution, and deletion-cost rails. Capacity conditions are exposed in /v1/capacity and the Workload Capacity condition, while /healthz remains process readiness and never changes because of capacity.

The availability contract is spot semantics with two budgets. The chart default gives a routine roll, upgrade, or scale-down 110 seconds of drain notice. A GCE Spot preemption supplies about 20 seconds through noded's preemption-notice watcher (drain.preemptionNoticeEnabled, armed on the hub since 2026-09-06, #5819, off in the chart default). Both edges publish the same drain_deadline_unix_ms signal. Sessions fence admission, interrupt the active turn, give the guest a bounded window to flush progress and state to the workspace, then bank and export at the existing between-turns safe point. They no longer wait for the twelve-hour invoke bound. New and queued invokes receive retryable drain responses until bank completes. A failed bank retains the fence. Relight resumes as the next turn on the same factory attempt and reservation. The deadline still bounds evacuation: node loss, a failed flush, or an export that cannot finish before it can lose progress since the last durable archive. The GKE hub overlay sets a 150 second drain (180 second termination grace) so the budget covers flush plus bank rather than a whole factory turn; the brick progress deadline stays the chart default because a cold rootfs rebuild still needs it. State durability within the stated archive interval is the guarantee, connection continuity is not. Artifact retention TTLs and the GC sweep behaviour are in deploy/README.md.

An ordinary brick rollout closes placement at the node drain edge and closes session invoke admission before evacuation. Closing new admission is not a drain by itself: guests that were already running remain on the old brick. A session therefore rejects queued and later invokes retryably, lets its one current invoke finish, and asks to bank immediately afterward. Temporary bank admission failures keep the fence closed and retry; a failed bank RPC restores the fence when the session process resumes. The old pod remains terminating until its guests leave the live registry or the externally configured deadline expires. This means one long factory turn can delay its GKE brick replacement for up to the configured 43,800 second bound. Sequential sync waves can compound that delay across a full fleet rollout. The Deployment progress deadline is longer than one termination grace plus a cold rootfs build. Promotion tooling with a shorter wait budget can still report a timeout while Kubernetes is gracefully terminating the old brick; recovery must resume observation and must not force-delete that pod.

That protection is intentionally limited. Spot preemption, force deletion, node loss, and expiration of the drain deadline can still destroy a running guest. Temporary control-plane or bank unavailability is not evidence that a guest stopped, so it never authorizes a replacement attempt. When EmberVM has durably recorded failed with terminal reason brick_gone, factory supervision may settle only the exact matching guest invocation and use only the graph attempt's remaining bounded retry. Guest, generation, invoke stamp, or node mismatches remain uncertain, as do outcomes with ambiguous external effects. No retry is admitted from absence or temporary unavailability alone.

Why drains bank in place rather than hand over. A drain edge is one signal (drain_deadline_unix_ms, SIGTERM or the Spot preemption notice, whichever is earlier) and the coordinator answers it one way: force-bank every class in durability order inside the deadline. A pod-surge roll keeps the node's disk, so the replacement daemon can relight locally; the bank also exports its durable artifacts before shutdown when the store is configured. A Spot preemption gives about 20 seconds, enough to bank and export but not to export, restore onto a peer, re-anchor, and evict, so pre-drain redistribution would race the deadline for no availability gain; when the node is gone, anchor-loss recovery (ADR embervm/040) restores the banked volume onto a peer from the store. The manual handover verb (POST /v1/stateful/:name/handover/:target) exists for the planned case: a node leaving the fleet on an operator's schedule. Automatic handover at the drain edge would need a drain cause the daemon does not publish and a workload whose downtime budget banking cannot meet; neither exists.

Why. Per-invocation pods and Kubernetes objects made pod churn and etcd the ceiling for short work, yet bypassing Kubernetes meant its autoscaler could no longer observe Firecracker demand (ADR embervm/001, ADR embervm/005). A pod per VM was rejected for restoring that churn, while whole-node daemons were rejected because they make capacity changes too coarse for mixed workload lanes (ADR embervm/013, ADR embervm/016). Fixed-size bricks expose schedulable capacity to Kubernetes and keep VM placement inside EmberVM, accepting bin-fragmentation and drain coordination as explicit costs.

8. Storage, artifacts, durability

Artifact model: one typed verb family, ExportArtifact / RestoreArtifact / EvictArtifact, over ArtifactRef {kind: BASE | SESSION | SERVING | STATEFUL | GROUP_SET | VOLUME}. Control-plane-driven, idempotent per key; evict refuses while referenced, and base retention holds when the current base is unverified (#4401). Keys are namespaced by workload (and vendor, below); a principal-scoped shared/<principal>/<sha256> keyspace is Planned (#5075), not implemented: ArtifactRef is {kind, workload, ref} with keys <kind>/<vendor>/<workload>/<ref>/<file>.

Baked rootfs cache (Built, #5772): Baked guest rootfs files use rootfs/<architecture-digest>/size-<rootfs-size>/format-<bake-format>/ as their remote namespace. The payload is <payload-sha256>.ext4, with rootfs.ext4.sha256 written last as the completeness marker. The marker records and verifies the image digest, rootfs size, bake format, payload key, and payload checksum. The local cache and the rootfs path pushed to noded carry the same size and format identity. Changing either input therefore misses once without reading or deleting legacy digest-only files or objects. A local miss reads the marker first, downloads that payload, verifies its metadata and bytes, and punches zero runs back into holes, so bases hydrate across nodes under the identity contract with a sparse local hardlink cache in front.

Rootfs construction has an opt-in bounded driver. The default rootfsBuilder.parallelEnabled: false preserves the legacy sequential per-workload init containers and performs no class filtering. When enabled, one init container launches at most rootfsBuilder.maxConcurrency independent builders (default 2), gives each builder its own extraction directory, and keeps the existing per-identity scratch lock plus the post-lock cache recheck. A child failure terminates and reaps every remaining process group before the init container fails. Classed bricks compare each workload's declared memMib with the same effective ceiling used by scheduling: nameplate memory minus the daemon reserve and rejection floor. Oversized images are logged and skipped. A missing or malformed declaration is logged and baked fail-closed. The wildcard legacy DaemonSet also bakes every image.

With rootfsBuilder.inPodBake.enabled, a long-running rootfs-baker replaces both init-bake forms. Guest digests and digest-bearing paths live in the directory-mounted rootfs-builder ConfigMap's tab-separated desired set, not in brick pod templates. The baker polls every 30 seconds and sends missing roots through the same bounded driver, class ceiling, content-addressed cache and lock. ConfigMap updates are projected by kubelet without restarting the pod. The chart defaults off; values-gke enables it. The first flip rolls bricks once; removing the enabled line restores init baking and rolls once again. The baker adds 100m CPU and 128Mi memory requests per pod (512Mi memory limit) to the scheduled container sum, unlike the old init request's max-with-sum accounting.

Why. Digest-only publishes rolled every brick and drained live sessions. The ReplicaSet history on 2026-10-01 in #6662 identifies guest digest changes as the only pod-template change in several rolls. Moving desired roots out of the template lets existing sessions continue while new images bake in place.

Noded starts before first-boot bakes finish. Before submitting each eligible missing root, the baker writes <rootfsPath>.pending and refreshes its mtime each poll while outstanding. Skipped workloads get no marker. Publication or failure removes it, and failed bakes retry on a later poll. With the flag-only EMBERVM_NODED_ROOTFS_PENDING_MAX_AGE=1800s, BuildBase returns ABORTED for a missing root with a fresh marker, without recording a failed base or changing its backoff. Missing or stale markers, unset env, and other filesystem errors retain existing behavior. Fresh serving, stateful and group boots and base restore admission use the same guard. GC and ENOSPC reclaim semantics are unchanged: .pending ends in neither .ext4 nor a reclaimable .tmp. name, and the baker never deletes a published rootfs.

Repository delivery is staged under #6662. Operational acceptance still needs a later digest-only publish to prove no brick ReplicaSet changes, new sessions use the baked base, and existing sessions survive without node_gone.

Remote baked-rootfs retention is staged separately under rootfsRemoteRetention. The repository defaults are enabled: false and ageDays: 30. The control plane uses its existing S3 client to produce an hourly dry-run plan and audit manifest. Current rendered image refs are protected without an age limit. Only non-current, identity-qualified prefixes older than 30 days are candidates; legacy layouts, malformed keys, incomplete markers, and unreadable listings are held. The 30-day horizon exceeds the existing seven-day warmth windows and provides migration grace for retained bases, but activation still requires reviewing real candidates and rollback assumptions. Disabling the gate stops future deletion; it cannot restore deleted objects.

GKE validation store (Staged, #6193): the production Application continues to use h0melab-ember-bases. This stage defines no production lifecycle deletion rule, and verifying the live production bucket has none remains an operator check. The repository carries an unreferenced, default-off dev preset for h0melab-ember-bases-dev, plus a dev-only seven-day delete lifecycle definition and an alerts-only USD 15 monthly Cloud Storage budget definition. The budget filters the whole h0melab project's Cloud Storage service, not one bucket, and alerts do not cap spending. The bucket, lifecycle, budget, notification recipient, credential, and activation remain operator work. The preset cannot become live through either checked-in Application or kustomization, and its required Secret references fail closed until a verified credential delivery path is supplied. It also disarms every application-level retention delete gate inherited from the base dev values, so validation does not introduce a second deletion path beside the reviewed bucket lifecycle.

Not planned rootfs plane (ADR 028, #4182, closed until its gate is met): OCI images convert to deterministic flattened EROFS manifests and immutable chunks. Private chunks deduplicate under rootfs/account/<account>/...; allow-listed published platform chunks may use rootfs/platform/.... A brick fully hydrates and verifies every chunk of an active manifest before reporting the rootfs READY, then presents a local-only read-only ublk device. The object store is a preparation dependency, never a live guest block-read dependency. Phase 0 (the rootfs/measure_chunks.py harness, raw runs under rootfs/phase0/, results on #4182) measured a 16.8 percent Account-scope saving with Gear CDC at 64 KiB / 256 KiB / 1 MiB and showed -Enoinline_data is required for rebuild stability; the build waits for EKS because converter placement and Account key custody are its prerequisites, and the two larger wins it uncovered (superseded-rootfs reclaim, dropping unused baked images) need no chunk store.

Failure	task	session	serving	stateful
VM process	Fresh VM is discarded; Built cold boot fallback	Lineage restores from warmth; Built	Endpoint wakes a replacement; Built	Volume remains authoritative; Built
brick	Assignment and primed pool reconcile; Built	Banked state can relight; Built	Cold wake fallback; Built	Volume is node-resident, and failover is now automatic when an export exists: the wake restores the exported volume onto a live peer and re-anchors it there, once the anchor is absent from capacity AND absent for the full registry-expiry window AND the CP has been up that long (Built, ADR 040). The manual handover RPC (`POST /v1/stateful/:name/handover/:target`) remains for moving a volume off a brick that is still alive
Kubernetes node	Recreate on another brick; Built	S3 warmth is available on deliberate restore; Built	Cold wake fallback; Built	Export at bank commit is Built and is what makes the brick row's automatic failover possible; recovery is to the last export, so a write made after it and before the node vanished is lost (ADR 040)
control plane	No new task admission; Built	Existing local state continues; Built	Hits continue; misses wait, node-local activator Planned	Existing local state continues; Built
object store	Local execution continues; Built	Local warmth continues, restore falls back cold; Built	Local warmth continues, restore falls back cold; Built	Local volume remains authoritative; archive is Decided direction

State durability within the stated archive interval is the guarantee. The matrix defines the current recovery and the gaps in automated recovery.

The control plane performs one synthetic metadata fetch through its real S3 client at boot and every five minutes by default. The current TLS result is reported on the always-on /healthz and authenticated /v1/health/store reads without changing readiness; plaintext and unconfigured stores are explicitly disabled.

Vendor pinning: Firecracker memory snapshots restore only within a CPU vendor (and a narrow intra-vendor matrix), so all warmth artifacts are keyed by (vendor, template) and never cross the boundary; the daemon refuses a vendor-mismatched restore loudly. Volume data is fully portable. Legacy artifacts cut before stamping existed are grandfathered: restorable on their home node forever, never distributed.

A base snapshot is also bound to the exact rootfs it was captured on: noded records the rootfs ext4 UUID at capture and refuses a restore, sibling adoption, or prime against a mismatched or missing identity, moving an in-use mismatched bundle aside rather than resuming a guest onto a filesystem it never saw (#5674). A rebaked rootfs after a node replacement therefore forces a fresh base build on that node unless the baked rootfs cache above serves the same bytes (#5772).

Principal artifact envelope encryption (Built, armed in dev and production, chart defaults off, #4691): each mutable principal artifact is zstd-encoded then AES-256-GCM chunk-framed with a per-file nonce; meta.json carries the opaque control-plane envelope and SHA-256 stays over plaintext. An enveloped restore needs a five-minute capability minted by the control plane for the exact target brick, HMAC-bound to the (principal, lineage, brick, workload, ref, kind, generation) tuple; legacy plaintext restores unconditionally, and a malformed or below-floor envelope suppresses the restore so the cold-boot fallback runs. Root, epoch, and custody transitions rewrite envelopes without reading payloads, by a bounded control-plane sweep and a lazy noded pass after access, both guarded by an exact-ETag compare-and-swap so a newer export always wins; raising an epoch floor or retiring a root stays a separate operator action. Arming order and flags: kekRoot, EMBERVM_ARTIFACT_ENCRYPTION, store.encrypt, requireRestoreCapability; mechanics in noded/server/store.go and control/lib/embervm/envelope_rewrap_sweeper.ex.

Decided direction:

Local disk is authoritative: local node NVMe relight needs no network.
S3 is a zstd content-addressed archive, written at bank commit, not a hot tier.
archiveInterval is the user-facing durability control in acceptable-loss units.
Failover and node rotation are deliberate operator actions; the banked-volume handover RPC that implements them is Built (POST /v1/stateful/:name/handover/:target).

Ownership arbitration is class-scoped (Decided direction):

Class	Exclusion	Two-incarnation cost	Mechanism
stateful	physical (one node, one writable attach)	cannot arise implicitly	none needed; grants are provenance
session	none exists	divergence from a common ancestor	durable relinquish record before handoff; divergence bounded, detected by generation comparison on reconnect
composite	none (warmth-only)	bundle-set divergence	inherits session rules, governed as one unit
serving / task	n/a	nothing durable	n/a

The divergence bound is the brick silence timeout: a brick that has not heard from the control plane for longer than the timeout (~6h, in the grant-expiry range, so a CP roll never trips it) stops serving everything it holds. Token TTL is a convenience, never the correctness parameter. Built (ADR 037, #5073): noded tracks last control-plane contact from its dial-home Register responses and WatchNode stream activity and refuses new node-local work (activator wakes, group wakes, blessing-lease self-advance) once silence exceeds the bound; live VMs keep running and warmth stays intact, and blessing-lease exhaustion remains the generation-count complement for quiet lineages.

Why. ADR embervm/011 chose replicated block storage to avoid whole-volume exports, then ADR embervm/025 superseded it for stateful storage because the CSI path added another placement owner and imposed the same replication cost on every workload; the reasoning against whole-file write amplification still holds. Timer-driven copies of live volumes were rejected as crash-inconsistent, and network demand-loading of root filesystems was rejected because an object store outage could stall a running guest (ADR embervm/025, ADR embervm/028). Authoritative local volumes, bank-time archives, and eager local chunk hydration accept an archive-interval loss window and slower restore after node loss. Mutable warmth is encrypted per principal because storage ACLs alone expose process memory to a compromised storage reader, accepting that loss of the platform root forces cold boot (ADR embervm/033, ADR embervm/036).

9. Identity, tenancy, security

Only principal ships now: it is the TokenReview-authenticated caller identity, and the op-log's existing tenant field is a deployment constant occupying the Account slot. Domain grouping and a reserved platform principal do not exist and are not planned (#5072). This decision does not remove or weaken existing principal authentication, authorization or artifact isolation.

Object-level authorization on the management API (Built): passing the submit allow-list says a ServiceAccount may use EmberVM, not that it may reach another principal's objects. A task (GET /v1/tasks/:id, its result, redrive, the dead-letter list) and a session (list, GET with a management token, DELETE) are addressable only by the principal that created them; another allow-listed principal receives the same 404 an unknown id gets, so the surface never confirms the object exists. The workload-level destructive verbs (serving force roll, stateful instance destroy, volume delete, handover, group force roll) have no single owner and are admin-only (403 otherwise). Admins are the values-configured usageAdmins, the platform's own ServiceAccount in the reference deployment. A session's own capability token still proves ownership on its routes. Why. Three principals share the production allow-list, including the public tier's ServiceAccount, which serves anonymous internet traffic; without object scoping a compromised public-tier pod could read the monolith's task results, destroy its sessions, or delete a stateful volume.

Embervm.KeyService is the platform key custodian (ADR embervm/036): it derives per-principal, per-epoch KEKs on demand from one current root and stores only the current and minimum accepted epochs, whose floor is the revocation fact; one previous root may coexist during rewrap. Customer-managed principals instead use a Secret-configured HTTPS KMS oracle that keeps the KEK and returns a data key plus an opaque wrapped key, so disabling the customer key or grant makes warmth unrestorable and restore-on-miss degrades to cold boot; a custody switch declares mode plus transition_from and accepts the old custody only in that window. No customer oracle is configured in the reference deployment, so that mode is available but inert. Mechanics in control/lib/embervm/key_service.ex and customer_kms.ex.

Decided direction:

Hierarchy:

Account      billing / grouping, NO isolation semantics
 └ Product   grouping, NO isolation semantics
    └ Principal   THE isolation boundary
       └ Workload

Definitions at scale: one product template plus N enrollment records, one Git CR per product expanded into CP definitions, and idempotent registration as a desired-set reconcile.

Credential handling (class taxonomy Decided direction; the brick-local proxy's egressTo-gated header injection, sidecar-only key material, and the TLS-MITM CA lane are Built in the reference deployment): material may sit where it can be stolen only if the platform can kill its validity on demand; otherwise the request moves to the credential. Secrets are classed: derivable short-lived (class 1, may enter PLATFORM-TRUSTED guest classes only, revoked at bank), fixed-but-rotatable (class 2, brick-lease only, injected at the egress hop), fixed-manual (class 3, never leaves a central key-sharded swap tier). A UNTRUSTED workload guest never receives any credential class. The brick-local proxy on every guest egress reads the plaintext request, sets the configured header to the real value (mounted only in the sidecar), and originates a fresh verified TLS connection onward. Injection fires only when the destination is in that secret's egressTo, so the credential is unreachable at every other host (projects/firecracker/substrate/egress-proxy/cmd/swap.go). Header injection is used rather than placeholder substitution because a guest-controlled placeholder can be spliced into a URL and reflect the credential into a request line. RAM scrubbing before snapshot is rejected as a mechanism to rely on; revocation at the validator is the control.

The egress CA mount remains optional so a brick can start while cert-manager issues its release-local certificate in the same ArgoCD sync. If the initial load fails, CA fetches and permitted credentialed TLS requests retry the fixed configured files at most once every five seconds. A successful load keeps that keypair for the process lifetime; this does not introduce CA rotation. Missing or invalid material yields no CA response, and destination denial still happens before credential resolution or a TLS retry.

Why. A one-shot startup load left S8 permanently unable to fetch its CA when certificate issuance finished after sidecar startup (#7187). Making the mount required had already deadlocked brick rollout. Bounded initial-load recovery lets the optional Secret arrive normally without a restart or a new egress permission, while retaining the trust anchor already fetched by running guests.

Muse 1.0.3-R2198.1 was verified on 2026-09-10 to trust only its bundled roots (1.4.1-R4503.1 keeps the same settings surface; re-confirm live). A 2026-09-30 capture run of 1.4.1-R4503.1 under the shim's settings showed every api.meta.ai request (catalog GET, turn POSTs) carrying the Bearer Authorization header the presence-keyed swap needs, with no other hosts and no telemetry traffic, so the api.meta.ai entry needs no injectAlwaysPaths. Its adapter therefore uses an http:// base URL on the sidecar's plaintext lane rather than the CA-backed HTTPS interception lane: the guest to sidecar hop is a host-local vsock, the sidecar injects the credential there, and it originates verified TLS to Meta itself, so nothing leaves the brick in the clear.

Token broker (projects/embervm/tokenbroker/, Built): the single owner of every mutable OAuth grant. A catalog credential marked with a broker grant is resolved by the egress sidecar as a short-lived access token fetched from the broker rather than from a static Secret, so a subscription refresh token (Codex ChatGPT) or an authentik service-account grant rotates in one place and never enters a guest. The grant Secret's data is broker-owned and ignoreDifferencesd in the Application.

Grant pools (Built, armed on the hub with codex-cluster and codex-b since 2026-09-10): a catalog entry may name brokerGrants, an ordered pool of grants on one provider, and the sidecar picks the active grant per connection from the broker's per-grant quota views: remaining quota in 25% bands taken from the worst unexpired window, the current grant kept until another is a full band ahead, ties to the sooner reset, an exhausted grant out of the ranking until its reset, a grant the broker cannot mint for skipped for a minute. Quota reports carry the grant they were served with, so the broker keeps a per-grant view beside the per-provider one and the monolith's model pools judge a class by its least-used open grant.

Why. The sidecar is one per brick pod, shared by every guest on it, and nothing per session reaches it: choosing an account per session would need the control plane, noded and the proto to carry it. Choosing per connection with hysteresis gets nearly the same cache locality (a brick stays on one account for long stretches and each account burns down in quarter steps) for no cross-service plumbing, and the occasional cache miss on a flip is cheaper than that plumbing. See #5974.

Why. The broker observes every pooled Codex grant with GET /wham/usage, including exhausted grants. The hub refresh age is 600 seconds; a one-minute tick starts at most one request per grant per refresh interval, with a ten-second timeout, no redirects, and a 64 KiB body limit. A manual reset can precede the reported reset by days. Waiting until that timestamp latched B at 100 percent. Usage polling reads only an existing access token with more than a minute left; it never rotates OAuth credentials or sends inference. Missing/expired tokens, authorization failures (401/403), transport failures and malformed usage retain the previous quota observation without extending its freshness. Ordinary token serving still owns OAuth refresh. An idle grant with an expired token needs that existing serving path before polling can resume.

The Codex sidecar excludes grants at 97 percent or explicit exhaustion, and never falls from a fresh usable tier into stale or exhausted quota after a token-fetch failure. The KG gate ignores exhausted accounts beside a fresh permitting account. Unknown inventory, malformed evidence, role floors and no fresh permitting account still defer. Cached sidecar observations age even while the broker is unavailable. Every sidecar carries the exclusion, so an old sidecar can no longer select an account the gate excludes.

Account preference (enabled on the hub for factory and interactive guests): grantPolicy: prefer-first on a two-member Codex catalog pool chooses its first member whenever fresh usage proves every reported window usable. The hub order is [codex-b, codex-cluster]. A fresh exhaustion report or the existing 60-second token-failure cooldown permits fallback to A. Stale, missing or invalid B telemetry pauses selection until another usage observation; it does not prove B exhausted. Before enabling that catalog policy, set SWARM_CODEX_PREFERRED_GRANT=codex-b on monolith using swarm.codexPreferredGrant: codex-b, together with the verified-sidecar compatibility setting. Preferred account admission requires current future-reset windows and refuses stale B, so it defers jobs before egress would deny them. If B's cached token has expired, quota observation needs ordinary token serving or operator recovery first; neither observation nor preference rotation initiates OAuth refresh as a probe.

A can displace usable B only when both have fresh observations, all windows are below 97 percent, all durations and future resets are known, and A's longest window resets before B's. Its remaining time must be at most min(12 hours, longest window duration * minimum remaining fraction across windows). This uses reported headroom and time without assuming account capacities or subscription weights. A short-window reset alone cannot preempt B's weekly budget. Once the condition stops holding, B wins again regardless of the previously selected account. Provider and model routing are separate.

Why. The shared sidecar receives only a destination preamble, and its broker SPIFFE identity identifies noded rather than a factory session. Enabling the policy in the current deployment would affect every Codex guest on that brick, including interactive guests. Joe accepted that shared scope on 2026-10-19, and the hub enables B-first for both factory and interactive guests after compatible sidecars and admission settings are verified. Factory-specific selection would require trusted workload metadata. Guest-supplied headers are not an account authorization boundary. No credentials, access rules or identity plumbing change here.

Why. Quota responses include advisory pacing against the longest reported window, with remaining daily budget and the reported next reset of a shorter window (#7122). A busy short window can coexist with spare weekly capacity. The broker derives this advice at read time from fresh observations; unknown durations stay unknown and passed resets require another report. The broker preserves reported reset timestamps without inferring future cycles. Observation storage, exhaustion, admission, grant selection and refresh keep their existing rules.

Agent MCP lane (Built on the hub): the guest half of the agents tier described in projects/mcp/ARCHITECTURE.md. The shim writes a strict MCP client config from a URL delivered by kernelBootArgs as an ember.env.* token (the only values-driven way a guest-read setting reaches the shim; initEnv is a base-signature input only). The sidecar's catalog entry for that destination resolves the broker's agent-mcp grant, opts into plaintextUpstream because the tier is an in-cluster plaintext listener, injects on injectAlwaysPaths /mcp and /mcp/ because the codex client sends no Authorization header, and serves one request per connection because the tier's 5 second keep-alive otherwise killed the first call after idle. The upstream therefore sees the broker's identity rather than the guest's, the gap phase 4 of the SPIFFE program below closes.

Public surface hardening (Built): the public routes are scoped at their HTTPRoutes, with serving routes additionally constrained by node Envoy authority matches. Node Envoy returns 404 for /shim/* before routing to a serving workload, so the workload handler never sees the reserved prefix. Inside the guest, hydrate is build-only and the shim refuses another hydrate once it is ready. The public Bazel-query demo exercised this pattern with server-controlled argv, zero egress and a disposable clone per query. Its exhibit retired in #6937; its unused guest image, Workload and build dependencies retired in #7056.

Why. No deployed stack or caller used the Bazel demo after the exhibit was removed. Building and publishing its runtime no longer served a workload.

Guest identity: a guest never holds a cluster credential (Built, by construction: no NIC, no mounted ServiceAccount); the platform holds real credentials and acts on the guest's behalf through the brokered egress path.

Decided direction (ADR embervm/041, #5706): one self-hosted SPIRE trust domain, embervm.jomcgi.dev, issues every EmberVM identity. Platform components (control plane, noded, egress proxy, token broker) get X.509-SVIDs registered as ClusterSPIFFEIDs, and the CP-to-noded hop moves to symmetric mTLS with credentials delivered as files by a spiffe-helper sidecar and read at dial time, on second listeners beside the plaintext bearer ones so each side flips independently before the bearer retires. Guests get per-principal JWT-SVIDs (spiffe://embervm.jomcgi.dev/ember/principal/<p>, 5 minute TTL) fetched by noded as the node agent's delegate and delivered over vsock at boot and every relight, never written to scratch or into a snapshot; the egress proxy then authorizes by (principal, host) and the MCP lane validates the guest's own SVID. GCP credential federation follows. State: SPIRE is live with the platform registration entries (phases 0 and 1); platform mTLS (phase 2) is Built, default off on both sides of the CP-to-noded hop and awaits staged acceptance on #5757 and #5758; guest identity, request-scoped injection, and federation (phases 3 to 5) are Planned.

Built, default off (phase 2 of #5706): noded opens a second gRPC listener on noded.spiffe.grpcTlsPort (9443) that requires a client X.509-SVID whose URI SAN is in an exact allowlist, derived by the chart from the control plane's ServiceAccount unless clientSpiffeIds overrides it (#5757). The control plane, with controlPlane.spiffe.enabled, runs a spiffe-helper sidecar that writes svid.pem, svid_key.pem, and svid_bundle.pem to a memory-backed volume in the SPIFFE Filesystem Delivery layout; with controlPlane.spiffe.dialNoded it dials every noded on the TLS port with a GRPC.Credential built from those file paths, verify_peer against the bundle, no partial-chain trust, SNI disabled, and a verify_fun that accepts only the exact noded SPIFFE ID the chart derives from the noded ServiceAccount (#5758). Both identity strings come from the same release fields on both sides, and the chart test pins that they agree. The bearer header keeps being sent on the TLS dial, so the dual window changes transport only. Rotation: :ssl reads the files at each dial and never caches their bytes in the control plane; noded bounds channel age with its gRPC MaxConnectionAge, after which the cached channel fails its next call, is invalidated, and the re-dial handshakes with the files on disk at that moment. Every flag defaults off in the chart and both checked-in overlays.

Why. Files rather than a Workload API client in the control plane: OTP's :ssl already reads PEMs by path and re-checks them, so the control plane needs no SPIFFE library, no long-lived source process, and no cached bytes, and a Kubernetes podCertificate projected volume can replace the sidecar later with no Elixir change. Exact-ID verify_fun rather than a trust-domain match: a compromised pod in the namespace holds a valid SVID in the same domain, so only the noded identity may terminate the dial. SNI disabled rather than a DNS SAN on the SVID: the dial target is a pod IP and the SVID carries only a URI SAN, so OTP's default hostname check can never succeed and must be replaced, not tuned.

Decided direction: GitHub leaves the agent egress catalog. Host-keyed injection bounds which host a credential reaches, never which request reaches it, so a prompt-injected guest can shape any GitHub API call and have the token attached to it. Agent principals instead reach GitHub through MCP tools whose URLs name the target repo, entitled per identity-provider group and backed by a per-group fine-grained PAT, so what a guest can reach is a fixed tool set rather than an API surface. The egress broker keeps the credentials that genuinely need host-keyed injection, the model providers among them.

Why. The original Kubernetes identity string conflated the credential holder, the isolation principal, and permission to use management actions, leaving object ownership checks unsafe for additional users (ADR embervm/032). TokenReview-only identity was rejected because it has no browser or device flow, while per-request identity-provider introspection was rejected because it adds a network dependency to every request. Provider-neutral actors, explicit principals, local token verification, and object-level authorization accept expiry-bounded revocation and loss of new login during an identity-provider outage (ADR embervm/024, ADR embervm/032).

10. Threat model

Built: Firecracker boundary; no NIC for task/session guests; principal-bound lineage; no cluster credential in guests; Envoy-only serving ingress; bound dial-home registration identity.
Accepted risk: privileged noded with /dev/kvm; external-allow guest egress via the broker; taint optional; CPU micro-architectural side channels between co-resident tenant guests. Tenants share bricks and Kubernetes nodes by design, and nothing pins a guest to cores or isolates SMT siblings: noded sets no CPU affinity, and bricks carry no CPU limit, so they are never Guaranteed pods the kubelet could give exclusive cores. Whether the Spot brick nodes run SMT is set in gcloud, not in git (#5255).
Built, staged (ADR embervm/039, #5255; arming tracked on #6965): with noded.jailer.enabled every guest launches through Firecracker's jailer: a per-VM chroot whose inputs are staged readable by the jail uid, a uid and gid from a block of the node's 20000-59999 range that each daemon leases for its lifetime (the lease is unique among one release's bricks, because they share its snapshot root; releases sharing a node pool must lease from disjoint ranges set by noded.jailer.uidBlockBase: production 0, and embervm-dev 20 before it arms), a cgroup v2 leaf with memory.max at the guest's memMib plus 192 MiB and memory.oom.group=1 under the capped embervm-vms parent (section 7), and a PID namespace noded creates with CLONE_NEWPID, so Firecracker is init of its own namespace while noded still waits on, kills and reads it as its own child (the jailer's --new-pid-ns is not used, because it leaves the VMM a grandchild noded cannot supervise). Snapshot outputs written inside a jail are handed back to the daemon, and the guest egress socket in the jail is chowned to the jail uid. It fails closed at startup and per launch: an armed brick that cannot delegate cgroups, find the jailer, or open every READY base rootfs through a jailed preflight exits before it registers; a jailed launch that cannot be contained is refused, never retried unjailed, and so is a legacy bundle whose metadata names no rootfs, which could only run direct; a startup sweep removes jails a dead daemon left. NodeStatus reports jailer_armed and cumulative launch_modes (jailed, direct_exec, refused), and conformance S7, once dev arms it, fails the dev gate on any unjailed or refused launch. Every deployment keeps enabled: false until its stage runs this chart; dev arms first, production after a dev soak, each with that one value.
Planned (ADR embervm/041 phase 2, in flight): X.509-SVID mTLS on the CP-to-noded hop and the token broker, retiring the static bearer.

EmberVM evaluates itself against the threat model published by agent-substrate/substrate at its cited 2026-06-25 revision, adopted as the external conformance frame (ADR embervm/033): the most complete public enumeration of what a multi-tenant agent execution plane must defend. The tables below condense it (their actor is Ember's guest, their worker pod a Firecracker slot on a brick, their atelet noded, their snapshot Ember's warmth artifact; threats are numbered 1 to 43 in upstream order as of its 2026-06-25 revision). Shared-worker threats with no Ember analogue are answered by the "no reuse across principals" row. Five threats remain unmapped and bind on Ember: 8 (template-author reach into storage), 21 (noded runs privileged with /dev/kvm), 26 (policy propagated out of band with scheduling), 31 (image-extraction resource limits), and 42 (detection integrations).

The boundary in one picture: what untrusted code can reach, and what never crosses toward it.

graph LR
    subgraph vm ["Guest VMs"]
        G1["task/session guest<br/>vsock only, no NIC"]
        G2["serving guest<br/>tap NIC"]
    end
    subgraph brick ["Brick: trusted host"]
        N["noded"]
        P["egress proxy<br/>(holds credentials)"]
        E["node Envoy"]
    end
    subgraph cp ["Control plane"]
        F["admission, facts,<br/>op-log audit"]
    end
    S[("artifact store<br/>SigV4 per-identity access, gateway enforces (#4708)<br/>per-principal encryption enabled per environment (#4691)")]
    X["external hosts"]

    G1 -- "vsock only" --> N
    E -- "DNAT into tap" --> G2
    G1 -- "plaintext egress" --> P
    G2 -- "plaintext egress" --> P
    P -- "fresh TLS, credential injected<br/>only for allowlisted hosts" --> X
    N -- "facts (dial-home), bound pod identity;<br/>bearer token + ingress policy enforced" --> F
    N -- "snapshot bytes, never via the CP,<br/>SigV4 per-identity access, gateway enforces (#4708)" --> S

Trust diagram legend: every edge is a current path.

Attacks from guests

Requirement (external mapping #)	Ember state
Hardened sandbox, never bare containers (15)	Built. Every guest is a Firecracker microVM; new execution technologies enter as lanes under existing classes (invariant 9).
Default-deny actor networking (17)	Built for task and session guests: they have no NIC and use vsock only. For tap-bearing serving and stateful guests, the generated `embervm_serving` table accepts established/related replies and drops NEW traffic that enters the `inet` forward hook from the shared serving bridge (`iifname <bridge> ct state new drop`: VM-originated forwarding, including external egress). That rule carries no same-bridge exemption, so its effect on two guests sharing the bridge depends on whether bridged frames traverse the inet forward hook on the deployed host at all; whether such guests can exchange bridge-local unicast, broadcast, ARP, or IPv4 traffic is runtime-unverified (#6159), not established by the generated inet rules. Composite members intentionally share a per-group L2 bridge: the generated `embervm_group` table keeps same-bridge member traffic (`iifname <group> oifname != <group> ct state new drop`) while dropping NEW traffic that leaves the group bridge toward another group, the serving bridge, or the overlay. That is composite lifecycle connectivity, not independent service authorization. #5813 is closed as NOT_PLANNED, and EmberVM ships no platform VM-to-VM L7 authorization layer. Brokered external egress remains the deliberate exception described in section 9.
No guest access to node-local endpoints or local substrate services (16, 19)	Built for task and session guests: with no NIC, their platform path is the vsock shim and brokered egress contract. Built at source level for tap-bearing serving, stateful and composite guests: both generators (`nftRuleset`, `nftGroupRuleset`) register an `inet` input-hook chain beside the forward chain that accepts only established/related replies and drops everything else entering from a guest bridge, so the bridge gateway, noded's pod IP and every wildcard listener on it (gRPC, the HTTP and L4 activators, health, metrics, the egress sidecar's loopback) are unreachable from a guest while host-originated probes and DNAT'd requests keep working; the forward chains drop INVALID as well as NEW guest-originated traffic; and every tap on the shared serving bridge is attached as an isolated bridge port (`bridge_slave isolated on`), so serving and stateful guests cannot exchange frames (unicast, broadcast or ARP) at L2 although they share one bridge. What remains unproven at source: the serving table still has no per-tap source IPv4 or MAC match, so a guest can spoof another port's addresses toward the host (an isolated port cannot deliver the spoofed frames to its neighbour). Installed rules, actual listener reachability, spoof outcomes and deployed hook traversal remain live checks in #6159.
No Kubernetes-internal topology discovery (34)	Built for task and session guests, which have no NIC and receive no cluster DNS path. For tap-bearing guests the generated input and forward posture denies every guest-originated flow toward the host and the overlay, which is where cluster DNS and the Service ranges live, but it is not a runtime proof about guest DNS configuration, pod or Service ranges or CNI policy. #6159 retains live validation of cluster DNS and topology exposure.
No Kubernetes or management-API escalation from guests (20, 22)	Built: no cluster credential by construction; the SPIFFE guest identity is Decided direction (section 9). Definitions are CP-owned and there is no self-modification verb.
Worker state fully reset between actors (18, 27, 30)	Built: no execution environment is reused across principals (invariant 3); placement is CP-owned, each VM sees an immutable rootfs plus private scratch, and ADR 028's planned chunk sharing exposes no other manifest or writable filesystem.
Credentials never inside the sandbox by default (28, 29)	Built: class 1 credentials enter PLATFORM-TRUSTED guests only and are revoked at bank; every other credential is injected at the sidecar hop for hosts in its `egressTo` (section 9). Planned: (principal, host) injection under SPIFFE (section 9).
Quotas and rate limits on creation and spend (9, 33)	Built, model-checked (`quota.tla`): admission fails closed, a budget of 0 is a hard stop, metering rides the operation (invariant 4). The per-principal daily budget is unset in the reference deployment, so spend is bounded by admission caps and concurrency.
Snapshot theft, substitution, or self-written snapshots (23, 24, 25, 32)	Built: the store gateway enforces SigV4 and only the `embervm` identity writes the ember buckets (#4708); envelope encryption and the restore capability bind mutable warmth to its tuple (#4691). Digest-verified manifests remain Planned.

Attacks from clients and the internal network

Requirement (external mapping #)	Ember state
No direct internet exposure of guests, nodes, or the CP (1, 2, 3)	Built. Nothing faces the internet directly; ingress rides the zero-trust edge tunnel, public routes are scoped at their HTTPRoutes, node Envoy returns 404 for `/shim/*` before routing, and hydrate is build-only (section 9).
Mutual authentication and encrypted transport between components (4, 10)	Built for CP-to-noded authentication: one bearer Secret rendered into the control plane and every noded pod, attached to every gRPC request, enforced in production and dev (#4693). The inert noded and token-broker Cilium policies were removed in #5816. The hub has no corresponding CRDs; the bearer remains the noded authentication control, not a substitute for network isolation. SPIFFE X.509-SVID mTLS is the additive upgrade the proto reserves, Built, default off on both sides as phase 2 of #5706 (section 9) and awaiting staged acceptance. Encrypted session-routing tokens and the actor / principal / permission split are Decided direction; management callers authenticate via Kubernetes TokenReview against an allow-list.
Control plane isolated from the data plane (6)	Built as a seam: the CP runs on Kubernetes, noded on bricks, payloads never traverse the CP (invariant 2). Accepted risk on a small fleet: guests co-locate with the etcd masters (section 11).
Runtime configurable only by administrators (7)	Built. A workload chooses class and source; the sandbox technology, kernel, and bases are CI-built platform artifacts it cannot substitute.
A sanctioned, secure path for secrets (11)	Built. The cluster's secret operator is the only secret source, and guests receive none (section 9).

Attacks from nodes and insiders

Requirement (external mapping #)	Ember state
Node storage access scoped to actors scheduled on it (36, 37)	Built: SigV4 at the gateway, `embervm` identity only (#4708); a brick receives a five-minute decryption capability for exactly the tuple it is waking (#4691), armed in dev and production.
Node API access scoped to its own actors (38)	Built: node reports are authoritative only for instances anchored to that node, wake grants are gated on the anchor (section 4), and the bound token's pod-uid and node-name claims must match the registration. A brick registers only itself.
Granular admin access and envelope encryption at rest (39, 40)	Built for mutable principal warmth: derived per-epoch KEKs or a customer KMS oracle (section 9). Immutable rootfs chunk encryption is Not planned (ADR 028, #4182); the op-log shares a Postgres cluster (section 11); payload separation and principal-scoped erasure are Decided direction.
Audit logging of all control actions (41)	Built. Every lifecycle and enforcement action is an ordered op-log append (invariant 7), prefix-compacted past 30 days; older audit lives in the observability stack.
Containment of a detected-bad actor (43)	Built for one lever: principal cutoff as an admission action (stop minting tokens, 402 at the edge). The volume quarantine is a data-integrity guard, not an adversary control; no brick- or principal-level quarantine exists and automatic containment is undecided.

Why. Runtime isolation did not protect banked memory from a compromised brick, storage reader, or insider with object-store access (ADR embervm/033). A self-authored threat list was rejected because it could be shaped around existing choices, and storage ACLs alone were rejected because they authorize more than a specific principal, lineage, and generation. The external conformance frame and per-principal envelope encryption make those gaps explicit, accepting restore latency, key-count growth, and cold boot when a key is unavailable (ADR embervm/033, ADR embervm/036).

11. Deployment shape

Why. Noded's image is amd64-only (#7065, Joe's 2026-10-28 decision): the fleet has no arm64 consumer. The daemon and all four Firecracker bundle files are retained in two amd64 layers on Wolfi. Shared archive registration still fetches both architectures.

Minimum platform contract

Requirement	Why	Fails how if absent
KVM-capable Kubernetes nodes with `/dev/kvm`	Firecracker execution	Bricks cannot start guests
Privileged noded	Firecracker, tap, and local device access	VM and network setup fails
Local scratch bind-mount at `/var/lib/embervm/scratch`	Warmth cache and local artifacts	Pod fails closed on missing scratch
S3-compatible object store	Artifact export and deliberate restore	Archive or restore is unavailable
Postgres (or SQLite-WAL single-node)	Durable op-log	Control plane cannot persist facts
Gateway API ingress	Edge serving route	Public serving ingress is unavailable
CNI permitting tap/DNAT on the Kubernetes node	Serving and broker network paths	Guest network paths fail
Secret operator	Platform secret source	Credential setup fails closed

EmberVM asks for the platform contract above. The reference deployment uses KVM-capable nodes, a local scratch disk per Kubernetes node, and an S3-compatible object store.

Scratch is a node-provisioning contract: every FC-labelled Kubernetes node bind-mounts its real device at /var/lib/embervm/scratch (hostPath type Directory fails closed if unsatisfied). Karpenter instanceStorePolicy RAID0 satisfies it on EKS; local NVMe elsewhere. Where no out-of-band bootstrap exists, the chart's scratchPrep DaemonSet provisions a size-capped loop file at that path (the GKE hub). ext4 remains the safe default. XFS with reflink is staged behind scratchPrep.filesystem: xfs for newly created managed images. Destructive replacement of an existing ext4 image requires the separate scratchPrep.migrateExt4ToXfs: true gate. The migration path fails closed unless the exact managed non-symlink image has no hard-link aliases, its ext4 loop fstab entry, host loop device and sole mount target agree, and no active consumer is found. The verified inode stays open across mkfs so a concurrent path replacement cannot redirect the reformat. It never reformats XFS, never force-unmounts, and leaves foreign bind mounts untouched. Both gates remain default-off pending the live checks in #5699. Bases under scratch are node-shared across co-located bricks. Scratch does not survive a Spot node replacement: every guest rootfs rebakes in the brick init containers and the control plane re-drives the dropped bases without a restart, about ten minutes end to end.

Why. Scratch is deliberately ephemeral. A per-node persistent disk would have to be named, attached and fenced across Spot replacement in an autoscaling pool, would carry the ext4 loop file's generation marker past the node that wrote it, and would cost more than the Spot saving it protects on a three-node pool. Preserving the filesystem also proves nothing about a database's recovery point: demo-postgres durability rests on its export path, not on scratch surviving. The reference design therefore pays for replacement with reconstruction: the rootfs cache (#5772) shortens the rebake and the control plane re-drives bases without a restart. Persistent scratch is reconsidered (#5773) only against a measured post-cache replacement that a named workload cannot tolerate, or a workload that declares local data must survive with a stated RPO and RTO, and even then reconstruction or export recovery is compared first. The filesystem choice for any such disk follows #5699.
Warmth never crosses a CPU vendor (until CPU templates land): artifacts are keyed per vendor, the gate fails closed at the daemon, and each vendor pool holds its own warmth. noded.warmRestoreWithVolumeClasses arms uniformly across the size classes, never partially, per the rollback contract; measured load-to-resume is 2.5ms on both vendor pools.
The control plane runs one replica with strategy: Recreate; the reference deployment configures its op-log as a Postgres database (a shared cluster is acceptable: a CP outage is a designed-for state, and CP rolls are the availability events the node-local activator exists to survive).

Why. One replica is a choice, not a limitation of the op-log: the Postgres tier is live and the CP's state is a reconciled cache of what bricks report, so a second replica could rebuild the same hot set by observing. What is not safe is the action side. claimForAssign already fences duplicate assignment at the node, but Destroy, Bank, StopStateful and Relight carry no ownership epoch on the noded RPC surface, and EndpointPublisher versions are boot-epoch prefixed so an older-booted writer is locked out permanently. A CP outage is a designed-for state (pod reschedule plus op-log projection load plus the node-local activator covers admitted work), and no workload or factory operation has yet named an availability or recovery bound that this fails. When one does (#3862), the sequence is fixed: first the fence (a per-brick ownership epoch mirroring wake_grant, and a multi-writer-safe publisher version), then the smallest topology that meets the requirement, which is a leader-elected actor tier on a Postgres advisory lock with observers on every replica. Brick-sharded ownership over libcluster and Horde is a fan-in scaling tool for a fleet of thousands of bricks, not an availability tool, and cells (#3855) were withdrawn, with the implementation PR closed unmerged. Multiple clusters alone do not reopen them. The ADR 007 loop walls (global prime budget, O(V^2) adoption sweep, timer reconciles) each buy more than replicas and are fixed first.
Small fleets may co-locate guests with control-plane nodes as an eyes-open risk acceptance, provided the cluster is GitOps-reconstructible and durable state lives in S3: quorum loss is bounded downtime, not data loss, and guests are the first OOM victims. Do not import this clause into a cluster whose etcd is precious.
A hard node taint is a recorded option, not required: co-tenancy runs on honest requests plus a disposable priority class for guests.

The reference deployment's concrete fleet (node roles, current brick mix, shared Postgres) is in deploy/README.md.

Promotion gate

EmberVM promotes the dev chart to production through Kargo: argocd-wait proves the Application Synced and Healthy, a soak interval catches failures that appear after reconciliation settles, and the in-cluster conformance runner's S1 to S4 scenarios (two concurrent restored-clone vsock exchanges, session sleep and relight, second-session restart latency, control-plane invariants) must all pass at /verdict, stamped with the chart version so an older deployment's evidence is refused. S1 sends a distinct marker into each clone, requires the same marker back, rejects the peer marker, and requires two 15-second guest holds to complete inside a 25-second overlap budget. The budget includes 10 seconds of explicit restore, admission, and HTTP headroom while remaining shorter than two serialized holds. The hold is bounded by the sandbox guest's own 25-second wall-clock cap, which S1 requests explicitly. Two more scenarios run after the rest once embervm-dev arms them with the jailer (conformance.requireJailer, conformance.egressProbe; #5255, #6965). S7 reads /v1/capacity before S1 and after the suite and requires every brick to report jailer_armed, no direct_exec or refused launch across the suite, at least one jailed launch on a brick whose counters ran unbroken across the suite, and a nonzero budget with its own admission predicate on every classed brick. A brick that registers or restarts mid-suite counts its startup preflight as a jailed launch and has lost any earlier counts, so it credits nothing. S7 fails at once when every brick from suite start is listed with its counters reset. When no listed brick kept its counters but one from suite start is missing from /v1/capacity, S7 keeps polling inside its budget, since the registry drops a brick whose stream is unknown or draining and restores it with its counters. S8 has a task guest dial the host's egress vsock port and requires the egress sidecar's CA certificate back, which proves a jailed Firecracker can connect to its egress socket. A failed or all-vacuous run blocks promotion; Freight approval is the explicit operator override. Phase 1 is tracked in #5224. The launch-mode, crash-relaunch, and architecture acceptance matrix for restore-time vsock overrides is a separate manual readiness gate in docs/drills/vsock-override-acceptance.md.

Known walls and provisional numbers (each states what would move it):

Item	Value	Status	Confidence
`statefulTcpPortRange`	10 ports (5400-5409), CRD-validated	Built	asserted; hard cap, remedy constrained to name-based L4 (SNI/PROXY protocol)
CPU pivot	1,024 MiB per vCPU	Decided direction (#5231)	not implemented: `vcpus` is an explicit CRD input; derive from `memMib` when the resource-model direction in section 6 lands
Active brick utilization target	>90%	Built	asserted; too high if shed events become common
Stateful continuity floor	8h (also the S3 stateful warmth TTL)	Built	asserted; validate against rotation cadence
Session lifetime cap	`maxLifetimeSeconds` (agent lanes 21600s by values, CRD default 86400s)	Built	asserted; version-convergence bound, not a durability claim; no platform-wide clamp
Brick silence timeout	21600s armed	Built	asserted; the divergence bound for the ownership-arbitration design (ADR 037)
Wake-grant gap budget	k=4 (default cadence class)	Decided direction	the Fork B grant; Fork A activator self-advance is what is Built
Definitions target	100k+	Planned	provisional; owner-set goal, not measured demand

Why. Per-workload Helm templates fail as an authoring surface well before a large informer relist makes etcd a recurring hazard (ADR embervm/026). A larger etcd with one CR per workload was rejected because it leaves the earlier authoring failure, and a pod or Service per execution was rejected because it restores the object churn EmberVM was created to remove (ADR embervm/001, ADR embervm/026). Kubernetes therefore owns low-churn definitions and brick scheduling while the standalone chart owns the execution plane, accepting KVM-capable nodes, local scratch, and an external durable store as the minimum deployment contract.

12. Direction

#7210: durable existing-session residency demand and factory capacity waits, default-off preparation; dev operational acceptance and production activation remain separately authorized.

Decided and not yet built, each with the issue that tracks it. A row leaves this table when the work ships or the issue closes without it.

Direction	Decided in	Tracks	State
Factory guests share credential-free bounded CI log reads	section 4	#6442	helper packaged; guest uptake and next-window repeated-call evidence pending
Brick scale-down couples durable archive acknowledgement with one brick per node (Joe, 2026-10-10)	section 7	#6533	single repository gate default-off; staged enable and live acceptance pending
Guest digest-only publishes bake rootfs files in place without rolling bricks or draining live sessions	section 8	#6662	repository implementation staged; ReplicaSet and session live acceptance pending
The Firecracker jailer arms on every brick, closing the direct-root-exec gap between co-resident guests	section 10	#5255	built and staged; dev arming then production after a dev soak (#6965)
The brick `maxReplicas` ceiling itself moves on sustained denial pressure, not only the replica count clamped inside it	section 7	#5505	built, hub dev 4gi bound-1 canary staged; live acceptance pending
SPIFFE-issued identity moves beyond issuance: mTLS on the CP-to-noded hop, per-principal guest JWT-SVIDs, and GCP federation	section 9	#5706	CP-to-noded mTLS built default off on both sides (#5757, #5758), rest not started

13. Decision history

The ADR files were removed on 2026-09-06 (#4667); git log -- docs/decisions/ has the full text.

ADR	Decision	Status	Disposition
embervm/001	EmberVM itself: BEAM control plane over a Go node daemon, hit/miss invariant, five classes, isolation model	Accepted; copy-never-rebuild for stateful withdrawn by 025	deleted
embervm/002	Op-log retention: read-time TTLs, 7 day terminal prune, 30 day journal horizon behind a durable marker	Accepted, Built; shape restructured by 019	deleted
embervm/003	Control-plane-managed snapshot distribution, Build / Restore / Export / Evict verbs	Accepted, Built; verbs generalized by 009	deleted
embervm/004	Back kubernetes-sigs/agent-sandbox through a deferred edge adapter, no native session API	Accepted historically; adapter not built and now not planned (#5806)	deleted
embervm/005	EKS scale-out: metal pool, multi-daemon bricks, EmberPool CRD, dial-home	Accepted; EmberPool never built, brick counts are a values knob behind `BrickController`; decision 3 superseded by 028 (#3849, #3851); privileged noded now tracked by #5702	deleted
embervm/006	TLA+ pilot with three conformance layers	Accepted; six specs run under TLC in the build, trace validation deferred to 034	deleted
embervm/007	Batched Postgres op-log tier, cells, hot-loop corrections	Accepted historically; Postgres Built, cell work closed unmerged and is not planned (#4753, #3855, PR #6069); tracker #3853 closed	deleted
embervm/008	Opt-in two-phase interruptible bank	Accepted, Built	deleted
embervm/009	Continuity before tenancy: R6 to R9, spot availability contract, S3 seam	Accepted; quickstart merged (#3856, #3858), clean-host KVM validation not yet done	deleted
embervm/010	Bazel warm-Skyframe public demo as a stateless query consumer	Retired with the exhibit (#6937); runtime removed in #7056	deleted
embervm/011	Vendor-bound warmth, single-writer fencing, CP-sequenced rollouts	Accepted; stateful Longhorn withdrawn by 025; sole-issuer rule amended by 017, 018, 040	deleted
embervm/012	Co-located fleet, etcd blast radius accepted, grandfather rule, registry survives restart	Accepted; dynamic sizing retired by 013; HA not planned (#3862)	deleted
embervm/013	Classes are reuse semantics, substrates are lanes; brick sizing; bricks everywhere	Accepted, Built	deleted
embervm/014	Worker-authoritative state and node-confirmed destruction; decision 2 async writes retired by #7059	Built (`adoption.tla`); decision 2 superseded by synchronous lifecycle appends (#7059); metering clause amended by 020	deleted
embervm/015	Isolated high-throughput lane with data-plane placement	Draft; not planned (#3864 closed, its #6132 reuse and evidence gate is the reopen criterion); fail-closed lease withdrawn by 020	deleted
embervm/016	Kubernetes scheduling contract: the pod is the ABI, priority projection, session ladder	Accepted; kwok drills never built; placement loop superseded by 020, ladder amended by 025, 027, 029	deleted
embervm/017	Bounded auto-heal of the checkpoint-abort quarantine	Accepted, Built (`generation_issuance.tla`)	deleted
embervm/018	Node-local activator (Fork A), brick-authoritative lifecycle (Fork B)	Accepted; Fork A shipped, Fork B not started (gated, see section 4)	deleted
embervm/019	Op-log payload separation, time partitioning, principal-scoped erasure	Draft, Decided direction	deleted
embervm/020	Admission-only control plane, token routing, peer redistribution, fail-open metering	Draft, Decided direction; decision 3 withdrawn to 023	deleted
embervm/021	`memMib` as the only dial, derived CPU, GB-seconds	Draft, Decided direction	deleted
embervm/022	Composition over bindings, domain seam, three-leg access fabric	Draft historical rationale; domain and service-binding programmes are not planned (#5072, #5813)	deleted
embervm/023	Class-scoped ownership arbitration, silence timeout as the divergence bound	Draft, Decided direction; the timeout is Built (037)	deleted
embervm/024	Identity hierarchy, platform principal, guest identity assertion	Draft historical rationale; domain/platform-principal work is not planned (#5072), guest identity mechanism superseded by 041	deleted
embervm/025	Local disk authoritative, S3 an archive, `archiveInterval`	Draft, Decided direction; export at bank commit Built	deleted
embervm/026	Templates not stamps, GitOps without per-workload CRs, desired-set registration	Draft, Decided direction	deleted
embervm/027	Snapshot modes as a declared workload property	Draft, Decided direction for persistence modes and retention; size budget superseded by the evidence gate in #5074	deleted
embervm/028	Eager-local rootfs: OCI ref, Account chunk store, ublk	Accepted, not planned (#4182); Phase 0 measured on #4182	deleted
embervm/029	Parked sessions count as disk, not against `concurrency.cap`	Accepted, Built	deleted
embervm/030	Lineage decoupled from session generation; the 6 h cap is a convergence bound	Accepted, Built	deleted
embervm/031	Historical export-failure and GC-age detector, never armed	Built (#4338), removed (#7058)	deleted
embervm/032	Actor / principal / permission split, authentik SSO for the management surface	Draft, Decided direction; SPIFFE seam resolved by 041	deleted
embervm/033	Substrate threat model as the conformance frame, per-principal encryption at rest, tuple-authorized restore	Accepted, Built and armed; customer KMS mode and digest-verified manifests open (#4691)	deleted
embervm/034	Conformance harness: two lanes, two tiers, anti-vacuity, Freight override	Accepted; SpecTrace runs in dev, harness Planned (#4761, #4763)	deleted
embervm/035	Website snapshotter task guest	Draft, Built (#4994)	deleted
embervm/036	Platform KEK custody derived in the control plane's key service	Accepted, Built	deleted
embervm/037	Brick silence timeout	Draft, Built and armed at 21600 s (#5073)	deleted
embervm/038	Image-lane serving fresh boot	Draft, Built	deleted
embervm/039	Fair-share brick resources, blast-radius ladder, capacity as a signal	Accepted; 75% memory requests Built, reversed to equal limits where reserved admission is armed (#4185, #6965); shedding Built and configured on in production; jailer Built and staged (#5255, #6965); signal Decided direction	deleted
embervm/040	Anchor-loss recovery, preemption budget split	Accepted; restore-onto-peer Built, preemption watcher Built and disarmed	deleted
embervm/041	SPIFFE workload identity on self-hosted SPIRE	Accepted; phases 0 and 1 done, phase 2 in flight (#5706)	deleted
embervm/042	Brick class ceilings move on denial pressure	Accepted, built; hub dev canary staged, live acceptance pending (#5505)	deleted
embervm/README	Reading order for 019 to 027 and the sparse-facts through-line	folded into section 6	deleted
embervm/brick-program-decisions	2026-07-19 run log: bricks everywhere, pod UID identity, scratch contract, staged cutover	carried by sections 7 and 11	deleted

Agent-platform ADRs whose decisions shipped through EmberVM. Their status column carries the wording the monolith map uses for the same rows.

ADR	Decision	Status	Disposition
agents/002	OpenHands sandboxes via agent-sandbox	Superseded by agents/004	deleted
agents/014	AX + Substrate as the agent runtime	Deprecated	deleted
agents/019	Substrate executor interface, AgentWorkflow over Argo	Accepted, not shipped as designed	deleted
agents/021	Discord-triggered AgentWorkflow with snapshot and resume	Draft, evolved into agent sessions	deleted
agents/022	Firecracker snapshot/restore controller (FC-direct)	Accepted, shipped through EmberVM	deleted
agents/023	Egress secret proxy	Draft; header injection shipped (section 9), placeholder substitution rejected	deleted
agents/025	Three-layer agent stack	Draft, evolved into EmberVM	deleted
agents/026	Fast microVM cold starts, stateful artifact iteration	Accepted, shipped through EmberVM	deleted
agents/028	Elastic agent-microVM capacity and state-preserving reclaim	Draft, shipped through bricks and the scheduler	deleted
agents/030	fc-invoke as one configurable surface	Draft, evolved into EmberVM	deleted
agents/031	Control-plane / data-plane split	Accepted, shipped through EmberVM	deleted
agents/032	Warm-snapshot Bazel worker as an MCP tool	Draft, partially shipped as the public Bazel demo	deleted
agents/033	Golden-template distribution via daemon-pulled OCI	Accepted, superseded by control-plane-managed base builds (003, 028)	deleted
agents/034	Per-tier guest MCP ACLs	Draft, not shipped (#3838)	deleted
agents/037	Label-driven Firecracker node enrollment	Accepted, shipped (section 2)	deleted
agents/040	Caller-provided context injection	Draft, not shipped	deleted
agents/041	Hot git mirror for agent workspaces	Draft; the sidecar was removed on 2026-09-06 (#5825), hydration is a direct clone (section 4)	deleted
agents/044	Code executor sandbox, self-describing guest runtimes	Accepted, shipped (task-class guests, monolith sandbox tools)	deleted
agents/045	FaaS on the fc-invoke sandbox runtime	Accepted, execution migrated to the EmberVM zip lane	deleted
agents/046	MMDS for dynamic per-workload guest env	Accepted, shipped as the metadata seam for stateful and group guests	deleted
agents/047	Per-principal egress credentials, broker identity envelope	Draft; broker grants shipped (section 9), per-principal scoping moves to 041 phase 4	deleted
agents/048	Codex OAuth single-owner token broker	Accepted, shipped (section 9)	deleted
agents/050	Workspace hydration from the git mirror	Accepted; hydration shipped as a direct clone, the mirror never served one (section 4)	deleted
agents/051	Guest-pushed mid-turn progress	Accepted, shipped	deleted
agents/055	Tool-mediated GitHub access	Superseded by agents/059	deleted
agents/057	Per-language sandbox guests	Draft, shipped (`runtimes/`)	deleted