EmberVM Architecture

How EmberVM works: the current state and the decided future, readable end to end. Assumes Kubernetes and Linux virtualization; EmberVM terms are in the section 2 vocabulary. Claims carry four flags:

• Built: current behaviour, and the default for any unflagged claim. If reality disagrees with this document, one of them is a bug to fix.
• Planned: designed, not yet or only partly landed; scheduled work carries its tracking issue.
• Decided direction: agreed design, not an implementation claim.
• Accepted risk: an eyes-open trade, stated where it applies.

model-checked means the named TLA+ spec in projects/embervm/specs/ satisfies the stated property under TLC in the build; implementation conformance (trace-to-action validation) is Planned (#4699).

1. What EmberVM is

Self-hosted Firecracker orchestration: a private Lambda equivalent on metal you own. An organization sizes (or elastically bounds) a Firecracker nodepool; EmberVM provides placement, fairness, isolation, metering, and lifecycle so internal workloads get Lambda-shaped submit / scale-to-zero / warm-serve behaviour without a hosted FaaS product, without a Kubernetes object per invocation, and without etcd in the execution path.

EmberVM deploys on any Kubernetes with KVM-capable nodes, a local scratch disk per FC-labelled Kubernetes node, and an S3-compatible object store; section 11 states the platform contract. The reference deployment in this monorepo is a small on-prem cluster whose concrete shape lives in deploy/README.md.

Goals: private Lambda ergonomics (HTTP invoke, zip or image source, caps, quotas, internal chargeback); honest scheduling on finite capacity; isolation by default; the control plane off the hit path; one Helm-installed system. That system comprises an Elixir control plane, fixed-size brick pods running noded, node and edge Envoy tiers, and a Workload CRD; the op-log database and the S3-compatible store are operator-provided.

Non-goals: a hosted multi-tenant cloud; "agent platform as the product" (agents are dogfood consumers); every workload class as an equal pillar (task, serving, session are the core; stateful and composite are optional advanced classes); pretending capacity is infinite (queue depth, saturation signals, and admission control are the product surface instead).

Capability matrix

Why. The original single-node RPC path had no durable task record, ownership, managed retry, cross-node placement, or fairness (ADR embervm/001). Argo Workflows was rejected because per-job Kubernetes objects move high-churn state into etcd and add pod startup to every short task; a bare pull queue was rejected because it discards placement and snapshot locality. A BEAM control plane over a Firecracker data plane was chosen with an accepted Elixir toolchain cost and a smaller contributor ecosystem (ADR embervm/001).

Hard multi-tenancy (virtual control planes) is deferred pending real demand. Internal rung IDs R0-R9 map onto the capabilities above.

2. System overview

Vocabulary

• Kubernetes node: a KVM-capable machine.
• brick: a fixed-size capacity pod scheduled onto a Kubernetes node, several per node.
• noded: the Go daemon on each brick that owns Firecracker and node-local I/O.
• op-log: the durable ordered journal of lifecycle and enforcement actions.
• bank / relight: suspend a VM into a warm snapshot, then restore that snapshot.
• primed / banked / cold / running: ready pristine, snapshotted, fresh-boot, or actively executing instance states. Cold and warm describe boot mechanics only; task isolation remains no reuse, including after snapshot restore.
• generation: the volume incarnation number used to prove state provenance and pair a stateful volume with a memory snapshot.
• principal: the identity boundary a workload runs as.
• dial-home: noded registration and status sent to the control plane.
• warmth: a reusable boot artifact, such as a base or banked snapshot, that can make the next start faster but is never correctness-critical.
• bundle: the published set of artifacts needed to relight a workload.
• blessing: the durable control-plane act that authorizes a volume generation.
• wake grant: a bounded authorization for an activator to advance a workload wake during control-plane absence.

The diagrams use vsock (Firecracker's host-guest socket), DNAT (destination network address translation), xDS (the Envoy configuration protocol), BEAM (the Erlang virtual machine), ETS (Erlang Term Storage), and CNPG (CloudNativePG).

graph TB
    subgraph cp ["Control plane (Elixir/OTP, 'ember')"]
        API["HTTP API<br/>/v1/workloads /v1/usage"]
        DISP["Admission + placement<br/>(dispatcher, class managers)"]
        POOL["Warm-pool refill<br/>(PoolManager)"]
        XDS["Serving endpoint publication<br/>(EndpointPublisher, sole xDS writer)"]
        OPLOG[("op-log<br/>Postgres in reference deployment,<br/>SQLite-WAL in chart/dev defaults")]
        ETS[("ETS hot set<br/>rebuilt on start")]
    end

    subgraph node ["brick (pod on a Kubernetes node)"]
        NODED["noded (Go)<br/>Firecracker driver, vsock,<br/>tap/DNAT, volumes, activator"]
        ENVOY["node Envoy<br/>(serving relay)"]
        PROXY["egress proxy"]
        VM1["task / session VM<br/>vsock only, no NIC"]
        VM2["serving VM<br/>tap NIC"]
        VOL[("vol.img on node NVMe<br/>stateful, authoritative")]
        SCRATCH[("/var/lib/embervm/scratch<br/>warmth cache: bases,<br/>banked snapshots")]
    end

    S3[("S3-compatible object store<br/>artifact export / archive")]
    K8S["Kubernetes<br/>Workload CRD (definitions),<br/>brick pods, PriorityClasses"]

    CALLER["task/session caller"] --> API
    API --> DISP -->|gRPC| NODED
    NODED -->|vsock| VM1
    EDGE["Gateway API HTTPRoute"] --> ENVOY
    XDS --> ENVOY
    ENVOY -->|DNAT into tap| VM2
    VM1 --> PROXY
    VM2 --> PROXY
    NODED ==>|dial-home NodeStatus:<br/>authoritative state| DISP
    NODED <-->|Export/Restore/Evict| S3
    NODED --- VOL & SCRATCH
    K8S -.->|watch definitions| API

Diagram edges: solid = request path; thick = facts; dotted = definitions. The edge Envoy tier is the Gateway API ingress in front of node Envoy.

Division of labour (each responsibility placed where it is cheapest to make correct):

Bricks dial home: on start and on a jittered interval each noded POSTs its identity {node, pod_uid, address, boot_id} to /v1/nodes/register; the control plane adopts it keyed by (node, pod_uid) and streams WatchNode. The control plane never lists-and-watches daemon pods. Growing the fleet is labelling a node (homelab.io/firecracker=true), not a values edit.

Decided direction (#4696): enrollment consolidates on a single chart-configurable key, embervm.jomcgi.dev/node, used as both the enrollment label and the key of the optional hard taint; the separate serving label is retired, since every enrolled node carries the serving relay.

How one invocation works

A serving hit (the steady state), where the control plane is not involved at all:

sequenceDiagram
    participant C as Caller
    participant E as node Envoy
    participant VM as Serving VM
    C->>E: edge HTTPRoute
    E->>VM: kernel DNAT into tap NIC
    VM-->>C: response

A wake (a serving or stateful miss) is normally coordinated by the control plane, which restores the VM and republishes the endpoint through its EndpointPublisher:

sequenceDiagram
    participant C as Caller
    participant E as node Envoy
    participant A as Fallback activator
    participant CP as Control plane
    participant VM as Guest VM
    C->>E: request (workload scaled to zero)
    E->>A: fallback endpoint, request parks
    A->>CP: wake request
    CP->>VM: single-flighted restore or cold boot
    CP-->>E: real endpoint published via xDS
    E->>VM: parked bytes splice through
    VM-->>C: response

The node-local activator (Planned, partially landed) runs the same wake node-side during control-plane absence: parked bytes splice via stable node-local DNAT with no mid-wake xDS republish.

A task (the all-miss case by policy), a fresh VM per invocation, async by default:

sequenceDiagram
    participant C as Caller
    participant CP as Control plane
    participant N as noded
    participant VM as Guest VM
    C->>CP: task POST /v1/workloads/:name/tasks
    CP->>CP: admission + quota, op-log append
    CP-->>C: 202 + task_id (?wait=true parks instead)
    CP->>N: gRPC assign (primed VM)
    N->>VM: payload over vsock
    VM-->>N: result
    N-->>CP: result, VM destroyed after one task

The platform contract is in section 11; the security posture summary is in section 10.

Why. Per-dispatch metering and placement made control-plane work grow with the number of sandboxes, while session endpoints in xDS would make configuration grow with session count (ADR embervm/020). Keeping central filter, score, and bind placement at dispatch rate was rejected as a scheduler throughput ceiling, and a global session directory was rejected because it adds a lookup to every request. The control plane now owns fleet facts and precomputed assignment while bricks own runtime facts, accepting reconcile-time metering gaps and bounded stale-state risk during partitions (ADR embervm/014, ADR embervm/018, ADR embervm/020).

3. Workload classes

Definitions are Kubernetes Workload CRs (schema-validated, GitOps-synced); kubectl get workloads reads back snapshotRef, observedGeneration, and readiness. Task invocation is POST /v1/workloads/:name/tasks. Session invocation is POST /v1/workloads/:name/sessions, followed by POST /v1/sessions/:id/invoke. source is a oneOf ladder: image (bring an OCI image, no SDK; contract is "listen on the declared port, answer a health path") and zip (runtime base + Lambda-compatible handler(event, context) shim; archives are fetched and unpacked inside the disposable guest).

A serving base built from an image registers its rootfs as the serving-images entry with no handler path, and a fresh boot runs the image's own entrypoint as the HTTP server; only zip-lane boots attach a handler drive (Built, ADR embervm/038, noded/server/activator.go).

Planned:

• Isolated high-throughput lane: Envoy routes straight to per-brick listeners, and each brick pops a fresh VM per request.
• Persistence as a declared property: workloads declare persistence flags; memory and filesystem persistence decouple from class.
• Composition: multi-component apps become independent Workloads wired by bindings rather than composite groups.

Why. Per-request tenant isolation (one fresh VM per request, destroyed after the response, re-primed locally with no control-plane hop) is the fast execution path the session and task classes do not offer, and it is retained, not dropped. It is not built as a second stack: the Semgrep scan pod (#6132) is extracting the task substrate (prime, assign, destroy, registry, budget, jailer) into a native library with a local pool loop, and the lane is that library behind an Ember-facing request entry. Fresh execution state per request is the invariant; restoring many guests from one clean base satisfies it, reusing a prior tenant's mutated state never does. Routing reuses Kubernetes Service plus pod readiness first; LEAST_REQUEST, retry-on-503 and outlier ejection are added only for a demonstrated gap, and a retry never replays tenant work after an uncertain result. The per-brick fail-closed quota lease was withdrawn by ADR 020 and stays withdrawn. The lane becomes agent-ready when the library exists and a workload needs it (#3864).

Why. Classes originally bundled persistence policy, leaving no class for a filesystem-backed workload without a memory snapshot (ADR embervm/027). A new class for each persistence shape was rejected because persistence is orthogonal to class scheduling and network semantics; composite groups were rejected as the default application graph because joint lifecycle and private networking couple otherwise independent components (ADR embervm/022, ADR embervm/027). Declared persistence properties and mediated bindings carry the added schema and migration cost while composite remains available for multi-kernel groups.

4. Lifecycle

Stateful bank/relight and the interruptible bank

stateDiagram-v2
    [*] --> serving: wake (cold boot or relight)
    serving --> banking: idle window elapses, 0 active connections
    banking --> banked: atomic bank (pause, snapshot, destroy)
    banking --> checkpointed: interruptible bank opt-in:<br/>pause + snapshot to temp
    checkpointed --> serving: connection parked -> ABORT<br/>(bless gen+1, delete temp, resume: HOT)
    checkpointed --> banked: nobody waiting -> COMMIT<br/>(publish bundle, destroy)
    banked --> serving: wake -> relight<br/>(only if bundle_generation == volume_generation)
    banked --> serving: pairing mismatch -> cold boot from volume<br/>(slower, never incorrect)

Facts that make this safe:

• Bank only starts at zero active connections; a long-lived connection is never severed by an idle bank (rolls are the exception, bounded by the drain contract: a graceful roll gets the platform's own 110 s budget, a GCE Spot preemption about 30 s, two numbers never to be conflated).
• Resume requires an exact (memory snapshot, volume generation) pair; mismatch discards warmth and cold-boots from the durable volume. Model-checked (bank_relight.tla), including the monotonic floor: a lagging node report can never regress the stored pair key, so a stale report cannot legalise a stale snapshot.
• The interruptible bank (spec.stateful.interruptibleBank: true, off by default; armed in the reference deployment for demo-postgres only) makes steady-state wakes always hot or warm, never cold. The three cold exceptions: genuine first boot, explicit operator reset, and the max-lifetime forced roll.
• Only a snapshot taken immediately before teardown may be kept. Any snapshot whose VM resumes is discarded (the abort orders bless-generation, delete-temp, resume, so a surviving temp is always refused by pairing).

Generation blessing and quarantine

The control plane is the adjudicator of volume generations, with exactly three legitimate issuance shapes:

1. CP-issued pre-dispatch: blessed durably before a wake/attach (op-log-before-dispatch). The default.
2. Checkpoint-abort auto-heal: noded's resolve-timeout auto-abort self-bumps by exactly +1 on the same vm_id. A durable checkpoint_dispatched{workload, vm_id, generation} record lets a restarted CP prove the +1 was its own checkpoint and bless it instead of quarantining. Anything unproven stays quarantined; the break-glass runbook is docs/runbooks/embervm-stateful-generation-quarantine.md.
3. Activator self-advance (Built, Fork A): during control-plane absence the volume's anchor brick (the brick holding its authoritative volume) self-bumps the generation on wake. On CP return the fenced-writer anchor-adoption rule trusts a self-bump from the current anchor and backfills it; anything it cannot prove is quarantined. A first form of Fork B is Built: every wake grants the anchor a durable bounded blessing lease (the blessing_lease_granted op: start and lease_end generations, reloaded on boot) that pre-authorises the advance. Decided direction (Fork B remainder): lease expiry, moving toward a steady-state lease with brick-owned idle-bank.

The advance changes who may issue a generation, never who may write a volume (invariant 6).

That fence is also what makes anchor-loss recovery safe (ADR 040). When a brick is confirmed gone and the volume has an export, the control plane restores it onto a live peer and re-anchors there, writing node_id alone: the blessing remains the sole generation authority, so a stale copy on a node that later returns carries an older generation and loses its pair check rather than competing for the write. The anchor pin from standing decision 11 is unchanged for a volume that exists on exactly one disk, which is still the case the fence was built for.

Wake path and the node-local activator

A request to a scaled-to-zero workload lands on a fallback endpoint, parks, and triggers a single-flighted wake; the real endpoint is then published and bytes splice. The activator (L7 serving, L4 stateful/composite) belongs in noded rather than the CP pod so that a CP Recreate roll cannot black-hole cold wakes: stable DNAT inside noded's own netns, NodeStatus advertises the noded pod IP and activator port as the activator endpoint, EndpointPublisher renders it as the fallback, and instance ids minted node-side carry origin: ACTIVATOR for the CP to adopt and backfill on reconcile. Planned: partially landed, soak ongoing.

Each published stateful endpoint shares its EDS assignment with a per-workload L4 activator fallback at the next Envoy priority. Serving workload clusters instead reuse one shared L7 activator endpoint, with the injected x-ember-workload header disambiguating the workload. Active TCP health checks eject unreachable live endpoints, so traffic falls through to the activator and returns to the live priority when it recovers. On the stateful L4 path, a connection that reaches the activator withdraws the stale resident health fact before starting the ordinary rate-limited wake, unless it is within the five-second publish cooldown or the registry's retained node inventory reports the exact VM with healthy: true and last_probe_unix_ms no older than one two-second status interval. In either guarded case, the activator treats the failover as a transient and splices to the published endpoint.

For serving, Envoy's default overprovisioning factor makes the priority shift fractional, so one unhealthy host out of two can still send some traffic to the activator. The node-local serving straggler lookup only splices to an endpoint whose probe is healthy and no older than its configured probe cadence plus the bounded probe timeout; otherwise, and only when no bank or active splice owns the entry, it withdraws and tears down that endpoint before following the ordinary bounded wake path (#5818). A bank-owned or actively spliced entry is left for its operation, and the witnessed-connect withdrawal above remains stateful-only.

Why Fork B stays unbuilt. ADR embervm/018 split brick-owned lifecycle into Fork A (node-local wake, shipped) and Fork B (brick-authoritative idle banking, generation advancement under a wake grant, checkpoint COMMIT on the node). Fork A answers the availability question that motivated the split: a request during a control-plane gap reaches a ready VM. Fork B would move generation authority off the control plane, the seam ADR embervm/011 made single-writer on purpose, and no workload has yet needed idle bank or relight through a sustained control-plane gap, nor has control-plane sweep load been measured as a problem. Fork B therefore waits for one of those two facts and starts with one bounded transition (ownership, authority, persistence, recovery stated), not the seven-phase plan in the ADR history. Wake-path retirement (#4013) is gated separately on the #4702 model and runtime evidence and does not imply Fork B.

Sessions: the durability ladder

Resume is one interface with four verbs: cold boot; base-snapshot restore; warm (memory) restore; base + workspace hydration. The CP picks the cheapest unexpired artifact. Two different windows apply: the same banked session is resumable for its warm-bank TTL (an hour in the agent lane, a deploy value, and bounded strictly below the warm-bank GC TTL: the chart render and the CP's Workload admission both reject a bankedTtlSeconds greater than or equal to warmthS3Gc.sessionTtlMs, because a banked snapshot has no expiry hold in that GC), after which resume takes the session-expiry 410 path; the durable workspace lineage is adoptable for 7 days, so a new session generation inherits the prior workspace rather than starting blank. Decided direction: capture decouples from bank (close-triggered for no-memory-snapshot workloads), and the workspace size cap becomes a declared soft budget. persistence.filesystem.retention is inert today, and artifact GC keeps only the newest artifact. latest + N remains the planned retention direction.

Repo-backed sessions hydrate their workspace with a direct HTTPS clone through the egress lane. The node-local git mirror sidecar (gitMirror) is Built but off in every deployment and has never served a hydration: the shim's mirror attempt is denied at the egress proxy and falls through to the direct clone.

A sequential Firecracker dirty-page diff bank path exists and is dormant: noded.diffBanking defaults false because the merge-at-bank step copies the previous full base on reflink-less ext4 scratch, so the Bank RPC gets longer, not shorter. Enable it only after an in-place merge or reflink-capable scratch lands (#4970, #5699). Only full bundles ever enter the archive.

maxLifetimeSeconds is a version-convergence bound, not a data lifetime: it stops a session riding a stale base forever and is never raised to buy continuity. Continuity comes from adoption: lineage is decoupled from session generation, so a later generation inherits the prior lineage's workspace.

Parked sessions count as disk, not against concurrency.cap: a parked session holds zero RAM, concurrency.cap bounds running VMs only, and wake does not re-check it; placement's memory admission and the per-principal wake-rate limit protect the receiving node.

The S3 artifact GC uses an 8-hour TTL for stateful warmth and 7-day TTLs for session memory, serving snapshots, session workspaces, and group sets.

Why. A control-plane restart during an interruptible bank could leave a benign generation advance indistinguishable from an unauthorized one, forcing quarantine (ADR embervm/017). A live-volume timer snapshot was rejected because blocks can be captured inconsistently, and keeping wake only in the control-plane pod was rejected because a control-plane roll could black-hole cold wakes (ADR embervm/018, ADR embervm/025). Durable checkpoint provenance and a node-local activator preserve the fail-closed generation rule, accepting a narrow quarantine window when provenance was never recorded and best-effort metering during a control-plane gap.

The control plane does not retry a session invoke, and the guest classifies its own failures. Unlike a task, a session invoke's guest error is the guest's answer, not a VM failure: run_invoke issues one Assign and returns whatever the guest said, and the router proxies that status and body verbatim. Embervm.Retry and a workload's invocation.retry are task-lane only, so nothing in EmberVM retries a session turn. That leaves the guest 422 body as the only place a transient cause can be declared, and the shim is the only layer that can tell one from a permanent one: it raises TransientTurnError and adds retryable: true to the body, and the caller owns the backoff.

Why. A guest 422 is a catch-all: the shim's except Exception maps a provider stream disconnect, a missing config file, a missing binary, and an exhausted usage limit to one status with one body shape. Over the 14 days to 2026-09-16 that was 40 turns, of which 16 were a codex config load failing, 8 a missing muse binary, 11 an unreachable model catalog, and exactly one the stream disconnect a retry clears, so a blanket retry would have doubled the loud failures rather than fixed them. That ratio is also why the transient set is limited to an ALREADY-ESTABLISHED leg dying, never a bare transport generic: a permanent egress fault (a host missing from egressTo, a dead catalog entry denying its host, a broker grant failing closed) emits "transport error" and "error sending request" on every attempt, so matching those would spend a whole ladder to arrive at the same failure. Classifying in the control plane was rejected because it would have to parse guest payloads and still could not separate these causes; retrying in the control plane was rejected because the shim has to classify first either way. The cost of NOT retrying is not one wasted boot: a 422 marks the factory attempt uncertain, and with factoryStopSupervisionEnabled false that parks the run holding a parallel slot and its cost reservation, so the cheap same-session retry replaces an expensive fresh-VM node attempt. The accepted cost is that a codex failure raised after turn/start may already have run tool calls under danger-full-access, and the retry re-delivers the same message to the same live thread, so a partially effectful turn can replay up to the ladder's bound. A header-based control-plane retry remains open (see Direction) for callers that do not go through the monolith transport.

Why snapshot anywhere uses the between-turns safe point (#6256). A provider stream can run for twelve hours, so waiting for it makes routine rollouts take hours and cannot fit a Spot notice. Every drain edge now fences session invoke admission and interrupts the exact active dispatch using the interrupt primitive. An operator stop surface can be added on top of that primitive when there is a caller. The shim stops the provider, preserves its native transcript and partial output in the workspace, syncs the session volume, and returns terminal_reason: interrupted_for_drain. Only that ended turn permits banking; a signaling acknowledgment or elapsed timer never proves a safe snapshot. The flush window is bounded by EMBERVM_SESSION_DRAIN_FLUSH_MS (default 60000) and the remaining drain deadline less the bank budget. The shim reserves another two seconds for sync and response delivery, with a one-second minimum provider wait. An idle guest banks immediately. Deadline expiry is a failure to complete evacuation, never permission to snapshot a running provider.

The control plane journals the interrupted turn sequence and CLI identity before banking. On the next invoke after relight, on any eligible brick, it prepends a continuation hint to the caller's prompt: the previous turn was interrupted for a drain and the transcript and workspace are intact. Factory retains the pending logical turn, graph attempt and reservation, and records the interrupted prefix and its cost. It emits no escalation card or terminal Discord notice. The next physical turn resumes the guest's own state; it does not replay the original prompt. Filesystem-only sessions use the same flushed safe point for park/archive and rejoin. The memory-enabled lane uses the existing memory bank/export path.

5. The invariants

These are the rules everything else rests on. Every design change is judged against them.

1. The hit/miss invariant. A task requires synchronous control-plane admission and assignment. A serving or stateful miss normally requires the control plane; the node-local activator (Planned, partially landed) additionally runs a bounded node-side wake during control-plane absence. Steady-state hits never touch the control plane.

2. Facts through the control plane, payloads never. Snapshot bytes move node-to-store via noded; serving traffic moves through Envoy; the control plane carries facts. The two deliberate exceptions are lifecycle-rate: the parked first request after a serving miss, and task dispatch/results.

3. No mutable VM or snapshot lineage ever crosses a principal. Memory snapshots, workspaces, volumes, and other mutable artifacts never deduplicate across principals (erasure would become a cross-tenant reference-counting problem). ADR 028 defines the narrow exception for immutable, reconstructable rootfs chunks: private chunks may deduplicate within one Account, explicitly published platform chunks may deduplicate globally, and principal-scoped mount authorization remains mandatory. The workload-class table in section 3 carries the per-class network and state boundaries.

4. Fail closed on enforcement, fail open on warmth, and metering is not enforcement. Containment (concurrency caps, fair queues, admission, the node-side pressure predicate, node-confirmed destruction) fails closed. Warmth (a missing snapshot, an unreachable store) fails open to a cold boot: slower, never incorrect. Metering is counting, not enforcement, and fails open by design: a brick out of contact keeps running and keeps counting, and unreconciled spend is written off. Cutting off a principal is an admission action (stop minting tokens, 402 at the edge), not a metering one. The quota gate is model-checked (quota.tla): budgets are opt-in, and for a principal with a budget both the submit gate and the dispatch gate fail closed when the usage cache is unreadable.

5. The node agent is authoritative for instance runtime state. What a brick reports over dial-home about its VMs, taps, and volumes is the truth; control-plane tables are a reconciled cache. The registration route binds that authority to the pod UID and node name claims in the brick's bound ServiceAccount token, so a brick can establish only its own stream. The one carve-out is destruction: an instance is recorded destroyed only after the owning node confirms teardown (gate nodeConfirmedDestroy: true in production and dev since 2026-08-22, #4758), and reconciliation is fail-closed toward destruction (an unrecognised node VM is an orphan to destroy, unless it carries the origin: ACTIVATOR marker, in which case it is adopted and backfilled). Model-checked (adoption.tla) against control-plane and node crashes between any two steps: the dispatch restart wedge, forget-before-kill resurrection, and reap-would-wipe-fleet bug classes are excluded by invariant. The carve-out has its own carve-out: a destroy whose owning node has LEFT the fleet completes as destroyed with terminal_reason: node_gone, both at request time and from the reconcile sweep, so a session stuck in destroying self-heals. A node counts as gone only when it publishes no capacity facts for the session's dial AND the node registry holds no runtime instance for it, or holds only a tombstone. Health is deliberately not part of the test. An exact stop is included, and terminalizes with its stop_completion left null rather than a synthesised tuple, so a caller holding the stop precondition can see that departure produced no proof and decide for itself whether that is acceptable. The same registry-departure proof terminalizes dormant session warmth that cannot wake: a parked row follows its volume_node_id, a banked row follows its node_id, and an expired owner moves either row to evicted. A stale conflicting secondary ID cannot override that state-specific owner. The row remains non-terminal only when a healthy peer positively reports the exact workspace volume or snapshot it needs, or when a banked row has an exact snapshot reference plus an eligible peer that can restore it from reachable object storage. Those are the recovery targets the established wake paths can actually use. Archival and node cleanup remain best effort, and a transient terminal-op append failure is retried without restamping a successful terminal row, so neither can hold irrecoverable warmth indefinitely.

   Why. A replaced brick can never send the confirmation the gate waits for, so before #6004 the session sat in destroying forever and held every downstream reservation bound to it, which is a worse failure than completing a destroy whose VM demonstrably died with its node; departure from the registry is the only sound signal for that, because the registry ages an instance to unknown and retracts its capacity row after five seconds of silence, so any "unhealthy for longer than a threshold" test would also fire on a healthy brick with a brief status gap under a destroy that had been retrying for minutes, and record destroyed for a VM still running.

6. Single-writer for stateful is a physical fact, not a protocol. The storage section explains the raw sparse file, one-writable-attach rule, and generation-as-provenance detail; failover remains a deliberate operator action and the partition window is bounded by the brick silence timeout.

7. Durable-before-observed for the journal. Every lifecycle and enforcement action is an ordered op-log append; callers await their (group-committed) batch. The op-log doubles as the audit record.

8. Kubernetes arbitrates pods; EmberVM arbitrates VMs; the pod is the ABI. No scheduler or autoscaler code is imported and no node provisioning is rebuilt. The capacity section's three-layer table shows the levers, and a Pending brick is the scale-up signal or the fleet-full page.

9. Classes are reuse semantics; substrates are lanes. New execution technologies (Hyperlight, wasm) enter as lanes under existing classes, never as new classes.

10. Users express posture, never mechanics. The surface is a small set of declared scalars in user-facing units, each under a platform ceiling with a non-escalation rule. Priority rungs, disruption budgets, grace periods, and CPU are platform-derived; users do not select those mechanics. The scalar details are carried by the storage and control-plane sections.

Why. Worker runtime state changed faster than synchronous durable writes could support, while stale reports could otherwise resurrect destroyed VMs or regress a generation (ADR embervm/014). The optional async path avoids keeping Postgres on four boot and wake appends, but the reference deployment currently keeps asyncLifecycleWrites: false, so those appends remain write-through (see: /projects/embervm/deploy/values.yaml and /projects/embervm/chart/values.yaml). Vocabulary-only conformance was also insufficient because a modeled action can exist in an enum and never be emitted (ADR embervm/034). Worker-authoritative runtime state, forward-only reconciliation, and model-checked invariants support the async path's late durable records while keeping enforcement fail-closed and warmth discardable (ADR embervm/006, ADR embervm/014).

6. Control plane internals

The control plane holds sparse facts and derives dense things on demand; it never stores or witnesses anything that scales with the fleet.

• State model: hot working set in ETS (rebuilt on start, healed by adoption from node reports); durable book-of-record in the op-log behind the Embervm.OpLog behaviour. The reference deployment configures Postgres on the shared CNPG cluster; the chart default and isolated dev deployment use SQLite-WAL. SQLite remains the zero-dependency single-node backend and the reference deployment's rollback path. Embervm.Application.op_log_mod/0 selects between them purely on EMBERVM_OPLOG_DSN being set, so the rendered pod spec names the runtime-selected backend (see: /projects/embervm/deploy/values.yaml, /projects/embervm/dev/deploy/values.yaml, and /projects/embervm/chart/templates/deployment.yaml). Either adapter creates its own schema on boot, so there is nothing to migrate. The dispatch path never reads the durable store.
• Retention: result TTLs enforced at read time; terminal tasks pruned past 7 days; the ops journal prefix-compacted past a 30-day horizon behind a durable compacted_through_seq marker. Long-horizon audit lives in the observability stack.
• Adoption: noded reports primed_vm_ids, session VMs, checkpoint-pending VMs, and banked artifacts on every NodeStatus; the dispatcher and managers reconcile on boot and every sweep. Adoption is ETS-only except the stateful bundle heal, which appends a stateful_banked op marked adopted: true. The protocols are model-checked: the specs in projects/embervm/specs/ run under TLC in the build, so a spec violation is a red build, and vocabulary.exs keeps them honest against the code's enums, though not against whether a modeled op kind is ever appended (#4756) or the gate it needs is armed (#4758). Planned: trace validation, op-log events checked against TLA+ actions. The debug-gated SpecTrace implementation (#4770) ships and runs in dev with specTrace.enabled; production keeps it off. The full harness (hermetic and deployed lanes, direct-checker and TLC tiers, anti-vacuity manifests, DRILL and VACUOUS as distinct verdicts) is Decided direction (ADR embervm/034), tracked in #4761 and #4763.
• Cells: the unit of horizontal scale is a cell, a complete single-writer control plane owning a bounded set of bricks and workloads, with one op-log appender (ordering is within-cell only). Planned (#4753): no cell_id, workload-to-cell assignment, or per-cell dial-home address exists in code yet; there is exactly one control plane today. A thin stateless fleet layer (route + capacity roll-up) arrives only with a second cell.
• Registry survives restarts: noded persists its last-synced registry to NVMe marked stale; a restarting noded with an absent CP serves warm workloads from cache. No dependency's brief absence may turn a warm node into a dead one.
• Bounded node dials: every node dial carries a 3 s TCP connect timeout, and channel dials run in callers so they never block the NodeChannel process (#5124).
• Health surface tiers (ADR embervm/031): /health latches unhealthy immediately on a sustained artifact-export failure streak (tier 1, a user's data at risk now) and only after more than 24 h without a completed warmth GC sweep, measured as the age of the newest gc-manifests/ object (tier 2, maintenance debt with days of runway). Both end in the health surface; Discord alerts are a companion signal, not the record.

Decided direction:

• Op-log restructuring: payloads separate from facts and ops are time-partitioned, making principal-scoped erasure an indexed delete.
• Admission-only control plane: the CP admits and mints encrypted session-routing tokens; peers redistribute under pressure and metering fails open.
• Resource model: memMib is the only declared dial, CPU is derived from it, and GB-seconds is the accounting unit.

Why. The original journal retained request payloads without a bound, and a full durable volume would stop submissions because lifecycle writes fail closed (ADR embervm/002). A larger unbounded volume was rejected because it delays the same outage; central placement and metering on every dispatch were rejected as control-plane throughput ceilings (ADR embervm/002, ADR embervm/020). Projections, prefix compaction, group commit, and forecast-time assignment keep durable work off dispatch. Separating payloads from journal facts adds a second write and an orphan-reclamation path, an accepted consequence of shorter payload retention and principal-scoped erasure (ADR embervm/019).

7. Capacity, bricks, and Kubernetes

A brick is the capacity unit everywhere: a fixed-size noded Deployment pod in a T-shirt size class (2gi through 16gi; small classes pack tasks densely, the largest hold 1-2 serving/session VMs), with honest Guaranteed requests, sized roughly 4-8x the largest VM of its class, a handful per node. The daemon is budget-agnostic: it reads its ceiling from its own cgroup, so a size class is a resources block, not a code fork, and a brick's size is fixed for its lifetime.

Why. The reserve constants are provisional and measured only once: daemonReserveMib (256 in the chart, 512 in the binary default when the chart does not inject it), vmOverheadMib (0 until a brick is switched to reserved), and the roughly 3 MiB per guest the overhead script reported on the first hub samples. Stats() is read once at release, so there is no recurring per-VM RSS series and no separation of daemon working set from guest RSS from shared page cache. That is deliberate: a sampler is cheap to write and expensive to keep honest (page-cache first-toucher misattribution, cgroup v2 no-internal-process rules, cardinality), and no sizing or incident decision has yet needed a number the existing facilities cannot supply. Recurring per-VM telemetry (#4773) is added when a named decision needs it, scoped to that decision, and never changes the reserve, admission or banking policy in the same change. The B5 shadow ledger (#6056) is the next measurement that exists by design.

On a fixed fleet a Pending brick is the fleet-full signal: the controller flags :fleet_full, the dispatcher refuses placement (503), and a human is paged, rather than overcommitting.

Built: every brick and the serving relay run a disposable-shaped priority class (homelab-disposable, or homelab-preemptible on GKE, see below), so guests are the first to yield under node memory pressure; per-workload arbitration happens only in CP dispatch. QoS is Burstable: CPU carries no limit (request-only, sized as a CFS fair-share weight rather than a reservation), and memory requests sit at roughly 75% of limit (PR #5519, merged 2026-09-01) since Firecracker guest RAM is demand-faulted and a brick's resident set runs far under its configured ceiling; the limit itself is untouched and still sets usable_mib, so no guest's admitted capacity changes (ADR embervm/039).

Decided direction (ADR embervm/039, from the 2026-09-01 capacity incidents): overcommit blast radius is a ladder, guest (Firecracker jailer per-VM cgroup, memory.oom.group) before brick (kubelet eviction of the lowest-priority pod) before node; and unsatisfiable demand (no class fits, or class and pool headroom are both exhausted) becomes one signaled capacity condition on /health and Workload status, held with hysteresis, rather than a retry at request frequency (today only the :fleet_full 503 exists). Pressure shedding is Built, disarmed (#5521): a brick sheds its least-recently-active idle guests via the existing bank paths once its observed memory crosses statefulSweeper.pressureBanking.highWaterFraction (0.85), with hysteresis back to the low-water mark (0.70) before shedding stops; the lane is off in every deployment pending live verification. homelab-preemptible (-9, platform chart) replaces homelab-disposable (-1000) on GKE bricks specifically so a Pending brick stays visible to the cluster autoscaler's scale-up (above its -10 expendable-pods cutoff) while remaining first-evicted; this reopens, and only partially answers, the mass-VM-death preemption concern ADR embervm/016 raised against below-default brick priority (scheduler preemption deletes rather than drains). The Firecracker jailer that backs the guest rung of the ladder is Shipped-but-suspended (#5520; section 10 has the containment detail). It lands dark pending live verification on one brick, after which a values-only change can arm it.

Why. Host memory usage cannot say whether a guest is reclaiming hard or has already killed its application: a guest OOM leaves the Firecracker process alive, a host cgroup OOM kills the VMM. Today noded classifies only the second (cgroup.OOMKilled() at process cleanup); everything else is an unclassified VMM exit and a guest application death is invisible until the workload's own health signal notices. Host-observed capacity stays authoritative: a guest report can never raise capacity, bypass a quota, or relax idle-only banking, and a missing or stale report is neither healthy nor proof of OOM. A guest feedback channel (available memory, PSI where the kernel has it, OOM evidence, freshness tied to the current VM activation rather than a snapshot-cloned flag) is Planned (#5805) behind a named diagnostic gap, because it means a new command on the frozen guest-agent contract and a rootfs rebake for every runtime. The first slice when the gate fires is exit classification alone (host cgroup OOM, guest kernel panic from the serial marker, unclassified) surfaced as a structured lifecycle event, which needs no guest change.

Decided direction: PriorityClass ranking of brick pools by lane with sacrificial balloon bricks for burst headroom, and remaining-node-lifetime as a placement input so a terminating node takes only work that fits its horizon. Disruption splits workloads into preemptible posture (task, isolated lane) and durable posture (session, stateful: continuity via banked-state durability, not node pinning).

Brick discovery and scale: bricks are the only source of node capacity, so the control plane picks the brick mix from placement demand rather than running one fixed daemon per node; discovery is dial-home, never a Service or a per-node DaemonSet. Desired per-class counts are a values knob reconciled by Embervm.BrickController through /scale, because ArgoCD ignores /spec/replicas fleet-wide and git-declared replicas would not sync; instance identity is the kubelet pod UID. Brick autoscale runs at rung up on the observe -> up -> full ladder: denial-driven scale-up acts, clamped to the chart's maxReplicas. Planned: promoting to full adds drain-aware scale-down. The current brick mix is deployment state and lives in the fleet section.

Decided direction (ADR embervm/042, Accepted 2026-09-05, not yet built): the maxReplicas ceiling itself becomes denial-driven, not only the replica count clamped inside it. A class stuck at a ceiling of 0 today needs a human to raise it by hand (the values-gke.yaml edits behind #5503 and #5504); instead, a new ceilingBound values field names the operator's outer bound per class (absent still means never, preserving today's explicit-zeros contract), and sustained denial pressure for a class below its bound raises the ceiling one step, unblocking the existing scale-up decision on the next tick. Scale-to-zero is symmetric, on a longer idle window than the replica-level one, and reuses the same drain-aware victim selection. maxReplicas in git becomes a bootstrap floor read only at first render, the same ownership shift #5498 already made for replica counts. Unsatisfiable-forever demand (no class fits at any bound, or a class pinned at its bound and still denying) rides the existing decision-6 capacity signal (ADR embervm/039) with two new reason codes rather than a second alerting path. This does not reopen in-place pod resize, which ADR embervm/013 section 7 and ADR embervm/039's own alternatives already declined to reopen. Implementation is tracked on #5505.

The availability contract is spot semantics with two budgets. The chart default gives a routine roll, upgrade, or scale-down 110 seconds of drain notice. A GCE Spot preemption supplies about 20 seconds through noded's preemption-notice watcher (drain.preemptionNoticeEnabled, armed on the hub since 2026-09-06, #5819, off in the chart default). Both edges publish the same drain_deadline_unix_ms signal. Sessions fence admission, interrupt the active turn, give the guest a bounded window to flush progress and state to the workspace, then bank and export at the existing between-turns safe point. They no longer wait for the twelve-hour invoke bound. New and queued invokes receive retryable drain responses until bank completes. A failed bank retains the fence. Relight resumes as the next turn on the same factory attempt and reservation. The deadline still bounds evacuation: node loss, a failed flush, or an export that cannot finish before it can lose progress since the last durable archive. The GKE hub overlay sets a 150 second drain (180 second termination grace) so the budget covers flush plus bank rather than a whole factory turn; the brick progress deadline stays the chart default because a cold rootfs rebuild still needs it. State durability within the stated archive interval is the guarantee, connection continuity is not. Artifact retention TTLs and the GC sweep behaviour are in deploy/README.md.

An ordinary brick rollout closes placement at the node drain edge and closes session invoke admission before evacuation. Closing new admission is not a drain by itself: guests that were already running remain on the old brick. A session therefore rejects queued and later invokes retryably, lets its one current invoke finish, and asks to bank immediately afterward. Temporary bank admission failures keep the fence closed and retry; a failed bank RPC restores the fence when the session process resumes. The old pod remains terminating until its guests leave the live registry or the externally configured deadline expires. This means one long factory turn can delay its GKE brick replacement for up to the configured 43,800 second bound. Sequential sync waves can compound that delay across a full fleet rollout. The Deployment progress deadline is longer than one termination grace plus a cold rootfs build. Promotion tooling with a shorter wait budget can still report a timeout while Kubernetes is gracefully terminating the old brick; recovery must resume observation and must not force-delete that pod.

That protection is intentionally limited. Spot preemption, force deletion, node loss, and expiration of the drain deadline can still destroy a running guest. Temporary control-plane or bank unavailability is not evidence that a guest stopped, so it never authorizes a replacement attempt. When EmberVM has durably recorded failed with terminal reason brick_gone, factory supervision may settle only the exact matching guest invocation and use only the graph attempt's remaining bounded retry. Guest, generation, invoke stamp, or node mismatches remain uncertain, as do outcomes with ambiguous external effects. No retry is admitted from absence or temporary unavailability alone.

Why drains bank in place rather than hand over. A drain edge is one signal (drain_deadline_unix_ms, SIGTERM or the Spot preemption notice, whichever is earlier) and the coordinator answers it one way: force-bank every class in durability order inside the deadline. A pod-surge roll keeps the node's disk, so the replacement daemon can relight locally; the bank also exports its durable artifacts before shutdown when the store is configured. A Spot preemption gives about 20 seconds, enough to bank and export but not to export, restore onto a peer, re-anchor, and evict, so pre-drain redistribution would race the deadline for no availability gain; when the node is gone, anchor-loss recovery (ADR embervm/040) restores the banked volume onto a peer from the store. The manual handover verb (POST /v1/stateful/:name/handover/:target) exists for the planned case: a node leaving the fleet on an operator's schedule. Automatic handover at the drain edge would need a drain cause the daemon does not publish and a workload whose downtime budget banking cannot meet; neither exists.

Why. Per-invocation pods and Kubernetes objects made pod churn and etcd the ceiling for short work, yet bypassing Kubernetes meant its autoscaler could no longer observe Firecracker demand (ADR embervm/001, ADR embervm/005). A pod per VM was rejected for restoring that churn, while whole-node daemons were rejected because they make capacity changes too coarse for mixed workload lanes (ADR embervm/013, ADR embervm/016). Fixed-size bricks expose schedulable capacity to Kubernetes and keep VM placement inside EmberVM, accepting bin-fragmentation and drain coordination as explicit costs.

8. Storage, artifacts, durability

Artifact model: one typed verb family, ExportArtifact / RestoreArtifact / EvictArtifact, over ArtifactRef {kind: BASE | SESSION | SERVING | STATEFUL | GROUP_SET | VOLUME}. Control-plane-driven, idempotent per key; evict refuses while referenced, and base retention holds when the current base is unverified (#4401). Keys are namespaced by workload (and vendor, below); a principal-scoped shared/<principal>/<sha256> keyspace is Planned (#5075), not implemented: ArtifactRef is {kind, workload, ref} with keys <kind>/<vendor>/<workload>/<ref>/<file>.

Baked rootfs cache (Built, #5772): Baked guest rootfs files use rootfs/<digest>/<payload-sha256>.ext4, with rootfs/<digest>/rootfs.ext4.sha256 written last as the completeness marker. The marker names the payload key and checksum. A local miss reads the marker first, downloads that payload, and verifies its bytes, so bases hydrate across nodes under the identity contract, with a local hardlink cache in front.

Planned rootfs plane (ADR 028, #4182): OCI images convert to deterministic flattened EROFS manifests and immutable chunks. Private chunks deduplicate under rootfs/account/<account>/...; allow-listed published platform chunks may use rootfs/platform/.... A brick fully hydrates and verifies every chunk of an active manifest before reporting the rootfs READY, then presents a local-only read-only ublk device. The object store is a preparation dependency, never a live guest block-read dependency. Phase 0 (rootfs/PHASE0-RESULTS.md) measured a 16.8 percent Account-scope saving with Gear CDC at 64 KiB / 256 KiB / 1 MiB and showed -Enoinline_data is required for rebuild stability; the build waits for EKS because converter placement and Account key custody are its prerequisites, and the two larger wins it uncovered (superseded-rootfs reclaim, dropping unused baked images) need no chunk store.

State durability within the stated archive interval is the guarantee. The matrix defines the current recovery and the gaps in automated recovery.

Vendor pinning: Firecracker memory snapshots restore only within a CPU vendor (and a narrow intra-vendor matrix), so all warmth artifacts are keyed by (vendor, template) and never cross the boundary; the daemon refuses a vendor-mismatched restore loudly. Volume data is fully portable. Legacy artifacts cut before stamping existed are grandfathered: restorable on their home node forever, never distributed.

A base snapshot is also bound to the exact rootfs it was captured on: noded records the rootfs ext4 UUID at capture and refuses a restore, sibling adoption, or prime against a mismatched or missing identity, moving an in-use mismatched bundle aside rather than resuming a guest onto a filesystem it never saw (#5674). A rebaked rootfs after a node replacement therefore forces a fresh base build on that node unless the baked rootfs cache above serves the same bytes (#5772).

Principal artifact envelope encryption (Built, armed in dev and production, chart defaults off, #4691): each mutable principal artifact is zstd-encoded then AES-256-GCM chunk-framed with a per-file nonce; meta.json carries the opaque control-plane envelope and SHA-256 stays over plaintext. An enveloped restore needs a five-minute capability minted by the control plane for the exact target brick, HMAC-bound to the (principal, lineage, brick, workload, ref, kind, generation) tuple; legacy plaintext restores unconditionally, and a malformed or below-floor envelope suppresses the restore so the cold-boot fallback runs. Root, epoch, and custody transitions rewrite envelopes without reading payloads, by a bounded control-plane sweep and a lazy noded pass after access, both guarded by an exact-ETag compare-and-swap so a newer export always wins; raising an epoch floor or retiring a root stays a separate operator action. Arming order and flags: kekRoot, EMBERVM_ARTIFACT_ENCRYPTION, store.encrypt, requireRestoreCapability; mechanics in noded/server/store.go and control/lib/embervm/envelope_rewrap_sweeper.ex.

Decided direction:

• Local disk is authoritative: local node NVMe relight needs no network.
• S3 is a zstd content-addressed archive, written at bank commit, not a hot tier.
• archiveInterval is the user-facing durability control in acceptable-loss units.
• Failover and node rotation are deliberate operator actions; the banked-volume handover RPC that implements them is Built (POST /v1/stateful/:name/handover/:target).

Ownership arbitration is class-scoped (Decided direction):

The divergence bound is the brick silence timeout: a brick that has not heard from the control plane for longer than the timeout (~6h, in the grant-expiry range, so a CP roll never trips it) stops serving everything it holds. Token TTL is a convenience, never the correctness parameter. Built (ADR 037, #5073): noded tracks last control-plane contact from its dial-home Register responses and WatchNode stream activity and refuses new node-local work (activator wakes, group wakes, blessing-lease self-advance) once silence exceeds the bound; live VMs keep running and warmth stays intact, and blessing-lease exhaustion remains the generation-count complement for quiet lineages.

Why. ADR embervm/011 chose replicated block storage to avoid whole-volume exports, then ADR embervm/025 superseded it for stateful storage because the CSI path added another placement owner and imposed the same replication cost on every workload; the reasoning against whole-file write amplification still holds. Timer-driven copies of live volumes were rejected as crash-inconsistent, and network demand-loading of root filesystems was rejected because an object store outage could stall a running guest (ADR embervm/025, ADR embervm/028). Authoritative local volumes, bank-time archives, and eager local chunk hydration accept an archive-interval loss window and slower restore after node loss. Mutable warmth is encrypted per principal because storage ACLs alone expose process memory to a compromised storage reader, accepting that loss of the platform root forces cold boot (ADR embervm/033, ADR embervm/036).

9. Identity, tenancy, security

Only principal ships now: it is the TokenReview-authenticated caller identity, and the op-log's existing tenant field is a deployment constant occupying the Account slot. Planned (#5072): domain (env or grouping within exactly one principal, so a same-domain-by-default binding can never cross the isolation boundary), and shared platform definitions (such as the sandbox-session and scan-fleet templates) owned by a reserved platform principal with an explicit broad instantiation grant, the widest and most-reviewed grant in the system. Neither exists in code today.

Embervm.KeyService is the platform key custodian (ADR embervm/036): it derives per-principal, per-epoch KEKs on demand from one current root and stores only the current and minimum accepted epochs, whose floor is the revocation fact; one previous root may coexist during rewrap. Customer-managed principals instead use a Secret-configured HTTPS KMS oracle that keeps the KEK and returns a data key plus an opaque wrapped key, so disabling the customer key or grant makes warmth unrestorable and restore-on-miss degrades to cold boot; a custody switch declares mode plus transition_from and accepts the old custody only in that window. No customer oracle is configured in the reference deployment, so that mode is available but inert. Mechanics in control/lib/embervm/key_service.ex and customer_kms.ex.

Decided direction:

Hierarchy:

Account      billing / grouping, NO isolation semantics
 └ Product   grouping, NO isolation semantics
    └ Principal   THE isolation boundary
       └ Domain   env or grouping within exactly one principal
          └ Workload

Definitions at scale: one product template plus N enrollment records, one Git CR per product expanded into CP definitions, and idempotent registration as a desired-set reconcile.

Credential handling (class taxonomy Decided direction; the brick-local proxy's egressTo-gated header injection, sidecar-only key material, and the TLS-MITM CA lane are Built in the reference deployment): material may sit where it can be stolen only if the platform can kill its validity on demand; otherwise the request moves to the credential. Secrets are classed: derivable short-lived (class 1, may enter PLATFORM-TRUSTED guest classes only, revoked at bank), fixed-but-rotatable (class 2, brick-lease only, injected at the egress hop), fixed-manual (class 3, never leaves a central key-sharded swap tier). A UNTRUSTED workload guest never receives any credential class. The brick-local proxy on every guest egress reads the plaintext request, sets the configured header to the real value (mounted only in the sidecar), and originates a fresh verified TLS connection onward. Injection fires only when the destination is in that secret's egressTo, so the credential is unreachable at every other host (projects/firecracker/substrate/egress-proxy/cmd/swap.go). Header injection is used rather than placeholder substitution because a guest-controlled placeholder can be spliced into a URL and reflect the credential into a request line. RAM scrubbing before snapshot is rejected as a mechanism to rely on; revocation at the validator is the control.

Muse 1.0.3-R2198.1 was verified on 2026-09-10 to trust only its bundled roots. Its adapter therefore uses an http:// base URL on the sidecar's plaintext lane rather than the CA-backed HTTPS interception lane: the guest to sidecar hop is a host-local vsock, the sidecar injects the credential there, and it originates verified TLS to Meta itself, so nothing leaves the brick in the clear.

Token broker (projects/embervm/tokenbroker/, Built): the single owner of every mutable OAuth grant. A catalog credential marked with a broker grant is resolved by the egress sidecar as a short-lived access token fetched from the broker rather than from a static Secret, so a subscription refresh token (Codex ChatGPT) or an authentik service-account grant rotates in one place and never enters a guest. The grant Secret's data is broker-owned and ignoreDifferencesd in the Application.

Grant pools (Built, armed on the hub with codex-cluster and codex-b since 2026-09-10): a catalog entry may name brokerGrants, an ordered pool of grants on one provider, and the sidecar picks the active grant per connection from the broker's per-grant quota views: remaining quota in 25% bands taken from the worst unexpired window, the current grant kept until another is a full band ahead, ties to the sooner reset, an exhausted grant out of the ranking until its reset, a grant the broker cannot mint for skipped for a minute. Quota reports carry the grant they were served with, so the broker keeps a per-grant view beside the per-provider one and the monolith's model pools judge a class by its least-used open grant.

Why. The sidecar is one per brick pod, shared by every guest on it, and nothing per session reaches it: choosing an account per session would need the control plane, noded and the proto to carry it. Choosing per connection with hysteresis gets nearly the same cache locality (a brick stays on one account for long stretches and each account burns down in quarter steps) for no cross-service plumbing, and the occasional cache miss on a flip is cheaper than that plumbing. See #5974.

Agent MCP lane (Built on the hub): the guest half of the agents tier described in projects/mcp/ARCHITECTURE.md. The shim writes a strict MCP client config from a URL delivered by kernelBootArgs as an ember.env.* token (the only values-driven way a guest-read setting reaches the shim; initEnv is a base-signature input only). The sidecar's catalog entry for that destination resolves the broker's agent-mcp grant, opts into plaintextUpstream because the tier is an in-cluster plaintext listener, injects on injectAlwaysPaths /mcp and /mcp/ because the codex client sends no Authorization header, and serves one request per connection because the tier's 5 second keep-alive otherwise killed the first call after idle. The upstream therefore sees the broker's identity rather than the guest's, the gap phase 4 of the SPIFFE program below closes.

Public surface hardening (Built): the public routes are scoped at their HTTPRoutes, with serving routes additionally constrained by node Envoy authority matches. Node Envoy returns 404 for /shim/* before routing to a serving workload, so the workload handler never sees the reserved prefix. Inside the guest, hydrate is build-only and the shim refuses another hydrate once it is ready. The pattern's proof is a public Bazel-query demo that serves each visitor query from a disposable CoW clone of a warm-Skyframe snapshot: server-controlled argv, zero egress, reaped per request.

Guest identity: a guest never holds a cluster credential (Built, by construction: no NIC, no mounted ServiceAccount); the platform holds real credentials and acts on the guest's behalf through the brokered egress path.

Decided direction (ADR embervm/041, #5706): one self-hosted SPIRE trust domain, embervm.jomcgi.dev, issues every EmberVM identity. Platform components (control plane, noded, egress proxy, token broker) get X.509-SVIDs registered as ClusterSPIFFEIDs, and the CP-to-noded hop moves to symmetric mTLS with credentials delivered as files by a spiffe-helper sidecar and read at dial time, on second listeners beside the plaintext bearer ones so each side flips independently before the bearer retires. Guests get per-principal JWT-SVIDs (spiffe://embervm.jomcgi.dev/ember/principal/<p>, 5 minute TTL) fetched by noded as the node agent's delegate and delivered over vsock at boot and every relight, never written to scratch or into a snapshot; the egress proxy then authorizes by (principal, host) and the MCP lane validates the guest's own SVID. GCP credential federation follows. State: SPIRE is live with the platform registration entries and no EmberVM consumer (phases 0 and 1); platform mTLS (phase 2) is in flight; guest identity, request-scoped injection, and federation (phases 3 to 5) are Planned.

Decided direction: GitHub leaves the agent egress catalog. Host-keyed injection bounds which host a credential reaches, never which request reaches it, so a prompt-injected guest can shape any GitHub API call and have the token attached to it. Agent principals instead reach GitHub through MCP tools whose URLs name the target repo, entitled per identity-provider group and backed by a per-group fine-grained PAT, so what a guest can reach is a fixed tool set rather than an API surface. The egress broker keeps the credentials that genuinely need host-keyed injection, the model providers among them.

Why. The original Kubernetes identity string conflated the credential holder, the isolation principal, and permission to use management actions, leaving object ownership checks unsafe for additional users (ADR embervm/032). TokenReview-only identity was rejected because it has no browser or device flow, while per-request identity-provider introspection was rejected because it adds a network dependency to every request. Provider-neutral actors, explicit principals, local token verification, and object-level authorization accept expiry-bounded revocation and loss of new login during an identity-provider outage (ADR embervm/024, ADR embervm/032).

10. Threat model

• Built: Firecracker boundary; no NIC for task/session guests; principal-bound lineage; no cluster credential in guests; Envoy-only serving ingress; bound dial-home registration identity.
• Accepted risk: privileged noded with /dev/kvm; external-allow guest egress via the broker; taint optional; CPU side-channel between co-resident tenant guests on the same brick, unmapped (#5255).
• Shipped-but-suspended (ADR embervm/039, #5520): the Firecracker jailer provides per-VM chroot, uid/gid drop, and cgroup containment, closing the direct-root-exec gap when enabled. The chart lands it dark (noded.jailer.enabled: false in every deployment) pending live verification on one brick. --new-pid-ns is deliberately omitted because a PID namespace breaks execProcess Wait and Pid supervision semantics.
• Planned (ADR embervm/041 phase 2, in flight): X.509-SVID mTLS on the CP-to-noded hop and the token broker, retiring the static bearer.

EmberVM evaluates itself against the threat model published by agent-substrate/substrate at its cited 2026-06-25 revision, adopted as the external conformance frame (ADR embervm/033): the most complete public enumeration of what a multi-tenant agent execution plane must defend. The tables below condense it (their actor is Ember's guest, their worker pod a Firecracker slot on a brick, their atelet noded, their snapshot Ember's warmth artifact; threats are numbered 1 to 43 in upstream order as of its 2026-06-25 revision). Shared-worker threats with no Ember analogue are answered by the "no reuse across principals" row. Five threats remain unmapped and bind on Ember: 8 (template-author reach into storage), 21 (noded runs privileged with /dev/kvm), 26 (policy propagated out of band with scheduling), 31 (image-extraction resource limits), and 42 (detection integrations).

The boundary in one picture: what untrusted code can reach, and what never crosses toward it.

graph LR
    subgraph vm ["Guest VMs"]
        G1["task/session guest<br/>vsock only, no NIC"]
        G2["serving guest<br/>tap NIC"]
    end
    subgraph brick ["Brick: trusted host"]
        N["noded"]
        P["egress proxy<br/>(holds credentials)"]
        E["node Envoy"]
    end
    subgraph cp ["Control plane"]
        F["admission, facts,<br/>op-log audit"]
    end
    S[("artifact store<br/>SigV4 per-identity access, gateway enforces (#4708)<br/>per-principal encryption enabled per environment (#4691)")]
    X["external hosts"]

    G1 -- "vsock only" --> N
    E -- "DNAT into tap" --> G2
    G1 -- "plaintext egress" --> P
    G2 -- "plaintext egress" --> P
    P -- "fresh TLS, credential injected<br/>only for allowlisted hosts" --> X
    N -- "facts (dial-home), bound pod identity;<br/>bearer token + ingress policy enforced" --> F
    N -- "snapshot bytes, never via the CP,<br/>SigV4 per-identity access, gateway enforces (#4708)" --> S

Trust diagram legend: every edge is a current path.

Attacks from guests

Attacks from clients and the internal network

Attacks from nodes and insiders

Why. Runtime isolation did not protect banked memory from a compromised brick, storage reader, or insider with object-store access (ADR embervm/033). A self-authored threat list was rejected because it could be shaped around existing choices, and storage ACLs alone were rejected because they authorize more than a specific principal, lineage, and generation. The external conformance frame and per-principal envelope encryption make those gaps explicit, accepting restore latency, key-count growth, and cold boot when a key is unavailable (ADR embervm/033, ADR embervm/036).

11. Deployment shape

Minimum platform contract

EmberVM asks for the platform contract above. The reference deployment uses KVM-capable nodes, a local scratch disk per Kubernetes node, and an S3-compatible object store.

• Scratch is a node-provisioning contract: every FC-labelled Kubernetes node bind-mounts its real device at /var/lib/embervm/scratch (hostPath type Directory fails closed if unsatisfied). Karpenter instanceStorePolicy RAID0 satisfies it on EKS; local NVMe elsewhere. Where no out-of-band bootstrap exists, the chart's scratchPrep DaemonSet provisions a size-capped ext4 loop file at that path (the GKE hub). Bases under it are node-shared across co-located bricks. Scratch does not survive a Spot node replacement: every guest rootfs rebakes in the brick init containers and the control plane re-drives the dropped bases without a restart, about ten minutes end to end.

  Why. Scratch is deliberately ephemeral. A per-node persistent disk would have to be named, attached and fenced across Spot replacement in an autoscaling pool, would carry the ext4 loop file's generation marker past the node that wrote it, and would cost more than the Spot saving it protects on a three-node pool. Preserving the filesystem also proves nothing about a database's recovery point: demo-postgres durability rests on its export path, not on scratch surviving. The reference design therefore pays for replacement with reconstruction: the rootfs cache (#5772) shortens the rebake and the control plane re-drives bases without a restart. Persistent scratch is reconsidered (#5773) only against a measured post-cache replacement that a named workload cannot tolerate, or a workload that declares local data must survive with a stated RPO and RTO, and even then reconstruction or export recovery is compared first. The filesystem choice for any such disk follows #5699.

• Warmth never crosses a CPU vendor (until CPU templates land): artifacts are keyed per vendor, the gate fails closed at the daemon, and each vendor pool holds its own warmth. noded.warmRestoreWithVolumeClasses arms uniformly across the size classes, never partially, per the rollback contract; measured load-to-resume is 2.5ms on both vendor pools.

• The control plane runs one replica with strategy: Recreate; the reference deployment configures its op-log as a Postgres database (a shared cluster is acceptable: a CP outage is a designed-for state, and CP rolls are the availability events the node-local activator exists to survive).

  Why. One replica is a choice, not a limitation of the op-log: the Postgres tier is live and the CP's state is a reconciled cache of what bricks report, so a second replica could rebuild the same hot set by observing. What is not safe is the action side. claimForAssign already fences duplicate assignment at the node, but Destroy, Bank, StopStateful and Relight carry no ownership epoch on the noded RPC surface, and EndpointPublisher versions are boot-epoch prefixed so an older-booted writer is locked out permanently. A CP outage is a designed-for state (pod reschedule plus op-log projection load plus the node-local activator covers admitted work), and no workload or factory operation has yet named an availability or recovery bound that this fails. When one does (#3862), the sequence is fixed: first the fence (a per-brick ownership epoch mirroring wake_grant, and a multi-writer-safe publisher version), then the smallest topology that meets the requirement, which is a leader-elected actor tier on a Postgres advisory lock with observers on every replica. Brick-sharded ownership over libcluster and Horde is a fan-in scaling tool for a fleet of thousands of bricks, not an availability tool, and cells (#3855) were withdrawn. The ADR 007 loop walls (global prime budget, O(V^2) adoption sweep, timer reconciles) each buy more than replicas and are fixed first.

• Small fleets may co-locate guests with control-plane nodes as an eyes-open risk acceptance, provided the cluster is GitOps-reconstructible and durable state lives in S3: quorum loss is bounded downtime, not data loss, and guests are the first OOM victims. Do not import this clause into a cluster whose etcd is precious.

• A hard node taint is a recorded option, not required: co-tenancy runs on honest requests plus a disposable priority class for guests.

The reference deployment's concrete fleet (node roles, current brick mix, shared Postgres) is in deploy/README.md.

Promotion gate

EmberVM promotes the dev chart to production through Kargo: argocd-wait proves the Application Synced and Healthy, a soak interval catches failures that appear after reconciliation settles, and the in-cluster conformance runner's S1 to S4 scenarios (task execution, session sleep and relight, second-session restart latency, control-plane invariants) must all pass at /verdict, stamped with the chart version so an older deployment's evidence is refused. A failed or all-vacuous run blocks promotion; Freight approval is the explicit operator override. Phase 1 is tracked in #5224.

Known walls and provisional numbers (each states what would move it):

Why. Per-workload Helm templates fail as an authoring surface well before a large informer relist makes etcd a recurring hazard (ADR embervm/026). A larger etcd with one CR per workload was rejected because it leaves the earlier authoring failure, and a pod or Service per execution was rejected because it restores the object churn EmberVM was created to remove (ADR embervm/001, ADR embervm/026). Kubernetes therefore owns low-churn definitions and brick scheduling while the standalone chart owns the execution plane, accepting KVM-capable nodes, local scratch, and an external durable store as the minimum deployment contract.

12. Direction

Decided and not yet built, each with the issue that tracks it. A row leaves this table when the work ships or the issue closes without it.

13. Decision history

The ADR files were removed on 2026-09-06 (#4667); git log -- docs/decisions/ has the full text.

Agent-platform ADRs whose decisions shipped through EmberVM. Their status column carries the wording the monolith map uses for the same rows.