Engineering

Motivation

The first version ran unattended Claude agents in sandboxed Kubernetes pods. When Claude's terms of service changed around unattended agent use, that dispatch model no longer held. I split the execution substrate apart from the model choice and rebuilt the substrate on Firecracker: every agent request gets its own hardware-isolated microVM, paused and snapshotted when idle so it costs nothing, and restored in tens of milliseconds when woken. The durable pieces from v1 carried over: on-cluster inference, the MCP gateway, the knowledge-graph surface.

MicroVM per request

Every agent request gets its own microVM with its own kernel, so code an agent writes runs behind a hardware boundary.

Snapshot / restore

An idle agent thread is paused and snapshotted (memory plus rootfs), releasing all compute, then restored on the next turn. Measured restore is 28ms cold, 6ms warm; trigger to first model call is ~140ms. A thin Postgres-reconcile controller owns the lifecycle, porting E2B's open-source snapshot architecture onto the Firecracker primitive.

Secret-swap egress

The guest is vsock-only and never holds a real credential. A TLS-terminating sidecar routes by SNI/Host and swaps a placeholder token for the real secret (a GitHub or model API key) at the network hop, so sandboxed code uses credentials it can never read or exfiltrate.

Postgres control plane

High-churn idle-agent state lives in Postgres, keeping thousands of waiting threads off the cluster control plane. The same registry backs the list and resume catalog exposed over MCP.

Local inference

llama.cpp serves Qwen3.8-27B (dense, 4-bit GGUF) on a single RTX 4090 for routine work; frontier models are reached over the swapped egress only where the task warrants it.

Motivation

fc-invoke proved the substrate but hardcoded one workload shape: a stateless invoke. EmberVM is the successor, built by forking the node daemon and putting a BEAM control plane in front of it. A public image renderer serves warm from it, and fc-invoke is frozen with the goose agent as its last tenant. The control plane owns placement and policy and stays off every serving hit path.

Workload classes

task (one-shot, no network card, one channel to the host), session (a stateful sandbox, banked to disk when idle and relit on the next invoke), serving (a warm HTTP endpoint reachable from the edge).

Serving data path

The control plane hands out the routes; requests go from the edge into the VM without touching it or Kubernetes.

Quotas and metering

Fail-closed: a principal with quota 0 is hard-stopped at submit. Metering rides the operation itself, bills success and failure alike, and is queryable at /v1/usage.

Public exposure

One public route, scoped at three layers: the HTTPRoute pins host and path, the node Envoy exact-matches the internal authority, and the guest shim keeps its control endpoints unreachable. 120 requests/min at Envoy plus a 3600 vCPU-second daily quota.

State

A Postgres op-log: a 30-day journal with 7-day terminal-task retention. The current design is maintained in projects/embervm/ARCHITECTURE.md.

Motivation

An on-cluster model breaks each of my notes into atomic facts and stores them so a question pulls back the right one.

Decomposition

An on-cluster Qwen model decomposes markdown into structured facts, with a self-critique pass before anything is committed to the graph.

Embeddings

voyage embeddings stored in Postgres pgvector with HNSW indexes. One database holds facts, edges, and vectors; no separate vector store to run.

MCP surface

Search, notes, tasks, and research-gap tools exposed over MCP, so any Claude session (local or scheduled) can read and write the graph.

Gap research

The graph files research gaps for itself. External questions are auto-researched by agents; judgment calls queue for human review.

This site

The notes view and Cmd+K search overlay on this site render the same live graph through the same API.

Motivation

Loom is a collaborative bet on a governed data platform where a new dataset, domain, or transform adds no new system to run. It treats governance as a safety property: the STPA hazard analysis defines unsafe as a governance violation.

Status

Pre-alpha, built with a collaborator, to be open-sourced. The control-plane library exists; services are in progress.

Control plane

A single Postgres owns queue, catalog, ontology, ACL, and lineage as schema-separated concerns. The job queue is SKIP LOCKED plus LISTEN/NOTIFY, no broker.

Compute

Rust services embed Apache DataFusion. Tables are DuckLake on S3-compatible object storage, with the catalog in the same Postgres.

Wire protocol

Quack: DuckDB-compatible remote access, so existing DuckDB clients connect without a custom driver.

Governance

OpenLineage events plus STPA-derived constraints. The ontology and ACLs live next to the data they govern.

Motivation

HuggingFace models are huge and slow to download. I wanted to reference a model in a pod spec the same way you reference a container image, and have it just work. The operator caches models in an OCI registry and streams them to pods without touching disk.

PodMutator

An admission webhook intercepts pods with hf.co/ volume references, resolves the OCI ref synchronously (pod specs are immutable after admission), creates a ModelCache CR, and gates scheduling until the model is synced.

State machine

Built with Sextant: Pending, Resolving, Syncing, Ready, with a Failed state. Guards distinguish permanent errors from transient failures for automatic retry.

hf2oci

Streams HuggingFace models into OCI layers: HTTP response to tar to io.Pipe to registry push. Zero disk I/O. Safetensors and GGUF formats.

Deduplication

The HuggingFace baseModels API resolves derivative models to their base, so derivatives share OCI layers with the base repo.

volumes:
  - name: model
    image:
      reference: hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF

Motivation

Every operator I wrote had the same bugs: invalid state transitions, forgotten error handling, missing metrics. Sextant defines Kubernetes operator state machines declaratively and generates the boilerplate for their reconciliation loops.

Compile-time safety

Each state is a Go struct and transitions return the next state's type. Going from Pending to Ready without passing through Creating is a compiler error.

Forced idempotency

Transition methods require request IDs in their signatures, which forces you to call the external API and record its ID before transitioning.

Guard conditions

Go expressions embedded in the YAML, evaluated at transition time. Invalid expressions fail at code-generation time.

CI drift guard

Generated code is committed, and CI regenerates from the spec and fails on any drift. The spec next to the operator is always the truth.

Generated metrics

Prometheus counters, histograms, and a state-duration gauge per machine, with automatic cleanup on resource deletion.

states:
  - name: Pending
    initial: true
  - name: Creating
    fields:
      requestID: string
  - name: Ready
    terminal: true

transitions:
  - from: Pending
    to: Creating
    action: StartCreation
    params:
      - requestID: string

Motivation

Every new service meant clicking through the Cloudflare dashboard: create a DNS record, create a Zero Trust application, update the tunnel config. I wanted to annotate a Deployment and have everything provisioned automatically.

Annotation-driven

cloudflare.ingress.hostname and cloudflare.zero-trust.policy annotations trigger reconciliation. No CRDs to manage for the common case.

State machine

Built with Sextant. Pending to CreatingDNS to CreatingZTApp to UpdatingConfig to Ready, each step idempotent.

Finalizers

Deleting the Deployment cleans up DNS records, Zero Trust apps, and tunnel routes. No orphaned Cloudflare resources.

Drift detection

Periodic reconciliation reverts manual dashboard edits. The operator is the source of truth.

metadata:
  annotations:
    cloudflare.ingress.hostname: myapp.jomcgi.dev
    cloudflare.zero-trust.policy: joe-only

Motivation

I wanted an easy way to share trips with friends and family. A GoPro on the dash captures photos automatically, and my homelab turns them into a map they can follow day by day, or replay later. Works for anything with GPS-tagged photos.

Capture

A Python asyncio controller drives the camera over WiFi: GPS-triggered interval capture, a persistent SQLite download queue, and exponential backoff on connection drops.

Ingest

Each geotagged frame is POSTed to a private endpoint gated by Cloudflare Access at the gateway (no app-level key). The server extracts EXIF, derives a trip point, and upserts it. The write path ships only in the private image, so it is unreachable from the internet.

Store

Postgres is the source of truth for trip points; folding trips into the monolith retired the old NATS event store. Image bytes live content-addressed in SeaweedFS.

Delivery

imgproxy resizes originals on the fly and Cloudflare caches at the edge.

Display

A read-only, SSR tier renders the public pages from localhost in-pod and is CDN-cached: MapLibre vector tiles with terrain hillshade, day-by-day galleries, and an elevation-profile scrubber. The public reads never touch the write path.

Motivation

Living near the coast, I wanted to see what ships are passing by in real time. AIS data is publicly broadcast by vessels, but there is no simple way to visualize it locally. This pipeline streams AIS data through my cluster onto a map.

AIS ingest

A background task inside the monolith holds a websocket to AISStream.io, filters to a Pacific Northwest box, and writes position reports straight to Postgres.

Storage

Postgres is the single source of truth. ships.positions is range-partitioned by day. A stateless persister reads affected vessels back, dedups, and upserts a latest-positions serving table.

Ships API

SSR-only REST is reached from localhost in-pod: a snapshot endpoint for the initial render and a per-vessel track on click. Both are CDN-cached with ETag 304s.

MapLibre UI

Vessels render as a GPU GeoJSON symbol layer with directional icons by heading. A separate WebGL layer maps distinct-vessel traffic density per ~500m cell.

Motivation

Finding a good stargazing night means combining how dark a site is with whether the sky will actually be clear once it gets dark. Stars pairs a light-pollution grid of road-accessible dark sites with rolling weather forecasts and surfaces the upcoming hours that are both dark and clear.

Site grid

A light-pollution grid of ~14k road-accessible dark sites is built offline and uploaded to SeaweedFS. A scheduled job wholesale-replaces the stars.sites table.

Forecast scoring

An hourly job fetches MET Norway forecasts for every site and scores each future hour for darkness (sun below the threshold, astronomy via astral) and clear sky (cloud below the threshold).

Metric

The unit is clear-dark hours. Qualifying hours land in Postgres (stars.site_hours); an hourly prune drops hours once their clock hour has elapsed.

Delivery

A wholly public, read-only domain folded into the monolith: a slim SSR payload lists sites with their upcoming windows, the per-site history graph loads lazily, and the page is edge-cached.

Motivation

I got tired of different build commands for every project. I wanted one system that works the same everywhere: laptop, CI, and agents working in the cluster. Everything is vendored, so there is nothing to install beyond Bazel itself.

format

One command runs every formatter, regenerates manifests and lock files in parallel, and finishes in seconds when nothing changed.

Custom rulesets

rules_helm (lint, template, package, OCI-push charts, plus an ArgoCD application macro), rules_wrangler for Cloudflare Pages, and apko image tooling. Each ships a Gazelle extension that writes the BUILD files.

GitOps manifests

Helm charts render through Bazel into the source tree, so PR diffs show exactly what changes in the cluster before it deploys.

Multi-platform images

apko Alpine images from YAML with pinned lock files. One target declares the base, the layers and the push, with the architecture as a flag.

BuildBuddy RBE

All builds run remotely with a shared content-addressed cache. Unchanged code never rebuilds.