Platform

Servers bill by the started hour, so a machine released early is paid time thrown away. Everything above a small permanent baseline is claimed from one pool of hour-billed machines, held until its boundary, and handed to whatever asks next. A machine carries one role at a time and only changes role from idle, through a rebuild that re-images the disk.

Prometheus runs in agent mode and remote-writes to Thanos, which holds the time-series database and the query path away from the node serving the application. Loki takes logs, Tempo takes traces, Grafana serves all three, and browser telemetry ships from the client rather than being scraped. The split exists because the metrics store had become the largest memory tenant on a node with an application to run.

Pull requests run only the checks their diff affects, main runs the full suite before deploying, and migrations run as a pre-upgrade hook so a failure rolls back with the old pods still serving. Visual regression runs Playwright against Storybook. Localisation is a testable concern rather than a retrofit, and missing translation keys report themselves from the browser as a metric.

Architectural decisions are recorded as ADRs, and a retrospective is written after an incident whether or not anyone was affected.

Agent sessions

Internal engineering tooling rather than a customer capability. A session runs an agent against one of our own issues, on a machine claimed from the pool for the duration.

One session per machine, and the reason is the container work. A session builds images and runs integration tests under podman, so packing several onto a host would nest podman inside podman and drop image builds onto a fallback storage driver. An earlier design did pack them, with a supervisor and a scheduler dispatching sessions to live hosts, and I reversed it before it shipped: the density it saved was a fraction of an hour that the allocator was already reusing anyway.

Sessions hibernate when idle. The machine goes back to the pool rather than being destroyed, because the hour has been paid for either way.

A demand-driven CI autoscaler

Self-hosted runners scale on a reconcile loop, and an incoming job triggers one immediately rather than waiting out the interval. That saves the wait, not the provisioning: a machine still takes around three minutes to exist, boot and register.

Jobs are bucketed into demand classes with their own service level objectives, and each class is sized from the sum of its outstanding jobs’ own duration estimates rather than a blended class mean. Classification happens at read time against the live configuration, so editing a class’s matchers or its objective takes effect without discarding the in-flight backlog.

Demand entries carry a TTL. A missed completion webhook would otherwise pin demand high forever; instead the stale entry expires, the next reconcile corrects, and idle machines are reaped.

Server types are derived from live inventory rather than pinned by name. A pinned type was deprecated underneath us, and its apparently obvious replacement was not offered in the datacentre we needed, so a create call simply failed. Deriving candidates from the API and walking them on stockout removes that failure.

Scale-down is hour-aligned, because a released machine still has paid time on it.

Hybrid document retrieval

Three signals, combined by weighted reciprocal rank fusion in a single query: trigram filename matching, Postgres full-text search with ts_rank, and semantic vector search on cosine similarity.

Embeddings come from an ONNX model running in-process rather than behind an API call, stored in pgvector behind an HNSW index. Text is chunked with overlap, so a passage that spans a boundary is still findable. Feeding it is an extraction pipeline that pulls text out of PDFs through PDFium and out of images through OCR, keeping confidence metadata, so a low-confidence extraction is flagged rather than trusted.

Embedding generation runs on its own pod role, so inference does not compete with request handling.