ymur.

laug

Ephemeral Temporal cells from a warm pool, for release validation.

What it is

laug (Old Norse for a warm pool, pronounced loyg) keeps Temporal cells warm and leases one in seconds. A cell is a complete Temporal server with its own Postgres database.

laug gate run puts a new server release beside the one it replaces, each on a fresh cell under the same steady load, runs the new one again under a planted fault on a third cell, and returns ADVANCE or BLOCK. The verdict is worked out from the run's files, so any run kept in the repo can be judged again.

laug's map. Above, one lease: a requester takes a warm cell from laug pool serve. laug load and laug fault run on it. Released, the cell is destroyed, and the pool builds warm cells back. Below, one gate: three items at once, each on a fresh cell: the incumbent, the release under load and the release under a fault. The judge reads their files and returns ADVANCE or BLOCK.
Top: one lease from the warm pool. Bottom: one gate. The old release, the new one, and the new one under a fault each get a fresh cell, then the judge gives the verdict. Open the figure

Why it exists

Temporal's promise is that your code finishes even when the machine running it doesn't. I wanted to learn how a release is checked against that promise, from the inside: build a cell, break it on purpose, and trust a gate only once it had shown it could fail a release.

From the 19 stops on the learning page:

  • A Temporal server reports healthy seconds before it can run a workflow.
  • Two frontends started at the same moment race to rewrite one record, and one of them can exit.
  • One of 3 gates blocked 1.32.1 for a database p99 up to 0.40 milliseconds above 1.31.0's, which may be a different mix of calls rather than slower ones.
  • A release can delete a reading: since 1.32, activities run eagerly by default, so the wait before one starts is no longer recorded.

Where it stands

laug is still building. To see how often it blocks a good release, I ran one against itself: 30 gates, all ADVANCE. Thirty gates can only say so much: the false-BLOCK rate is at most 9.5%, with 95% confidence. Across 63 such gates, 4 BLOCKed, each a host's first or second gate after a boot.

To see that it can fail a release, I slowed one: 25 microseconds added to every packet into its Postgres, too little to trip a fixed limit. All 3 gates returned BLOCK.

A gate hands its verdict about 10 minutes after it starts. Two gates had their worker killed partway through, and the next worker handed the same verdict in the same time.

Every gate ran on one host type, with one load, one store and one history process a cell, so no shard has moved. It says nothing yet about several tenants or a region failure.

The repo says how to run it: offline in five minutes, or on a real cell by runbook 2.

What's next

An upgrade in place, several history processes in a cell so a shard can move, and the old release under the same fault as the new one. The learning page's What's next has the rest, with the ten questions the stops leave open.

What would change it

A run that contradicts a stop, on the same release and the same kind of machine. A host where a release run against itself returns BLOCK more often than the bound allows. A release that changes what can be measured, as 1.32 did. The stops follow the runs, and when a run shows otherwise, I'll change the stop.