LeadRefinery — How the Platform Works

Deployment pipeline

Traced at commit ffe4e48 · tools/deploy/{stage-autodeploy,build-and-push,gate-dry-run,promote,release-watch}.sh · infra/{stage,prod}/*.compose.yml

Summary

  1. A job on the stage machine finds a merge to main within 5 minutes.
  2. The job polls the repository. The platform uses no webhook.
  3. The job builds 6 images and pushes them to a private registry.
  4. The job deploys stage, then runs a full rehearsal against it.
  5. A passing rehearsal records the commit as promotable.
  6. A failing rehearsal stops the commit there.
  7. Production is a separate step. A person tags a release.
  8. Production pulls images. Production never builds.
A merge to the main branch is picked up within five minutes by a job on the stage machine. The job builds six images, pushes them to a private registry, deploys them to stage, then runs a full rehearsal of about 1,125 dials against them. Only a passing rehearsal records the commit as promotable; a failing one stops it there. Production is a separate step started by a person tagging a release, and it pulls images that were already proved. Production never builds anything because it has two gigabytes of memory and a build of this stack needs more than three.
Workflow 1 of 2 — merge to stage. Automatic, on every merge. The rehearsal is the gate.

Workflow 1 — merge to stage

  1. A merge lands on main.
  2. A job on the stage machine polls every 5 minutes and notices the new commit.
  3. The job builds 6 images.
  4. The job pushes the images to a private registry.
  5. The job deploys stage from the compose project that serves it.
  6. The job runs a full rehearsal: about 1,125 dials and 10,000 events.
  7. A passing rehearsal writes a marker that records the commit as promotable.
  8. A failing rehearsal writes nothing. The commit stops there.

Workflow 2 — tag to production

A person tags a release. A watcher on the stage machine finds the newest tag every five minutes and records it before doing any work, so a crash cannot repeat the same release. If the tagged commit has not passed unit tests the watcher runs them. If no images exist for that commit the watcher builds them. If the commit has not passed a rehearsal the watcher rehearses it. Any one of these failing stops the release, with the tag still recorded so it is not retried every five minutes. Only after all three does the watcher promote, production pulls the images, and the watcher polls the health endpoint until it answers.
Workflow 2 of 2 — tag to production. Each step is conditional, so a commit already proved on stage goes straight to promotion.
  1. A person tags a release.
  2. The tag states the intent. The tag does not grant the release.
  3. If the tagged commit has not passed a rehearsal, the platform rehearses it first.
  4. The platform promotes the commit only if that rehearsal passes.
  5. Production pulls the images that were already proved.
  6. The platform then polls the production health endpoint.

Production never builds. The production machine has 2 GB of memory. A build of this stack passes 3.6 GB and takes the deployment tool down with it. Stage is the build machine. Production only pulls.

Design choices

  1. The job polls instead of receiving a webhook.
  2. A webhook needs a public endpoint, a shared secret and a way to rotate that secret.
  3. This path is not sensitive to a delay of 5 minutes.
  4. The job finds its deployment target from the labels of the running containers.
  5. The job does not use a stored name. A stored name goes stale when somebody recreates the app.
  6. A stale name deploys successfully to a stack that serves nobody.

Known gap

  1. A separate job builds the documentation site.
  2. The site is not one of the 6 images.
  3. A merge does not rebuild the documentation site.
  4. Run tools/deploy/deploy-docs.sh to publish a documentation change.

Rehearsals — what the gate actually runs, and what it cannot prove.