Deployment and releases

How a change reaches production - the CI checks, the release images, the deployment pull request, Argo CD's sync waves on the homelab cluster - and how rollbacks, secrets and backups work.

Platform / How the platform works

A change reaches production in two deliberate merges. First a pull request merges to main once CI passes, and the Release workflow publishes one image per app. Then a deployment pull request pins those images; merging it is the deployment, which Argo CD applies to the studio's homelab Kubernetes cluster. Nothing deploys just because code was merged.

From pull request to production

A change reaching production. A pull request runs CI and merges to main. The Release workflow publishes five images to GHCR and opens or updates the deployment pull request, which pins the new image tags in the production overlay. Merging that pull request is the deployment: Argo CD sees the change on main, runs the database migration, and rolls out the apps only if it succeeds.
  1. Open a pull request

    CI runs on it (below). The pull request's title must follow the commit format, since a squash merge uses it as the commit message.

  2. Merge to main

    The Release workflow publishes the five images (worker, api, ai, panel, www) to GHCR as ghcr.io/blockoutgames/platform-<app>, tagged sha-<7 characters> and main. A version tag such as v1.2.0 publishes the same images under that version.

  3. The deployment pull request is prepared

    The workflow updates the deploy/production branch, setting the five newTag values in the production overlay to the commit just built, and opens or updates the deployment pull request. If GitHub's policy refuses the workflow that, its summary says Manual pull request required and links the branch for a maintainer to open it.

  4. Merge the deployment pull request

    That merge is the deployment. Several merges in a row update the same open deployment pull request, so what deploys is always the newest build. To hold a release, leave it open.

  5. Argo CD applies it

    Argo CD watches main, sees the overlay change and syncs it in waves (below).

The CI checks on a pull request

What changed

Decides which jobs a pull request's changes can affect; pushes to main run everything.

Format, types, lint, unit tests

pnpm build then pnpm check: formatting, types, lint, the architecture rules and every unit test. Also an advisory audit of production dependencies.

Integration tests

pnpm test:integration, against Postgres and Redis started in Docker.

End-to-end tests

Playwright against the built panel and the built API.

Images build

Builds every affected app's Docker image without publishing it.

Kubernetes manifests render

Renders the manifests and tests the deployment-request script.

Title follows the commit format

From the separate pull-request workflow, which also labels the pull request.

CI runs on the studio's own GitHub Actions runners. A new push to a pull request cancels its older run.

Argo CD's sync waves: migration first

One Argo CD sync. Argo CD first applies the secrets and the data stores, Postgres and Redis. In wave 1 the migrate job runs the database migrations. Only if it succeeds does wave 2 roll out the API, the worker and the website. Wave 3 rolls out the panel after the API, so its pages meet the matching API, and the assistant app, so an assistant waiting for its secrets never holds the rest back. A failed migration stops the sync with the old images still running.

The migration job retries a failed container in the same pod, up to six times within 180 seconds, because a new pod's network permission to reach the database can take a few seconds to arrive. A migration that really fails stops the rollout. The panel, the API and the website replace one copy at a time, and a disruption budget keeps one of each running while a node is drained.

Rolling back a release

  1. Pick the earlier build

    Find the sha- tag of the last good release; images are never deleted from GHCR by the workflow.

  2. Pin it

    Open a pull request against main that sets the overlay's newTag values to that tag.

  3. Merge it

    Argo CD rolls the apps back exactly as it rolls them forward.

Where production runs: the homelab cluster

Production runs on the studio's homelab Kubernetes cluster, in the blockout namespace. This repository holds every object (infra/k8s/base) and the production overlay that pins exact image tags with the sealed secrets (infra/k8s/overlays/production). The homelab repository holds the Argo CD application that points at that overlay on main, and the two public routes.

Cloudflare

DNS for blockoutgames.com, TLS, and the tunnel that carries www.blockoutgames.com, the bare domain and studio.blockoutgames.com to the cluster's edge gateway.

Public routes

The website, and the panel at studio.blockoutgames.com. Nothing else has a route from outside.

Inside only

The API, the worker, the assistant app, Postgres and Redis, behind network policies that refuse every connection not listed.

Copies

Two each of the panel, the API and the website; one each of the worker, the assistant app, Postgres and Redis.

  • infra/
    • k8s/
      • base/every object: Postgres, Redis, migrate, worker, API, AI app, panel, www, backups, network policies
      • overlays/
        • production/the base pinned to image tags, plus sealed-secrets.yaml
      • scripts/
        • seal-secrets.shseals the cluster's secrets
    • scripts/
      • deploy-request.shprepares the deployment pull request
How every pod is hardened

Every pod meets Kubernetes' restricted Pod Security profile: it runs as a non-root user with the default seccomp profile, drops every capability, cannot escalate privileges, and mounts no service-account token. Every container but Postgres has a read-only root filesystem. Each workload receives only the secret keys it reads; the worker, for example, never holds the sign-in, assistant or contract-signing secrets.

Secrets: sealed in the repository

The cluster's secrets (the database password, the credential root key, the sign-in secret, the assistant's service secret, the push keys, the image pull token) are sealed with infra/k8s/scripts/seal-secrets.sh into sealed-secrets.yaml, which is safe to commit: only the cluster's sealed-secrets controller can open it. Provider keys (each Roblox group's key, the GitHub App's key, the GitHub sign-in app's secret) are not cluster secrets: they are sealed in Postgres under the credential root key, which can be rotated without signing anyone out.

Backups

Nightly dump

The postgres-backup CronJob writes a custom-format dump every day at 08:15 by the cluster's clock, to a 50 GiB volume, keeping 14 days.

Off the volume

When the backup bucket is configured, every dump is also copied to it and kept 60 days.

Volume snapshots

The Postgres and backup volumes are in the cluster's daily snapshot and weekly backup.

Local

pnpm db:backup writes a dump of the local Docker database (newest 14 kept); pnpm db:restore -- --verify proves the newest restores.

A restore never overwrites the live database: it restores beside it, is checked, and then the two are switched. History that was never collected cannot be restored by any backup; see Gaps after an outage.

Related: Architecture, Data model, Security and privacy.

Updated