A change reaches production in two deliberate merges. First a pull request merges to main once CI passes, and the Release workflow publishes one image per app. Then a deployment pull request pins those images; merging it is the deployment, which Argo CD applies to the studio's homelab Kubernetes cluster. Nothing deploys just because code was merged.
From pull request to production
Open a pull request
CI runs on it (below). The pull request's title must follow the commit format, since a squash merge uses it as the commit message.
Merge to main
The Release workflow publishes the five images (
worker,api,ai,panel,www) to GHCR asghcr.io/blockoutgames/platform-<app>, taggedsha-<7 characters>andmain. A version tag such asv1.2.0publishes the same images under that version.The deployment pull request is prepared
The workflow updates the
deploy/productionbranch, setting the fivenewTagvalues in the production overlay to the commit just built, and opens or updates the deployment pull request. If GitHub's policy refuses the workflow that, its summary says Manual pull request required and links the branch for a maintainer to open it.Merge the deployment pull request
That merge is the deployment. Several merges in a row update the same open deployment pull request, so what deploys is always the newest build. To hold a release, leave it open.
Argo CD applies it
Argo CD watches
main, sees the overlay change and syncs it in waves (below).
The CI checks on a pull request
- What changed
Decides which jobs a pull request's changes can affect; pushes to
mainrun everything.- Format, types, lint, unit tests
pnpm buildthenpnpm check: formatting, types, lint, the architecture rules and every unit test. Also an advisory audit of production dependencies.- Integration tests
pnpm test:integration, against Postgres and Redis started in Docker.- End-to-end tests
Playwright against the built panel and the built API.
- Images build
Builds every affected app's Docker image without publishing it.
- Kubernetes manifests render
Renders the manifests and tests the deployment-request script.
- Title follows the commit format
From the separate pull-request workflow, which also labels the pull request.
CI runs on the studio's own GitHub Actions runners. A new push to a pull request cancels its older run.
Argo CD's sync waves: migration first
The migration job retries a failed container in the same pod, up to six times within 180 seconds, because a new pod's network permission to reach the database can take a few seconds to arrive. A migration that really fails stops the rollout. The panel, the API and the website replace one copy at a time, and a disruption budget keeps one of each running while a node is drained.
Rolling back a release
Pick the earlier build
Find the
sha-tag of the last good release; images are never deleted from GHCR by the workflow.Pin it
Open a pull request against
mainthat sets the overlay'snewTagvalues to that tag.Merge it
Argo CD rolls the apps back exactly as it rolls them forward.
Where production runs: the homelab cluster
Production runs on the studio's homelab Kubernetes cluster, in the blockout namespace. This repository holds every object (infra/k8s/base) and the production overlay that pins exact image tags with the sealed secrets (infra/k8s/overlays/production). The homelab repository holds the Argo CD application that points at that overlay on main, and the two public routes.
- Cloudflare
DNS for
blockoutgames.com, TLS, and the tunnel that carrieswww.blockoutgames.com, the bare domain andstudio.blockoutgames.comto the cluster's edge gateway.- Public routes
The website, and the panel at
studio.blockoutgames.com. Nothing else has a route from outside.- Inside only
The API, the worker, the assistant app, Postgres and Redis, behind network policies that refuse every connection not listed.
- Copies
Two each of the panel, the API and the website; one each of the worker, the assistant app, Postgres and Redis.
- infra/
- k8s/
- base/every object: Postgres, Redis, migrate, worker, API, AI app, panel, www, backups, network policies
- overlays/
- production/the base pinned to image tags, plus sealed-secrets.yaml
- scripts/
- seal-secrets.shseals the cluster's secrets
- scripts/
- deploy-request.shprepares the deployment pull request
- k8s/
How every pod is hardened
Every pod meets Kubernetes' restricted Pod Security profile: it runs as a non-root user with the
default seccomp profile, drops every capability, cannot escalate privileges, and mounts no
service-account token. Every container but Postgres has a read-only root filesystem. Each workload
receives only the secret keys it reads; the worker, for example, never holds the sign-in,
assistant or contract-signing secrets.
Secrets: sealed in the repository
The cluster's secrets (the database password, the credential root key, the sign-in secret, the assistant's service secret, the push keys, the image pull token) are sealed with infra/k8s/scripts/seal-secrets.sh into sealed-secrets.yaml, which is safe to commit: only the cluster's sealed-secrets controller can open it. Provider keys (each Roblox group's key, the GitHub App's key, the GitHub sign-in app's secret) are not cluster secrets: they are sealed in Postgres under the credential root key, which can be rotated without signing anyone out.
Backups
- Nightly dump
The
postgres-backupCronJob writes a custom-format dump every day at 08:15 by the cluster's clock, to a 50 GiB volume, keeping 14 days.- Off the volume
When the backup bucket is configured, every dump is also copied to it and kept 60 days.
- Volume snapshots
The Postgres and backup volumes are in the cluster's daily snapshot and weekly backup.
- Local
pnpm db:backupwrites a dump of the local Docker database (newest 14 kept);pnpm db:restore -- --verifyproves the newest restores.
A restore never overwrites the live database: it restores beside it, is checked, and then the two are switched. History that was never collected cannot be restored by any backup; see Gaps after an outage.
Related: Architecture, Data model, Security and privacy.
Updated