Jobs and queues

Every background job the worker runs, its queue, schedule and cost, how a run is locked so it never overlaps itself, how failures are retried, and the run log every run writes.

Platform / How the platform works

The worker (apps/worker) is one NestJS process that runs every background job from queues in Redis, with BullMQ. A scheduled job is put on its queue at a fixed interval; some jobs only plan work and queue batches for others. Every run is locked so it never overlaps itself, and every run writes one row to the run log. What the jobs read from Roblox, and why the collector is never stopped, is in Collector and freshness.

The three queues

collector

16 slots (WORKER_CONCURRENCY). Snapshots, discovery, recommendations, creators, icons, Rolimon's, metrics, notifications and push, studio releases, invariant checks and the dead-game cleanup.

github-organization

One slot. The GitHub organization collection, every 30 minutes.

owned-analytics

One slot. The studio games' owned analytics, once a day.

Giving GitHub and owned analytics a queue of their own means a slow or failing run of either never holds up the market collection.

Every job and its schedule

JobRunsDoes, and what it spends
snapshot-tickEvery 30 sQueues up to 250 due games as batches of 50; none while 5 batches wait
snapshot-batchQueued by the tickOne game-details and one votes request of up to 50 games: a snapshot each
discover-sortsEvery 10 minUp to 20 pages of Roblox's sorts (about five in practice)
recs-tickEvery 5 minQueues up to 400 hot, active and warm games not read for a day
recs-batchQueued by the tickOne recommendations request per game
creators-tickEvery 10 minOne request per creator, up to 150 not looked up for a week
icons-tickEvery 10 minUp to 50 icon requests of 100 games: cold games weekly, the rest daily
rolimons-tickEvery 10 minRolimon's list, and up to 50 place-to-game lookups on Roblox
metrics-tickEvery 5 minRecomputes growth, signals and the breakout score; database only
notifications-tickEvery 5 minDetects notifications from stored data; database only
notifications-pushEvery 15 sSends up to 500 new notifications to phones and browsers
studio-releasesEvery 5 minMarks approved studio games released and awards points; database only
github-organizationEvery 30 minUp to 2,200 charged GitHub requests and 4,000 unchanged answers, within 25 min
owned-analyticsOnce a day, and a minute after a startOpen Cloud with each studio group's key, never the collector's Roblox budget
invariant-checksOnce a dayCounts rows that break the database's rules; read-only
dead-game-cleanupOnce a dayCounts (or deletes) games dead for over a week

The job names are WORKER_JOBS in packages/contracts/src/admin.ts, the same names the run log and the Collector page use. Each interval is a constant beside its budget in its use case (SNAPSHOT_TICK_MS, RECOMMENDATION_TICK_MS, METRICS_TICK_MS and so on).

How a tick queues batches

The snapshot loop. Every 30 seconds snapshot-tick finds the games that are due, hot first, and queues them as batches of 50, at most 250 games and never while 5 batches still wait. Each snapshot-batch asks Roblox for the games' details and votes through the rate limiter, writes one snapshot per game, sets each game's tier and next due time, and bumps the market data version so the API's cache refreshes.

BullMQ runs jobs with no priority first, then by priority number. Scheduled jobs carry none, so they are never stuck behind batches waiting on the rate limiter; among batches, snapshots (priority 1) come before recommendations (10), because a missed snapshot is history that cannot be recovered.

A job never runs twice at once (locks)

BullMQ queues a schedule's next run when the current one starts, so a run that outlasts its interval would otherwise run beside the next. Every run of a scheduled job takes two locks:

In the process

Since one worker's 16 slots could take two runs of one job.

In Postgres

An advisory lock, pg_try_advisory_lock(hashtext('collector-job:<name>')), on one dedicated connection, for a second worker process during a deploy or a local worker pointed at the same database.

A run that finds either lock held is skipped at once. The log says <job> skipped: its previous run is still going, and no run-log row is written. Batches queued on demand (snapshot-batch, recs-batch) are not locked: each is safe to run twice, since a snapshot batch writes one reading per game and time.

Retries: what is tried again and what is not

Batchesattempts

3, with exponential backoff from 5 seconds. A failed snapshot or recommendations batch is retried by the queue.

Scheduled ticks

Not retried by the queue: the next tick is the retry.

GitHub and owned analyticsattempts
  1. A failed run waits for its next schedule (30 minutes, or the next day).

One request to Robloxattempts

Up to 4 on a timeout, network error or 5xx, inside the job, with a circuit breaker per endpoint. See what happens when Roblox is slow or down.

Finished runs are kept in Redis only briefly (the last 100 scheduled runs and 500 batches that succeeded, 500 and 2,000 that failed); the run log keeps the history.

Jobs whose process died (stalled jobs)

lockDurationseconds

120, renewed every 60, so a job keeps its queue lock through a Redis or node stall of up to a minute.

stalledIntervalseconds

60: how often a worker looks for jobs whose process died.

maxStalledCount

2: a job may be taken over twice (one restart, one node failure) before it fails.

The run log: every run writes one row

Every run, on every queue, goes through RunLog.track (apps/worker/src/shared/queue/run-log.ts). It times the run on a monotonic clock and writes one collector_runs row: the job, when it started, how long it took, how many items it handled, and whether it succeeded. A failure stores one sentence anyone can read, such as "Roblox did not answer in time." or "The run stopped on an unexpected error; details are in the worker log.", while the detail stays in the worker's log. The GitHub run also stores its HTTP requests and unchanged (304) answers.

The heartbeat the Collector page reads

Every 15 seconds the worker writes its status to Redis (collector:heartbeat, two minutes to live): snapshots in the last hour, overdue games, games by tier, each Roblox endpoint's requests, refusals, timeouts and breaker, the last Rolimon's and GitHub outcome, the last cleanup, memory and event-loop delay. The API reads it for GET /v1/collector and never calls the worker. A stopping worker removes it, so the page says Not running.

Stopping and restarting the worker

  1. It stops taking jobs

    On SIGTERM each queue's processor stops taking new jobs.

  2. It drains

    It waits up to 25 seconds for running jobs, then closes without them; their queue locks expire and the next process runs them again.

  3. It exits

    The process exits by itself 50 seconds after SIGTERM at the latest, inside Kubernetes' 60-second grace period, so a stop is never a kill.

Database connections and memory

The worker's pool has 10 connections, each with a 120-second statement timeout and a 30-second idle-in-transaction timeout; the lock session is an eleventh. Its memory limit is 768 MiB, and the largest reads are bounded (a snapshot batch is 50 games; notification detection reads GitHub snapshots six pairs at a time).

Health probes

/healthz

Liveness: the process answers, nothing else asked. A Redis stall therefore never gets a working collector killed.

/readyz

Readiness: the database and Redis each answer within 2 seconds.

/stats

Everything the heartbeat carries, read fresh. Not a probe.

For developers: adding a job

Add the job in its module's presentation/ (a job handler) and its use case in application/, give it a name in WORKER_JOBS, declare which data versions it changes (changes in its job definition) so the API's cache refreshes, and let it go through the run log. A job that asks Roblox needs its cost read against the Roblox budget first.

Related: Collector and freshness, API and caching, Architecture.

Updated