The worker (apps/worker) is one NestJS process that runs every background job from queues in Redis, with BullMQ. A scheduled job is put on its queue at a fixed interval; some jobs only plan work and queue batches for others. Every run is locked so it never overlaps itself, and every run writes one row to the run log. What the jobs read from Roblox, and why the collector is never stopped, is in Collector and freshness.
The three queues
- collector
16 slots (
WORKER_CONCURRENCY). Snapshots, discovery, recommendations, creators, icons, Rolimon's, metrics, notifications and push, studio releases, invariant checks and the dead-game cleanup.- github-organization
One slot. The GitHub organization collection, every 30 minutes.
- owned-analytics
One slot. The studio games' owned analytics, once a day.
Giving GitHub and owned analytics a queue of their own means a slow or failing run of either never holds up the market collection.
Every job and its schedule
| Job | Runs | Does, and what it spends |
|---|---|---|
snapshot-tick | Every 30 s | Queues up to 250 due games as batches of 50; none while 5 batches wait |
snapshot-batch | Queued by the tick | One game-details and one votes request of up to 50 games: a snapshot each |
discover-sorts | Every 10 min | Up to 20 pages of Roblox's sorts (about five in practice) |
recs-tick | Every 5 min | Queues up to 400 hot, active and warm games not read for a day |
recs-batch | Queued by the tick | One recommendations request per game |
creators-tick | Every 10 min | One request per creator, up to 150 not looked up for a week |
icons-tick | Every 10 min | Up to 50 icon requests of 100 games: cold games weekly, the rest daily |
rolimons-tick | Every 10 min | Rolimon's list, and up to 50 place-to-game lookups on Roblox |
metrics-tick | Every 5 min | Recomputes growth, signals and the breakout score; database only |
notifications-tick | Every 5 min | Detects notifications from stored data; database only |
notifications-push | Every 15 s | Sends up to 500 new notifications to phones and browsers |
studio-releases | Every 5 min | Marks approved studio games released and awards points; database only |
github-organization | Every 30 min | Up to 2,200 charged GitHub requests and 4,000 unchanged answers, within 25 min |
owned-analytics | Once a day, and a minute after a start | Open Cloud with each studio group's key, never the collector's Roblox budget |
invariant-checks | Once a day | Counts rows that break the database's rules; read-only |
dead-game-cleanup | Once a day | Counts (or deletes) games dead for over a week |
The job names are WORKER_JOBS in packages/contracts/src/admin.ts, the same names the run log and the Collector page use. Each interval is a constant beside its budget in its use case (SNAPSHOT_TICK_MS, RECOMMENDATION_TICK_MS, METRICS_TICK_MS and so on).
How a tick queues batches
BullMQ runs jobs with no priority first, then by priority number. Scheduled jobs carry none, so they are never stuck behind batches waiting on the rate limiter; among batches, snapshots (priority 1) come before recommendations (10), because a missed snapshot is history that cannot be recovered.
A job never runs twice at once (locks)
BullMQ queues a schedule's next run when the current one starts, so a run that outlasts its interval would otherwise run beside the next. Every run of a scheduled job takes two locks:
- In the process
Since one worker's 16 slots could take two runs of one job.
- In Postgres
An advisory lock,
pg_try_advisory_lock(hashtext('collector-job:<name>')), on one dedicated connection, for a second worker process during a deploy or a local worker pointed at the same database.
A run that finds either lock held is skipped at once. The log says <job> skipped: its previous run is still going, and no run-log row is written. Batches queued on demand (snapshot-batch, recs-batch) are not locked: each is safe to run twice, since a snapshot batch writes one reading per game and time.
Retries: what is tried again and what is not
- Batchesattempts
3, with exponential backoff from 5 seconds. A failed snapshot or recommendations batch is retried by the queue.
- Scheduled ticks
Not retried by the queue: the next tick is the retry.
- GitHub and owned analyticsattempts
A failed run waits for its next schedule (30 minutes, or the next day).
- One request to Robloxattempts
Up to 4 on a timeout, network error or 5xx, inside the job, with a circuit breaker per endpoint. See what happens when Roblox is slow or down.
Finished runs are kept in Redis only briefly (the last 100 scheduled runs and 500 batches that succeeded, 500 and 2,000 that failed); the run log keeps the history.
Jobs whose process died (stalled jobs)
- lockDurationseconds
120, renewed every 60, so a job keeps its queue lock through a Redis or node stall of up to a minute.
- stalledIntervalseconds
60: how often a worker looks for jobs whose process died.
- maxStalledCount
2: a job may be taken over twice (one restart, one node failure) before it fails.
The run log: every run writes one row
Every run, on every queue, goes through RunLog.track (apps/worker/src/shared/queue/run-log.ts). It times the run on a monotonic clock and writes one collector_runs row: the job, when it started, how long it took, how many items it handled, and whether it succeeded. A failure stores one sentence anyone can read, such as "Roblox did not answer in time." or "The run stopped on an unexpected error; details are in the worker log.", while the detail stays in the worker's log. The GitHub run also stores its HTTP requests and unchanged (304) answers.
The heartbeat the Collector page reads
Every 15 seconds the worker writes its status to Redis (collector:heartbeat, two minutes to live): snapshots in the last hour, overdue games, games by tier, each Roblox endpoint's requests, refusals, timeouts and breaker, the last Rolimon's and GitHub outcome, the last cleanup, memory and event-loop delay. The API reads it for GET /v1/collector and never calls the worker. A stopping worker removes it, so the page says Not running.
Stopping and restarting the worker
It stops taking jobs
On SIGTERM each queue's processor stops taking new jobs.
It drains
It waits up to 25 seconds for running jobs, then closes without them; their queue locks expire and the next process runs them again.
It exits
The process exits by itself 50 seconds after SIGTERM at the latest, inside Kubernetes' 60-second grace period, so a stop is never a kill.
Database connections and memory
The worker's pool has 10 connections, each with a 120-second statement timeout and a 30-second idle-in-transaction timeout; the lock session is an eleventh. Its memory limit is 768 MiB, and the largest reads are bounded (a snapshot batch is 50 games; notification detection reads GitHub snapshots six pairs at a time).
Health probes
- /healthz
Liveness: the process answers, nothing else asked. A Redis stall therefore never gets a working collector killed.
- /readyz
Readiness: the database and Redis each answer within 2 seconds.
- /stats
Everything the heartbeat carries, read fresh. Not a probe.
For developers: adding a job
Add the job in its module's presentation/ (a job handler) and its use case in application/, give it a name in WORKER_JOBS, declare which data versions it changes (changes in its job definition) so the API's cache refreshes, and let it go through the run log. A job that asks Roblox needs its cost read against the Roblox budget first.
Related: Collector and freshness, API and caching, Architecture.
Updated