The collector is the studio's background worker (apps/worker). It reads Roblox, Rolimon's and GitHub on schedules and stores what it reads in Postgres. Every page reads that stored data; no page asks Roblox or GitHub directly. Roblox does not serve history, so a reading not taken at the time is lost: the collector is never stopped on purpose.
The collector's jobs and schedules
| Job | Runs | Does |
|---|---|---|
snapshot-tick | Every 30 seconds | Queues up to 250 games that are due, in batches of 50; none while 5 batches still wait |
snapshot-batch | When queued | Reads up to 50 games' details and votes from Roblox: one snapshot each |
metrics-tick | Every 5 minutes | Recomputes growth, the breakout score and the signals from stored snapshots |
discover-sorts | Every 10 minutes | Reads Roblox's discovery sorts and ranks |
recs-tick | Every 5 minutes | Queues up to 400 hot, active and warm games whose recommendations are a day old |
creators-tick | Every 10 minutes | Looks up the other games of up to 150 creators not checked for a week |
icons-tick | Every 10 minutes | Refreshes icons: cold games' weekly, the rest daily |
rolimons-tick | Every 10 minutes | Reads Rolimon's list and matches up to 50 new places to games |
notifications-tick | Every 5 minutes | Detects notifications from stored data |
notifications-push | Every 15 seconds | Sends new notifications to phones and browsers |
studio-releases | Every 5 minutes | Marks approved studio games released when first read public, and awards their points |
github-organization | Every 30 minutes | Collects the GitHub organization for the Code pages |
owned-analytics | Once a day, and after a start | Reads the studio games' owned analytics with their group keys |
invariant-checks | Once a day | Checks the database's rules, read-only |
dead-game-cleanup | Once a day | Counts (or deletes) games dead for over a week |
Only the snapshot, discovery, recommendation, creator, icon and Rolimon's jobs ask Roblox's public endpoints; their cost is in The Roblox budget. Metrics, notifications, releases, invariants and cleanup read only the database. Each job's queue, lock, retries and run log are in Jobs and queues.
How often a game is read
A game's players, visits, favorites and votes are read every 5 minutes (hot), 15 minutes (active), hour (warm), day (cold) or week (dead), by its tier. The thresholds are in Tiers and genres; the full list of how often every source is read is in Freshness and empty numbers.
How fresh the numbers are
New collected data reaches the panel within about 30 seconds of being stored. Owned analytics are Roblox's daily numbers, re-read for the last 7 complete UTC days each day, so today's are never there yet. The Code pages say "Collected ... ago" in their header.
What happens when Roblox rate limits the collector
The collector paces itself to stay under Roblox's limits: each endpoint has its own budget, and game details are sent at 500 games a minute, under the roughly 550 Roblox allows. If Roblox still answers 429 (too many requests), that endpoint pauses until the time Roblox gives for the limit to reset, then carries on. Nothing is sent faster to catch up: due games wait, hot ones first. The Collector page counts refusals per endpoint, and the status says Collecting with errors.
What happens when Roblox is slow or down
A timeout or a server error (5xx, no answer): the one request is tried again, up to 4 attempts, waiting 1 to 30 seconds at random between them (never less than Roblox asks).
Five failures in a row on one endpoint open its circuit breaker: that endpoint pauses for 30 seconds, doubling up to 5 minutes while it keeps failing; then one trial request goes first. Jobs wait out the pause instead of failing again and again.
Timeouts are 20 seconds for game details and sorts, 10 for recommendations, 30 for analytics, 15 for the rest.
Games not read meanwhile become overdue and are read when Roblox answers again, hot ones first.
A breaker only ever holds requests back; nothing is ever sent because of one.
What happens when Rolimon's or GitHub fails
Rolimon's gets no retry within a run: the next 10-minute run is the retry. Three failed runs in a row skip the next ones for 20 minutes, doubling up to 2 hours.
GitHub: a failing request is tried up to 3 times. One repository that fails keeps its previous data and the run is stored as partial. Three repositories failing in a row stop the run; two stopped runs in a row skip the next for 30 minutes, doubling up to 2 hours.
GitHub collection
The GitHub collection runs every 30 minutes, on its own queue with one slot, where the worker collects the BlockoutGames organization: repositories, pull requests with their reviews, releases, members and contributor statistics. Runs are incremental: only pull requests updated since the last run are read again, and every request asks "has this changed?", which GitHub does not charge when the answer is no. A run may send up to 2,200 charged requests (of GitHub's 5,000 an hour) and receive 4,000 unchanged answers, within 25 minutes. A run that runs out keeps what it read and the next one continues. Member profiles are read once a day, contributor statistics once an hour. See Code metrics.
Owned analytics read
Once a day, on its own queue with one slot, the worker reads each studio game's owned analytics, store items and bans with the key of the group that owns it, and once more a minute after the worker starts. Its requests are counted against each key on Roblox's side, never the collector's public budget. A slow or failing read never holds up the market collection. See Studio game stats.
A job never runs twice at once
Each scheduled job takes a lock in the process and in Postgres, so a run that outlasts its interval is not joined by the next one: the next run is skipped and the log says "its previous run is still going". Every job is safe to run twice: a snapshot batch writes one reading per game and time. See locks.
Restarts and stops
On a stop, the worker stops taking jobs and waits up to 25 seconds for the running ones; anything unfinished runs again in the next process. It exits on its own within 50 seconds. A job whose process died is taken over by another after its lock runs out, at most twice.
Where to check it is running
The Collector page, listed in the Admin rail, is open to every member (see Collector page). It shows the status (Collecting, Collecting with errors, Behind, Stalled or Not running), the last snapshot, snapshots in the last hour, overdue games, how long it has been running, tracked games by tier, job runs in the last hour, and requests to Roblox per endpoint with refusals, errors, timeouts and each breaker's state. Owners also see the run history (runs and failures, run time per job, items per run, recent failures with their errors).
The status words are defined in Freshness and empty numbers. Owners also have Data health (see Audit log and Data health), which flags a stale source: Roblox snapshots older than 15 minutes, GitHub older than an hour, owned analytics older than 36 hours.
When owners are told
Owners get a collector failing notification when a job's last three runs in the day all failed, or no snapshot has been stored for 15 minutes, once until it recovers, and collector recovered when it does. See Notifications and push.
The run log
Every run of every job writes one row (collector_runs, kept 30 days): when, how long, how many items and, on a failure, one sentence anyone can read ("Roblox did not answer in time."). The detail stays in the worker's log. A skipped run writes no row.
Dead games and the cleanup
A game Roblox left out of 3 reads in a row is dead and read weekly. Once a day the dead-game-cleanup job finds games dead for at least a week (a fourth miss, and their last reading over 7 days old) that are not studio games and not watched by anyone. Today it only counts them (dry run); deleting them, with their whole history, is a founder's decision.
Related: Jobs and queues, The Roblox budget, Architecture, Admin pages.
Updated