Collector and freshness

What the collector reads, its jobs and schedules, what it does when Roblox rate limits or is down, and where to check that it is running.

Platform / How the platform works

The collector is the studio's background worker (apps/worker). It reads Roblox, Rolimon's and GitHub on schedules and stores what it reads in Postgres. Every page reads that stored data; no page asks Roblox or GitHub directly. Roblox does not serve history, so a reading not taken at the time is lost: the collector is never stopped on purpose.

The collector's jobs and schedules

The collector's schedule. The worker runs its jobs on fixed intervals. Every 15 to 30 seconds it queues due snapshots and sends push notifications. Every 5 minutes it recomputes metrics, queues recommendations, detects notifications and checks studio releases. Every 10 minutes it reads Roblox's sorts, creators' games, icons and Rolimon's list. Every 30 minutes it collects GitHub. Once a day it reads owned analytics, checks the database's rules and counts dead games.
JobRunsDoes
snapshot-tickEvery 30 secondsQueues up to 250 games that are due, in batches of 50; none while 5 batches still wait
snapshot-batchWhen queuedReads up to 50 games' details and votes from Roblox: one snapshot each
metrics-tickEvery 5 minutesRecomputes growth, the breakout score and the signals from stored snapshots
discover-sortsEvery 10 minutesReads Roblox's discovery sorts and ranks
recs-tickEvery 5 minutesQueues up to 400 hot, active and warm games whose recommendations are a day old
creators-tickEvery 10 minutesLooks up the other games of up to 150 creators not checked for a week
icons-tickEvery 10 minutesRefreshes icons: cold games' weekly, the rest daily
rolimons-tickEvery 10 minutesReads Rolimon's list and matches up to 50 new places to games
notifications-tickEvery 5 minutesDetects notifications from stored data
notifications-pushEvery 15 secondsSends new notifications to phones and browsers
studio-releasesEvery 5 minutesMarks approved studio games released when first read public, and awards their points
github-organizationEvery 30 minutesCollects the GitHub organization for the Code pages
owned-analyticsOnce a day, and after a startReads the studio games' owned analytics with their group keys
invariant-checksOnce a dayChecks the database's rules, read-only
dead-game-cleanupOnce a dayCounts (or deletes) games dead for over a week

Only the snapshot, discovery, recommendation, creator, icon and Rolimon's jobs ask Roblox's public endpoints; their cost is in The Roblox budget. Metrics, notifications, releases, invariants and cleanup read only the database. Each job's queue, lock, retries and run log are in Jobs and queues.

How often a game is read

A game's players, visits, favorites and votes are read every 5 minutes (hot), 15 minutes (active), hour (warm), day (cold) or week (dead), by its tier. The thresholds are in Tiers and genres; the full list of how often every source is read is in Freshness and empty numbers.

How fresh the numbers are

New collected data reaches the panel within about 30 seconds of being stored. Owned analytics are Roblox's daily numbers, re-read for the last 7 complete UTC days each day, so today's are never there yet. The Code pages say "Collected ... ago" in their header.

What happens when Roblox rate limits the collector

The collector paces itself to stay under Roblox's limits: each endpoint has its own budget, and game details are sent at 500 games a minute, under the roughly 550 Roblox allows. If Roblox still answers 429 (too many requests), that endpoint pauses until the time Roblox gives for the limit to reset, then carries on. Nothing is sent faster to catch up: due games wait, hot ones first. The Collector page counts refusals per endpoint, and the status says Collecting with errors.

What happens when Roblox is slow or down

When Roblox fails. A request that times out or gets a server error is tried again up to 4 attempts, waiting 1 to 30 seconds at random. A 429 pauses that endpoint until the time Roblox gives. Five failures in a row open the endpoint's circuit breaker: it pauses 30 seconds, doubling up to 5 minutes, then one trial request goes first. Games not read meanwhile become overdue and are read when Roblox answers again, hot games first.
  • A timeout or a server error (5xx, no answer): the one request is tried again, up to 4 attempts, waiting 1 to 30 seconds at random between them (never less than Roblox asks).

  • Five failures in a row on one endpoint open its circuit breaker: that endpoint pauses for 30 seconds, doubling up to 5 minutes while it keeps failing; then one trial request goes first. Jobs wait out the pause instead of failing again and again.

  • Timeouts are 20 seconds for game details and sorts, 10 for recommendations, 30 for analytics, 15 for the rest.

  • Games not read meanwhile become overdue and are read when Roblox answers again, hot ones first.

A breaker only ever holds requests back; nothing is ever sent because of one.

What happens when Rolimon's or GitHub fails

  • Rolimon's gets no retry within a run: the next 10-minute run is the retry. Three failed runs in a row skip the next ones for 20 minutes, doubling up to 2 hours.

  • GitHub: a failing request is tried up to 3 times. One repository that fails keeps its previous data and the run is stored as partial. Three repositories failing in a row stop the run; two stopped runs in a row skip the next for 30 minutes, doubling up to 2 hours.

GitHub collection

The GitHub collection runs every 30 minutes, on its own queue with one slot, where the worker collects the BlockoutGames organization: repositories, pull requests with their reviews, releases, members and contributor statistics. Runs are incremental: only pull requests updated since the last run are read again, and every request asks "has this changed?", which GitHub does not charge when the answer is no. A run may send up to 2,200 charged requests (of GitHub's 5,000 an hour) and receive 4,000 unchanged answers, within 25 minutes. A run that runs out keeps what it read and the next one continues. Member profiles are read once a day, contributor statistics once an hour. See Code metrics.

Owned analytics read

Once a day, on its own queue with one slot, the worker reads each studio game's owned analytics, store items and bans with the key of the group that owns it, and once more a minute after the worker starts. Its requests are counted against each key on Roblox's side, never the collector's public budget. A slow or failing read never holds up the market collection. See Studio game stats.

A job never runs twice at once

Each scheduled job takes a lock in the process and in Postgres, so a run that outlasts its interval is not joined by the next one: the next run is skipped and the log says "its previous run is still going". Every job is safe to run twice: a snapshot batch writes one reading per game and time. See locks.

Restarts and stops

On a stop, the worker stops taking jobs and waits up to 25 seconds for the running ones; anything unfinished runs again in the next process. It exits on its own within 50 seconds. A job whose process died is taken over by another after its lock runs out, at most twice.

Where to check it is running

The Collector page, listed in the Admin rail, is open to every member (see Collector page). It shows the status (Collecting, Collecting with errors, Behind, Stalled or Not running), the last snapshot, snapshots in the last hour, overdue games, how long it has been running, tracked games by tier, job runs in the last hour, and requests to Roblox per endpoint with refusals, errors, timeouts and each breaker's state. Owners also see the run history (runs and failures, run time per job, items per run, recent failures with their errors).

The status words are defined in Freshness and empty numbers. Owners also have Data health (see Audit log and Data health), which flags a stale source: Roblox snapshots older than 15 minutes, GitHub older than an hour, owned analytics older than 36 hours.

When owners are told

Owners get a collector failing notification when a job's last three runs in the day all failed, or no snapshot has been stored for 15 minutes, once until it recovers, and collector recovered when it does. See Notifications and push.

The run log

Every run of every job writes one row (collector_runs, kept 30 days): when, how long, how many items and, on a failure, one sentence anyone can read ("Roblox did not answer in time."). The detail stays in the worker's log. A skipped run writes no row.

Dead games and the cleanup

A game Roblox left out of 3 reads in a row is dead and read weekly. Once a day the dead-game-cleanup job finds games dead for at least a week (a fourth miss, and their last reading over 7 days old) that are not studio games and not watched by anyone. Today it only counts them (dry run); deleting them, with their whole history, is a founder's decision.

Related: Jobs and queues, The Roblox budget, Architecture, Admin pages.

Updated