The system had eleven cron jobs. Each one woke up, scanned a table for work, and processed whatever it found. It worked for two years and then stopped working all at once.

Why it broke

Cron gives you a schedule. What we actually needed was ordering per entity and back-pressure, and a schedule provides neither.

  • Two jobs could pick up the same record within the same minute.
  • A slow downstream API turned a five-minute job into a fifteen-minute one, so the next run started before the previous finished.
  • Every failure meant re-scanning the entire table on the next tick.

The shape of the fix

Instead of scanning for work, we push work onto a queue keyed by the entity it affects. Jobs for the same key run strictly in order; different keys run in parallel.

await queue.add(
  'sync-account',
  { accountId },
  { jobId: `sync-account:${accountId}`, removeOnComplete: 100 },
);

Two properties fall out of this for free:

  1. Ordering per account — no two syncs for the same account overlap, ever.
  2. Natural deduplication — a stable jobId collapses a burst of triggers into one pending job.

What improved

Before After
11 cron jobs on overlapping schedules 1 worker, keyed by entity
Spike every hour on the hour Load spread across the hour
Failures re-scanned everything Failures retry one job

The traffic graph was the convincing part. The hourly sawtooth flattened into something close to a line.

The catch

Queues move the complexity rather than removing it. You now own retry policy, dead-letter handling, and a dashboard nobody looks at until an incident. Budget for that work — it is real, and it is worth it, but it is not free.