Event streams over cron: rethinking a lead-scoring pipeline

The lead-scoring job ran at 2am. Sales saw scores at 9am. A lead that filled in a form at 9:05am waited twenty-eight hours for a number that decided whether anyone called them. Nobody had designed that delay — it was the residue of a cron line written years earlier, when a nightly window was the only thing anyone had.

What the batch job was actually doing

Strip away the scheduling and the job did three things per lead: pull recent activity, run a scoring model, write the result back. None of those needed a batch. They needed a trigger.

The rewrite was smaller than the discussion about it. Producers publish a lead.activity event, a Lambda consumes the topic, scores the single lead in the event, and writes back. The scoring code itself was untouched.

What actually got hard

Ordering. Two activity events for the same lead, processed out of order, write the older score last. Partitioning by lead id fixed it, and partitioning by lead id is also what made a hot lead a hot partition. That tradeoff never fully goes away; it just moves.

Replay. A batch job re-run is a re-run. A stream needs an explicit replay story — ours is “reset the consumer group to a timestamp,” which works because scoring is idempotent. Making it idempotent was most of the migration work.

The long tail of the batch. The nightly job also quietly did three other things nobody had documented: backfilling missing fields, expiring stale leads, and emitting a daily summary. Those are genuinely periodic. They stayed on a schedule, and that is correct — the goal was never to delete the cron, only to stop it being the only clock in the system.

Was it worth it

Median time from activity to score went from roughly nineteen hours to under four seconds. The infrastructure bill went up by a rounding error. The operational surface got larger: a topic, a consumer group, a dead-letter queue, and a lag dashboard that someone now has to look at.

That last part is the real cost, and it is the one that never appears in the design doc. Streaming does not remove work; it converts a scheduling problem into a plumbing problem. It was the right trade here because the delay was the product problem. If the scores had only ever been read at 9am, the correct fix would have been to move the cron to 7am and spend the quarter on something else.