What MongoDB change streams actually cost you

We had a worker that polled a collection every two seconds looking for ledger entries in a pending state. It worked for about eighteen months. Then traffic grew, two workers occasionally picked up the same document, and a customer got paid twice.

Change streams fixed the duplication. They also handed me a set of operational problems that nobody mentions in the tutorials, so here they are.

What a change stream actually is

A change stream is a tailable cursor over the oplog, wrapped in a friendlier API. You open it, you get events, and each event carries a resume token that points at a position in the oplog.

const stream = db.collection('entries').watch(
  [{ $match: { 'fullDocument.state': 'pending' } }],
  { fullDocument: 'updateLookup', resumeAfter: lastToken }
);

for await (const change of stream) {
  await handle(change.fullDocument);
  await saveToken(change._id);
}

That is the whole shape of it. The interesting part is everything the loop above gets wrong.

Problem one: the resume token has a shelf life

The oplog is a capped collection. If your process is down long enough for the oplog to wrap past your stored resume token, the stream cannot resume and the driver throws ChangeStreamHistoryLost.

On a busy cluster with default sizing, that window can be shorter than an hour. Ours was about forty minutes during month-end, which is exactly when a deploy is most likely to take a while.

Two things to do about it:

• Size the oplog for your worst realistic outage, not your average one. We went to 24 hours of headroom.
• Treat history loss as a recoverable state, not a crash. On ChangeStreamHistoryLost, fall back to a reconciliation query that scans for anything still pending and older than the last successful checkpoint, then reopen the stream from now.

The fallback path is the important half. A change stream is an optimisation over polling, and an optimisation you cannot fall back from is a single point of failure.

Problem two: the token is saved after the work

Look at the loop again. It handles the change, then saves the token. If the process dies between those two lines, the event is replayed on restart.

That is the correct ordering — at-least-once is far better than the alternative — but it means handlers must be idempotent, and “idempotent” has to mean something specific. For us it meant every write carries a deterministic key derived from the source document:

await payouts.updateOne(
  { idempotencyKey: `entry:${entry._id}:v${entry.version}` },
  { $setOnInsert: buildPayout(entry) },
  { upsert: true }
);

A unique index on idempotencyKey turns a replay into a no-op instead of a second payment. This one index is doing more work for our reliability than the change stream itself.

Problem three: one stream, one consumer

A change stream is not a queue. Every process that opens one gets every event, and there is no built-in way to distribute events across a pool of workers.

So you have a choice:

Approach	Good for	Cost
Single watcher process	Simplicity, ordering	It is a single point of failure
Watcher plus real queue	Throughput, retries	One more moving part
Watcher per shard key range	Parallelism without a broker	Manual rebalancing, easy to get wrong

We took the middle option. One small process watches the stream and does nothing but push jobs into BullMQ. It has no business logic, so it almost never needs deploying, which keeps the resume-token window comfortable.

Was it worth it

Yes, but not for the reason I expected. The latency win was real — median time from entry creation to payout went from about 1.1 seconds to under 200 milliseconds — and that is nice.

The actual win was that the duplicate-payout bug became structurally impossible. Not “unlikely”, not “guarded against in the happy path”. The unique index means a second attempt cannot produce a second payment, whatever the delivery layer does.

If you take one thing from this: the idempotency key is the fix. The change stream is just a faster way to notice there is work to do.