Events, notifications and recovery

The problem#

A marketplace is made of operations touching several systems at once: payment, order, notification, payout, integration. Each can fail on its own, and the worst case is not total failure but partial success.

Money taken with no order, an order with no notification, a refund executed at the provider and unrecorded locally. These are rare and certain.

The principle#

Minimum version#

• Logs carrying an operation identifier that ties entries into one history.
• Visibility of failed background jobs, even as a list to review.
• A ledger of events arriving from providers, keyed by their identifiers.
• The ability to retry an operation by hand, with no side effects on the retry.

Notifications#

Sending an email is a separate operation from the business event that triggered it. Fusing them creates two problems at once: a mail outage blocks selling, and a successful send looks like proof the rest succeeded.

• The business event is recorded whether or not the notification went out.
• A failed send is retried, and once attempts run out it stays visible rather than disappearing.
• A missing notification never blocks the operation it concerns.

What not to build yet#

• A notification centre with user preferences.
• Full distributed tracing with cross-service correlation.
• Automatic repair of discrepancies.
• Alerts on everything. An alert that fires daily stops being an alert.

Plan ahead#

An operation identifier that travels#

One identifier assigned at the entry point and passed along lets you reconstruct what happened. Adding it afterwards cannot be applied retroactively to events that already occurred.

Failed jobs stay, they do not vanish#

The default behaviour of many queues is to drop a job after the last failed attempt. Then there is nothing to look at. A job that disappeared is worse than a job that visibly failed.

A discrepancy as something you measure#

Comparing your record with provider data does not have to be automatic. It has to be possible. Even a weekly comparison done by hand is incomparably better than discovering a discrepancy three months later.

Triggers#

• the first payment with no order;
• the first question about whether a notification went out at all;
• the first discrepancy in amounts;
• the first case where retrying an operation did something twice;
• a provider outage after which nobody knows what went through.

Next stage#

After the first and third triggers: a discrepancy listing, done by hand. After the second: notification send history. After the fourth: idempotency where it is missing, money operations first. After the fifth: only then alerts, and only for things somebody will actually act on.

Common mistakes#

What works in practice#

What bought the most peace of mind was keeping failed background jobs visible instead of letting them vanish. Queue defaults often drop them, so a mail provider outage or a brief database problem ends in a silently lost notification that you hear about from the customer.

The second thing: a ledger of incoming events. It answers whether something arrived and guards against processing the same thing twice. It is hard to name a cheaper element with that much effect on peace of mind around money operations.

Checklist#

• An operation carries an identifier through logs and jobs.
• Failed jobs are visible, not deleted.
• Incoming events have a ledger keyed by the provider identifier.
• Retrying a money operation does not perform it twice.
• A notification never blocks the operation it concerns.
• There is a way to compare your data against the provider data.