Webhooks Fail. Build Them to Fail Safely.

Why your webhook handler must assume failure from day one

Webhooks represent a direct dependency on a network environment that you do not control. Your team cannot control third-party network routes. You cannot control external service health. You also cannot control the unpredictable timing of incoming HTTP requests. Despite these hazards, many backend developers build their webhooks as if they are reliable, internal API calls. This is a severe mistake that leads to corrupted data.

I have seen production databases suffer deep corruption because of this specific design blind spot. In one payment integration, a Stripe webhook fired twice within fifty milliseconds due to a minor network blip. Our backend processed both requests concurrently. This race condition resulted in a single user getting two active subscriptions. The customer was charged twice, and their profile state was completely broken.

These issues do not occur because your server crashed or threw an error. They occur because the delivery succeeded too many times. External billing platforms use at-least-once delivery semantics. If your system takes too long to respond to an event, the platform assumes the delivery failed. It then sends the event again.

Relying on simple API endpoints to process these events will lead to silent financial losses. Your team must design every webhook handler defensively from the first line of code. Assume that every delivery will eventually fail. It will also time out. Some will repeat themselves. Defending your database against duplicates is the only path to a stable payment flow.

Exactly-once delivery is a distributed systems myth

Many developers construct their applications assuming a payment provider will deliver each webhook event exactly once. This assumption contradicts the basic laws of networking. Expecting a perfect network will guarantee critical state issues in production.

The impossibility of exactly-once delivery is a proven mathematical theorem. It traces back to the FLP impossibility result published in 1985. This research proved that consensus is impossible in an asynchronous network if even one participant can fail. It aligns with the classic Two Generals Problem in computer science.

A network cannot guarantee delivery without feedback. However, that feedback mechanism can also fail. If your server processes a payment and updates the user profile, but the network drops the HTTP acknowledgment on the way back, the payment platform will retry. A duplicate event is born.

Building a reliable system means shifting your perspective. Do not expect the sender to solve this problem. You must construct your backend to handle duplicate events safely. You should acknowledge payloads immediately. You must also deduplicate on stable unique keys. Finally, you must run reconciliation jobs to verify state.

Why application-level existence checks fail under concurrency

The standard approach to avoiding duplicate database records is an application-level guard. A developer writes a database check before running any business logic. The code queries the database to see if it already processed the event. If the record exists, the code exits early.

This approach fails completely during concurrent requests. When two duplicate requests hit your server within the same millisecond, they execute in parallel. Both threads perform the check query at the same instant. Both threads search the database. Both find no existing record. Then, both proceed to execute the write operation.

In my own work, relying on these simple database checks has been the single greatest cause of duplicate subscription bugs. The database read and the database write are separate actions. They are not isolated from other web requests. When traffic spikes, these race conditions occur with alarming frequency.

These overlaps are common when handling high-value payment webhooks. A minor network bottleneck can delay an event, causing Stripe to trigger an automated retry. Your system must enforce integrity at the database layer rather than the application layer to block duplicate writes.

Treat event processing as atomic database transactions

To guarantee database integrity, you must delegate concurrency checks to your database engine. Databases are built to manage concurrent operations safely. They use atomic transactions and unique index constraints to prevent conflicting operations. This database layer is where your application achieves true safety.

You must create a unique index on the provider event identifier column. When integrating Stripe, this identifier is the unique event ID. This column is your ultimate defense against duplicate payloads.

I wrap the entire webhook processing flow in an explicit database transaction. When a concurrent duplicate request arrives, the database engine blocks the second insert. The unique key constraint forces a database error. The database halts the second write before your application can duplicate any credit allocations.

Your application code must catch this specific database violation exception. Do not throw a server error back to Stripe. Instead, catch the unique constraint error, log the duplicate attempt, and return a clean success status code.

The external payment platform receives your successful HTTP status code, which halts the retry loop. Your database remains consistent. The user does not receive double credits, and your financial records stay clean.

CREATE TABLE processed_events (
    event_id VARCHAR(255) PRIMARY KEY,
    provider VARCHAR(50) NOT NULL,
    processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Acknowledge requests quickly to prevent timeout-induced retries

Slow execution in your own application is the most common cause of duplicate webhook deliveries. If your handler takes too long, the provider assumes delivery failed.

When an event arrives, your server might attempt to run multiple complex tasks. It might call a third-party provisioning API. It might also generate a PDF invoice. It might then try to dispatch a welcome email. Running these tasks synchronously consumes valuable time.

Webhook providers enforce strict limits on response times. Stripe waits only a few seconds before timing out. Shopify uses a strict connection timeout of one second and a total request timeout of five seconds. If your code takes longer than these limits, the provider aborts the request.

The sender immediately schedules a retry attempt. Meanwhile, your original slow process finishes successfully in the background. Now, you have a duplicate retry request running on your server, which will double-process the event.

You must decouple webhook reception from webhook processing. You should verify the signature. You must write the raw event payload directly to a database queue table. Finally, you should return an immediate status code.

This approach keeps your HTTP response times under one hundred milliseconds. A separate background worker picks up the queued items. This background worker executes the slow business logic asynchronously without blocking the incoming webhook.

Route exhausted events to a dead letter queue

Sometimes background workers fail repeatedly due to an unhandled exception. They might also fail due to a database deadlock. An external API outage can also halt them. Retrying the event endlessly will clog your message queues. It will also overwhelm your primary database. You must isolate these errors to keep your system healthy.

You must implement a system to capture these terminal failures without discarding the event payload. A dead letter queue is a dedicated storage table for failed webhook payloads.

When a worker exhausts its retry limit, the system moves the event to the dead letter queue. The system records the raw payload in the table. It also stores the event ID. It then logs the specific error message.

This architecture protects your system against permanent data loss. Your development team can inspect the failed payloads. They can identify the bug, deploy a patch, and trigger a manual retry utility to process the event. This prevents silent payment processing failures.

async function handleWebhook(event) {
  try {
    await processEvent(event);
  } catch (error) {
    await db.query(
      'INSERT INTO webhook_dlq (event_id, payload, error) VALUES ($1, $2, $3)',
      [event.id, JSON.stringify(event), error.message]
    );
    throw error;
  }
}

Verify raw body signatures to protect your endpoints

Processing unverified payloads on your webhook endpoints introduces critical security vulnerabilities. An attacker could forge an invoice payment event. They could gain unauthorized access to premium features without ever paying your business.

Payment platforms sign their webhook payloads using a shared secret key. Your application must verify this cryptographic signature before parsing the payload or triggering any internal workflows.

Signature verification requires raw, unmodified request bytes. If your backend framework automatically parses JSON, it can alter whitespace. It might also change character encoding. Sometimes it reorganizes the key order. These minor mutations will break the cryptographic comparison.

Configure your web framework to capture the raw request stream before any JSON body parser executes. Use the official SDK provided by the payment platform to perform the verification. Ensure your servers synchronize their clocks using NTP to prevent clock skew validation failures.

A failed signature check must trigger an immediate HTTP error code. This alerts you to unauthorized requests. It also stops malicious actors from discovering valid endpoints through brute-force attempts.

Catch missed events with periodic reconciliation jobs

Webhooks travel across public, unpredictable network paths. A DNS routing issue can block an event before it reaches your application server. A firewall misconfiguration can do the same. An internet service provider outage can also disrupt the route.

Because of these networking limitations, you cannot rely on webhooks as your sole source of truth. You must build an independent, out-of-band reconciliation mechanism to guarantee state consistency.

This safety net is a scheduled background job, such as a nightly cron task. The job queries the payment provider's API directly to retrieve every successful transaction from the previous twenty-four hours.

The job iterates through the retrieved list of transactions. It checks each transaction ID against your local database of processed events. If the database does not contain the event, the job inserts the missing payload into your processing queue.

This dual-channel design ensures that your database remains correct. Even if a major network outage drops every webhook for six hours, your systems will correct themselves. You recover lost revenue without requiring manual engineering interventions.

An actionable checklist for production-ready webhooks

Building resilient webhook integrations requires defensive backend patterns. You can use this checklist to evaluate your webhook handler before exposing it to production traffic. It will protect your billing records from catastrophic state errors.

Create a unique database constraint on the provider event ID column. This database constraint prevents concurrent duplicate requests from executing twice.

Wrap your database mutations in explicit database transactions. Catch the unique key violation error and return a success response code to the provider.

Grab the raw request body bytes before your framework parses the JSON. Use the official SDK to perform the signature verification.

Respond to the webhook provider within one second. Offload heavy processing tasks to an asynchronous background queue.

Send events that fail repeatedly to a dedicated dead letter queue. This step preserves the data so your team can debug.

Configure a daily scheduled job to query the payment provider's API. This task reconciles any missed events and repairs silent state drift.