Webhook Idempotency: Defending Your Database from Duplicate API Events

Why at-least-once delivery guarantees duplicate webhooks

Every major webhook provider operates on an at-least-once delivery guarantee. This is a fundamental constraint of network communication. When a platform sends an event to your server, it waits for an HTTP 2xx response to mark the job complete.

If your network drops, your server crashes, or your application takes too long to respond, the provider assumes the delivery failed. It then schedules a retry. These retries are inevitable. If you design your backend to assume a one to one mapping between events and processed records, you will eventually corrupt your data state.

You must stop thinking about events as simple function calls. Instead, view them as asynchronous messages that might arrive in any order or volume. This perspective shift is necessary to build a truly robust integration.

The danger lies in the assumption of perfection. Developers often build for the ideal sunny day scenario. They ignore the reality of distributed systems. A blip in the infrastructure can cause a cascade of duplicate events that your system will be forced to handle.

Your system is a black box to the external provider. It does not know if you failed to process the event or if the acknowledgment failed to travel back across the network. It only knows it received no confirmation. Consequently, it sends the data again. This loop is the foundation of the at-least-once delivery pattern.

Why unique constraints beat application-level checks under concurrency

Many developers attempt to prevent duplication by querying the database before taking action. They check if an event ID exists and only proceed if it is missing. This pattern is fundamentally broken in concurrent environments.

When two identical webhook requests arrive within milliseconds, they often hit two separate threads or processes. Both threads perform the existence check at the same time. Both conclude the record is missing. Both proceed to write the data. This race condition is a leading cause of double charging customers or issuing duplicate SaaS credits.

I have watched production databases get corrupted because of this specific logic. The separation between reading state and writing state creates a time window for disaster. In high traffic periods, these windows close with alarming speed.

You cannot solve race conditions with application logic. The database is the only authority that can guarantee the state. By the time your code checks for an ID, the concurrent request has already performed its own query. You need an approach that locks the operation at the storage level.

Relying on code-level checks is a form of optimism that your production environment cannot afford. You must treat every incoming webhook as a potential duplicate. Do not leave the gate wide open for these race conditions to ruin your financial data.

Treating state changes as explicit database events

To stop duplicates, you must delegate integrity checks to the database itself. Databases are designed to handle concurrent operations through atomic transactions and unique constraints. You should map every incoming webhook to a dedicated idempotency table in your schema.

Create a unique index on the provider provided event ID. When the second request tries to insert the same event ID, the database engine will reject the write at the storage layer. Your application should wrap the entire logic in a transaction. This ensures that business state updates only occur if the idempotency record is created successfully.

The following SQL structure provides the necessary guardrail for your integration.

CREATE TABLE processed_webhooks (id SERIAL PRIMARY KEY,
  event_id VARCHAR(255) UNIQUE NOT NULL,
  created_at TIMESTAMP DEFAULT NOW());

This table is your new primary defense. When you insert an ID, the database handles the collision. If the insert succeeds, you continue. If it fails due to a unique violation, you know the event is a duplicate. This happens in a single, atomic step that no race condition can bypass.

Apply this pattern to your primary business logic too. Do not store status as a delta. Instead, store it as an absolute state. Setting a status to paid is a safe operation that can run multiple times without consequence. Incrementing a balance, however, is a dangerous operation that requires total protection.

Handling unique constraint violations gracefully

When your database rejects a duplicate insert, you must handle the error silently. If you bubble up a 500 server error, the provider will think your system is failing. It will continue to retry the event, potentially causing a feedback loop.

Catch the specific unique constraint exception in your application code. Once caught, you must return a 200 OK status code. This signals to the third party gateway that the event has been accounted for. It successfully stops the retry mechanism.

I have spent hours debugging systems that lacked this logic. The provider would retry endlessly, causing my error logs to overflow with irrelevant noise. You must treat a duplicate event as a successful event. It is a win, not an error.

Log the incident at the info level, but do not trigger alerts. The user does not care about retries, and your support team does not need to intervene. You are successfully maintaining data integrity by acknowledging the request and moving on.

This handling keeps your system healthy during high load. It prevents unnecessary processing while satisfying the provider's need for a clean response code. It turns a potential crisis into a quiet, routine event.

Decoupling webhook ingestion from background worker processing

Webhook providers often have strict timeout windows. Some are as low as one or two seconds. If your code tries to call a heavy third party API or generate a PDF report within that same request, you will inevitably hit a timeout. A timed out request is just another trigger for a retry.

The optimal pattern is to ingest the raw payload into a message queue or a database pending table immediately. Perform the minimum required verification, such as signature checking, and return an HTTP 200 response. Process the heavy business logic asynchronously in a background worker. This ensures your response is always fast, significantly reducing the likelihood of retries.

Speed is an idempotency strategy. The faster you respond, the less likely you are to enter a timeout loop. A queue acts as a buffer. It isolates the incoming request from the variability of your backend tasks.

This approach also aids in debugging. When a job fails, you have the raw payload stored in a table or queue. You can inspect it, fix the bug, and replay it manually. This is a much better workflow than losing the event entirely because of a crash during a synchronous process.

I recommend building a dead letter queue for events that exceed your retry limit. If a background worker fails five times in a row, move the payload to a separate table. Do not delete it. Let a developer examine it later.

Catching missed events with scheduled reconciliation

Even with perfect idempotency, webhooks can be lost entirely. Network routing failures or DNS issues can block an event before it ever touches your server. Do not assume your webhook log is the complete source of truth.

Implement a nightly reconciliation job. This job should query the payment provider's API directly to list all transactions from the last twenty four hours. Compare this list against your internal database of processed events. If a transaction exists on the provider side but is missing in your system, trigger the ingestion logic manually to fill the gap. This dual source approach ensures data consistency regardless of transient network failures.

This job is your final safety net. It corrects silent failures that webhooks cannot catch. I sleep better knowing that my nightly task validates every state change against the external authority.

It keeps your financial books clean. If an ISP drops a connection or a firewall misbehaves, your system recovers automatically. You do not need a support ticket to find missing payments. The cron job finds them for you.

Use the provider's SDK to list events by date. Iterate through them and verify that your event ID matches. If it is missing, create a synthetic webhook trigger. It creates a robust bridge between the provider's truth and your own.

An actionable checklist for production readiness

Before you deploy your next webhook handler, walk through this list to ensure you are not leaving your database vulnerable to duplicate events or timeout induced retries.

Verify the signature using the provider's official SDK before parsing the body. Add a unique index to your database for the event ID. Wrap all logic in a database transaction and catch unique constraint violations as successes. Save raw payloads to a queue and return a 200 OK within 500 milliseconds. Build a daily background task to verify your state against the source API. Move events that cause permanent processing errors into a separate table for manual inspection.