Laravel Queue Reliability: Failed Jobs, Retries, and Horizon in Production

Why queue reliability is defined by your failure strategy

A background queue that operates properly only when every external dependency responds cleanly is not an architecture. It is an assumption. When you shift work to a background worker, you gain responsiveness in the HTTP request cycle but inherit a complex set of distributed system risks. Reliability is a commitment to managing partial failures, acknowledging that network timeouts, service outages, and memory exhaustion are common events.

All my AI features run through queue workers. The web request returns a 202 status immediately, while a job executes the inference call, persists the output, and notifies the frontend. That design only stays honest if the queue layer is resilient. Reliability relies on granular failed job handling and a deliberate retry policy that does not overwhelm the provider or your own database connections.

Production reliability requires designing for inevitable failures. If a worker crashes, the system must recover the state without human intervention. This means enforcing strict retry ceilings and providing operational visibility into jobs that cannot self-heal. Without this layer, your queue is simply a place where tasks go to disappear when the environment becomes slightly unstable.

Configuring explicit retry limits and exponential backoff

Default queue settings are dangerous for production because unconfigured workers retry failures immediately. Without a backoff strategy, a service returning a 429 rate limit error will be hammered repeatedly, ensuring your API key gets blocked. Retry policy needs to be explicit per job class. You must define the number of attempts and the exponential backoff duration. A job that exhausts its attempts must land in the failed jobs table with its full context intact.

If a downstream service returns a temporary error, immediate retries are a recipe for failure. By providing a backoff array, you instruct Laravel to increase the interval between each successive attempt. This gives the external service time to recover and keeps your queue throughput manageable. You should also set the job timeout property higher than your client-level network timeouts. If your HTTP client waits ninety seconds, a job timeout of sixty seconds causes the worker to terminate before your retry logic can execute.

<?php namespace App\Jobs;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Throwable;
class ProcessAiCompletion implements ShouldQueue {
  use Queueable;
  public int $tries = 4;
  public int $timeout = 120;
  public array $backoff = [15,
    60,
    300];
  public function handle(): void {
  } public function failed(?Throwable $exception): void {
  }
}

Guaranteeing idempotency with database unique constraints

Queue systems operate on at-least-once delivery guarantees. This means duplicate executions are a normal operational event rather than an anomaly. Every job must remain safe to run multiple times without introducing side effects. When you rely on application-level checks, such as querying the existence of a record before acting, you create a race condition. Two workers can perform the same check, find no record, and both decide to perform the work concurrently.

The only way to guarantee safety under high concurrency is to delegate integrity checks to the database schema. A unique index on a transaction ID or event identifier acts as an atomic lock. When the second worker attempts to write the duplicate record, the database storage engine rejects the transaction. This forces the duplicate job to fail harmlessly while the initial successful job completes. Application-level checks are merely suggestions. Database constraints are absolute laws.

<?php use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration {
  public function up(): void {
    Schema::create('processed_events', function (Blueprint $table) {
      $table->id();
      $table->string('event_id')->unique();
      $table->string('status');
      $table->json('payload');
      $table->timestamps();
    });
  }
};

Inspecting failed jobs and replaying raw payloads safely

When a job exhausts all assigned retry attempts, Laravel serializes the job state into the failed_jobs database table. This record captures the original payload, the connection, the queue name, and the entire exception stack trace. This store prevents data loss and acts as a dead-letter queue. You can review why jobs stalled by querying the database or using command-line tools. Having this history is essential when debugging remote failures that occur in production but cannot be reproduced in local testing environments.

Once you resolve the underlying issue, such as deploying a code fix or waiting for a service outage to pass, you can replay the jobs back onto the queue. You do not need to ask users to re-submit actions. By using the uuid from the failed jobs list, you can selectively retry individual jobs or flush the entire failed list if needed. This process turns a critical system failure into a manageable recovery task. It keeps the business running without needing manual data entry or state correction.

# List all failed jobs
php artisan queue:failed

# Retry a specific job by UUID
php artisan queue:retry ce7bb17c-cdd8-41f0-a8ec-7b4fef4e5ece

# Forget a corrupted job entry
php artisan queue:forget 91401d2c-0784-4f43-824c-34f94a33c24d

Tuning Laravel Horizon supervisors for mixed workloads

Running fast transactional jobs and slow background inference workloads on a shared supervisor is a design error. Notification jobs should finish in milliseconds. AI inference jobs often require tens of seconds to stream data. When high traffic arrives, the slow inference jobs occupy every available process, causing a queue backlog for critical tasks. This creates a hidden performance bottleneck where the queue depth increases despite having available workers.

Horizon lets you define multiple worker supervisors in your configuration file. You should isolate your workloads by latency profile. Give the AI jobs a separate supervisor pool with specific timeout and process limits. This ensures that a spike in AI requests cannot seize the workers responsible for sending emails or processing payments. Code-driven configuration means you can track these supervisor settings alongside your application code in version control. You can tune the process counts and balances independently based on the throughput requirements of each service queue.

return ['environments' => ['production' => ['supervisor-default' => ['connection' => 'redis',
        'queue' => ['default'],
        'balance' => 'auto',
        'maxProcesses' => 10],
      'supervisor-ai' => ['connection' => 'redis',
        'queue' => ['ai-inference'],
        'balance' => 'simple',
        'maxProcesses' => 6,
        'timeout' => 180]]]];

Setting alerts for queue depth, failure rates, and stuck workers

Monitoring queue health requires active telemetry. Checking dashboards manually is not enough. You need alerts that surface issues before users begin reporting broken experiences. Track the p95 execution time rather than the average, as a few runaway jobs can easily skew the mean while masking real latency problems. Alert your engineering team when your high-priority inference queues show sustained growth in wait times or queue depth.

A failure rate above two percent suggests systemic issues, like an incompatible deployment or a downstream provider outage. Monitor your retry rates alongside total failures. If your retry rate climbs past fifteen percent, it is likely that your job backoff logic is being triggered by widespread rate limiting. In extreme cases, workers can vanish silently due to memory leaks or host process termination. You should implement a heart-beat check or compare job dispatch counts against completion counts to detect if workers have dropped off the grid without creating a record in the failed jobs table.

Adding reconciliation jobs to catch silent drops

Even with perfect configurations, infrastructure failures can cause jobs to terminate without reaching the failed jobs table. Node restarts, memory pressure, or unexpected hardware failures can cause a process to die instantly. Reconciliation is your safety net. You should run a scheduled task every fifteen minutes that scans your database for records stuck in a processing state. This loop verifies what should exist against what currently exists.

If a task has been marked as processing for over twenty minutes, you should re-dispatch the job or explicitly mark it as failed for human review. This reconciliation logic fills the gap left by network interruptions that skip the standard error handling lifecycle. It turns silent failures into identified anomalies. This script is often the most vital component of a resilient system, as it catches the edge cases that human operators would never see.

<?php namespace App\Jobs;
use App\Models\InferenceTask;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
class ReconcileStuckTasks implements ShouldQueue {
  use Queueable;
  public function handle(): void {
    $threshold = now()->subMinutes(20);
    InferenceTask::where('status', 'processing')->where('updated_at', '<', $threshold)->chunkById(100, function ($tasks) {
      foreach ($tasks as $task) {
        $task->update(['status' => 'stalled']);
        ProcessAiCompletion::dispatch($task->id);
      }
    });
  }
}

Production readiness checklist

Verify your configuration against this list before the next deployment. Define explicit retry attempts and backoff intervals for every background job. Keep your HTTP client connection timeouts shorter than the job execution timeout. Use unique database indexes on all processed event IDs to block race conditions. Isolate long-running tasks into dedicated Horizon worker pools to protect your high-speed transactional queues. Build alert triggers for queue depth and p95 latency. Implement a nightly or hourly reconciliation job to catch records stranded by silent worker termination. Following these steps transforms a fragile queue into a robust pillar of your infrastructure.