Handling LLM Rate Limits in Laravel Without Breaking Your Queue

Why Your LLM Requests Belong in the Queue

I often see developers firing off Http post calls to LLM providers directly from a web controller. This is a design flaw. An LLM might take fifteen seconds to stream a response. Your web server has a finite number of processes. Locking one for the duration of a slow inference call ruins your application concurrency. Once the process hits its timeout, the user is left with a 504 Gateway Timeout and a half finished job.

The only scalable architecture is to treat AI inference as an asynchronous background operation. When a user submits a prompt, your controller should validate the input and immediately return a 202 Accepted response. The actual heavy lifting shifts to a Laravel queue worker. This decouples your web traffic from the volatility of external AI APIs. If the model is slow, only the background worker feels the pressure. Your front end stays responsive and fluid.

Synchronous requests turn your web server into a bottleneck. By moving this work to a queue, you gain the ability to manage retries, inspect job state, and implement sophisticated error handling that simply cannot occur within the request cycle. A background worker provides the persistence required to survive the long, unpredictable response times that define modern generative AI.

Stop Hammering APIs with Tight Retry Loops

When you receive a 429 Too Many Requests response, the worst thing you can do is retry immediately. Standard Laravel queue retries often default to immediate execution. If your job hits a rate limit five times in fifteen seconds, you have not actually tried to fix the issue. You have simply wasted five attempts and potentially invited a temporary IP ban from the provider. Tight retry loops are the fastest way to get your API keys blocked.

Rate limits are not fatal errors. They are explicit signals from the provider asking you to wait. A professional implementation reads the Retry After header and releases the job back to the queue with a matching delay. This preserves your retry budget and respects the provider infrastructure. By explicitly managing the backoff, you transform a failed request into a scheduled attempt that has a genuine chance of succeeding.

Many developers treat 429 responses as unexpected anomalies. They are not. They are a core, expected component of operating at any significant scale. Your software must be designed to gracefully pause its activity when the signal arrives. By pausing, you maintain the health of your application ecosystem and ensure your workers remain available for tasks that are not currently throttled.

Essential Job Properties for Inference Workloads

Your job class is the boundary for interaction with the LLM. If you do not define strict timeouts, you risk hung processes that consume workers while doing nothing. I set both connection and request timeouts on every Guzzle or Laravel HTTP client instance. A hanging network request is a silent killer of throughput.

In addition to timeouts, you must define tries and backoff properties on your Job class. Without a defined backoff, Laravel will treat all retries as immediate. I recommend setting a backoff array, such as sixty, one hundred twenty, and three hundred seconds. This ensures your workers give the provider breathing room after consecutive failures. This keeps your queue clean and prevents a backlog of failing jobs from overwhelming your database connection pool.

The retry until method is your safety valve for these jobs. An AI request that has been stuck in the queue for an hour is usually worthless by the time it executes. Define an explicit window for expiration. If the work is no longer relevant, stop trying. This saves your compute resources and prevents obsolete data from ever hitting your application.

Implementing Exponential Backoff via Job Release

To handle rate limits effectively, wrap your API call in a try catch block specifically designed to identify 429 status codes. Do not rely on generic exception handling. When you catch the 429, inspect the response headers for Retry After. If the header is missing, fall back to a sensible default like sixty seconds. This logic must be explicit to avoid silent failures.

The key is using the release method. This instructs the queue to requeue the job with a specific delay. Crucially, this does not increment your job retry count. It treats the rate limit as a scheduling adjustment rather than a failure of logic. If the job hits the maximum number of retries, it should then fail. Until that point, you are simply pausing and waiting for the provider to recover from your request volume.

Tracking the failure state is necessary for the end user. If a job is released to the queue, update your database record. Mark the status as retrying or pending. This visibility prevents the user from thinking the system is broken when it is simply negotiating a high volume of traffic. Transparency is the difference between a bug report and a satisfied customer.

Code Implementation

The following implementation pattern ensures the job respects provider rate limits while keeping the queue state transparent and robust.

public function handle() {
  $response = Http::timeout(30)->post('https://api.openai.com/v1/chat/completions', $this->payload);
  if ($response->status() === 429) {
    $retryAfter = $response->header('Retry-After', 60);
    return $this->release($retryAfter);
  } if ($response->failed()) {
    throw new Exception('API request failed: ' . $response->body());
  } $this->handleSuccess($response->json());
}

Per User Fairness and Global Throttling

Rate limits usually apply to your entire API account. If one user triggers a loop of twenty requests, they can exhaust your organization quota and impact every other user. I use the queue as the single enforcement point for per user throttling. By applying Laravel rate limiting middleware to the job, you can ensure that even if the system is healthy, one heavy user cannot monopolize the provider budget.

This pattern creates a double layered defense. The middleware throttles the user locally, preventing the job from ever being dispatched if they are too aggressive. The job logic handles the global provider rate limit if the system wide threshold is reached. By separating these two concerns, you ensure your application remains responsive for everyone while protecting your costs and your relationship with the API vendor.

Individual user throttling acts as a circuit breaker for your business logic. When one account exceeds their fair share, you block their ability to saturate the queue. This preserves the remaining capacity for other users who are behaving within normal parameters. It is an essential component of any multi tenant system relying on expensive third party APIs.

Actionable Checklist for LLM Production Readiness

Before you deploy, verify these points to ensure your queue backed AI features will not collapse under production volume.

First, set strict connect and request timeouts on all outgoing HTTP clients to prevent process starvation. Second, ensure your job uses a custom backoff strategy that respects 429 Retry After headers rather than immediate retries. Third, implement a visible state in your database so the user interface can communicate retry status to the user. Fourth, register global rate limiters to prevent individual users from consuming your entire organization API quota. Finally, configure a dead letter state to catch jobs that finally exceed their retry count so you can debug the payload later.