LLM Retry vs Fallback: Reliability Patterns Without Multiplying Cost

A retry sends another request after a timeout, transport error, rate limit or other failure condition. The key question is whether the failure is transient. Retrying a deterministic configuration error will only repeat the failure and create more traffic.

Use bounded retry counts, explicit retryable conditions and backoff where appropriate. The retry should remain attached to the same trace or logical operation so operators can see that one user action generated multiple provider calls.

Fallback sends work to a different provider, model or configured route after the preferred path becomes unavailable or unhealthy. That can restore service when a provider is degraded, but the alternate route may have a different context window, response style, latency profile or price.

For workflows where output behavior is sensitive, fallback should be an explicit product decision rather than an invisible infrastructure trick. A successful HTTP response is not enough if the alternate model changes the meaning or quality of the task.

Every additional inference attempt can consume tokens and provider capacity. If a slow request times out after substantial generation and is then retried, the user may see one response while the platform pays for more than one attempt. Fallback can create the same effect.

Track cost per logical operation as well as cost per provider call. That makes reliability overhead visible and prevents a policy that improves success rate from quietly doubling the economics of a workload.

A model call itself may be safe to repeat, but an agent workflow can include tools that send email, update a record or trigger another system. Retrying the entire workflow without idempotency can therefore repeat real-world actions.

Separate model-call retry policy from tool-execution policy. Preserve stable operation identifiers and make side-effecting tool calls idempotent or explicitly non-retryable where the downstream system cannot guarantee safe repetition.

Start by classifying failures: provider timeout, rate limit, network error, invalid request, model unavailable or application cancellation. Then measure which conditions are transient and how much latency and spend each recovery attempt adds.

A good reliability dashboard should answer how often retries happen, how often they recover, which provider or model triggers them and what they cost. The same evidence should exist for fallback so the team can tune policy instead of treating it as permanent configuration.