harshaA fallback that returns a response can look like a reliability success. Whether it actually helped depends on what the caller expected from that response.
LiteLLM supports fallback model groups after failures and configured retries. That provides an important mechanism, but a mechanism is not the whole application contract. The model selected next may have different behavior, limits, or supported capabilities. The caller still needs an answer it can use. LiteLLM's fallback documentation describes the routing mechanism.
Define compatibility before the outage
Consider an illustrative application that expects structured output with a particular schema. If its backup returns persuasive prose instead, the request may be technically successful while the workflow has failed. A second application may rely on a tool call that the backup handles differently. A third may be subject to data-routing restrictions.
The right fallback therefore starts with requirements: accepted output shape, required tools, data constraints, time budget, and cost boundary. A provider should qualify against those requirements before it becomes part of the recovery plan.
I would rather document a limited, explicit fallback than imply that every model is interchangeable.
Partial output changes the problem
Before a response starts, switching providers can be relatively straightforward from the user's perspective. After content has been displayed, the application has a harder decision. Restarting may duplicate text or contradict the unfinished answer. Continuing with another model may not preserve the same context or meaning.
The interface should know whether it has a complete result, a recoverable interruption, or an incomplete artifact that needs a fresh attempt. A single generic success flag cannot express all three.
This is why testing only a provider's first-request failure leaves a gap. Failure after visible output deserves its own scenario.
Budget the recovery
Retries and fallbacks also consume time and potentially money. A recovery chain should fit inside a request's overall deadline. Otherwise, the application can keep trying after the user has already abandoned it.
A dependable fallback is a deliberately narrower promise: if the preferred path fails under specified conditions, this alternate path can still satisfy these requirements. That promise is much easier to test than “the system always finds another model.” It is also more honest about the boundaries of resilience.