Litelm Keeps the Tool Call, Drops the Fallback

Litelm carried a tool-call ID through two HTTP requests in my local test. Then I passed it a fallback model and made the first route return 429. It raised RateLimitError after one request. It never tried the fallback.

Both outcomes fit the implementation. Litelm provides a small provider adapter with message translation and familiar response types. An application migrating a completion call also has to account for policy: retries, fallback, caching and spend limits. Matching a function signature leaves room for different behavior.

The solid paths represent code inspected or exercised here. The application policy box marks responsibilities outside this adapter.

A Small Runtime, a Larger Test Environment

I checked version 0.5.2 at commit 4a260c7dd47f900d7a40deae98b86f3822c11c16, on Windows with Python 3.12.8 and uv 0.12.10.

The package manifest declares two direct runtime dependencies, openai and httpx. Anthropic and Bedrock support add optional SDKs. The development group pulls in DSPy and its dependencies, including LiteLLM. My all-extras development install added 79 packages. That is the test environment's size; it does not describe the minimal runtime install.

I started with the documented command:

uv run --extra all pytest tests/ -x --ignore=tests/ported --timeout=10

The first run timed out in test_context_window_prompt_too_long. Its traceback was inside the Anthropic SDK import path. I reran with a 60-second per-test limit:

uv run --extra all pytest tests/ -x --ignore=tests/ported --timeout=60

collected 317 items
262 passed, 55 skipped in 12.32s

The second run used a warm filesystem and a larger timeout, so I cannot isolate which change resolved the stall. The 55 skips cover live-provider and DSPy smoke tests. I did not supply provider credentials or run those tests. I also excluded the imported upstream suite, as the command shows. These results establish a useful local baseline, not complete LiteLLM compatibility.

Two Requests on Loopback

The probe script runs a temporary HTTP server on 127.0.0.1 and an OS-assigned port. It uses the real Litelm call path and OpenAI SDK against fixture responses. No model produces the answers in this test.

From the checked-out project, with the script saved one directory above it:

uv run --frozen --extra all python ../litelm-probe.py

The first call uses openai/fixture-model, a function named test_summary, and tool_choice="required". Litelm sends fixture-model to /v1/chat/completions; the provider prefix belongs to local routing. The fixture replies with call_42 and the argument string {"suite":"unit"}.

The harness parses that argument, supplies a fixed tool result, and sends another completion request. The second request contains this tool message:

{
  "role": "tool",
  "tool_call_id": "call_42",
  "content": "{\"passed\":14,\"failed\":1}"
}

The script asserted the request paths, model name, argument value and matching ID. Its output was:

tool round-trip: 2 HTTP requests, call_42 preserved, arguments parsed
normalized response: 14 passed, 1 failed

This verifies request serialization and response normalization around a tool exchange. It does not test whether a model selects the correct tool. The fixture chooses the call, and the harness supplies the result.

Following the Adapter Boundary

The code is small enough to trace without starting a proxy service:

Module	Responsibility	Detail worth checking
_providers.py	Resolve model prefix, URL and credentials	api_base permits a local fixture endpoint.
_completion.py	Prepare arguments and dispatch calls	Some accepted compatibility arguments disappear here.
_dispatch.py	Load native provider handlers	Anthropic, Bedrock, Cloudflare and Mistral use custom handlers.
_client_cache.py	Reuse SDK clients	Cache keys include endpoint, credential and retry configuration.
providers/_anthropic.py	Translate message and response structures	Tool arguments become objects; tool results move into user content blocks.

The dispatch registry imports a native handler when needed. OpenAI-compatible routes use the SDK path. The client cache caches connections, not completion answers. Keeping those two meanings of cache separate helps when reviewing a migration.

I also called the Anthropic translation helper on the same conversation. It emitted:

[
  {
    "role": "assistant",
    "content": [{
      "type": "tool_use",
      "id": "call_42",
      "name": "test_summary",
      "input": {"suite": "unit"}
    }]
  },
  {
    "role": "user",
    "content": [{
      "type": "tool_result",
      "tool_use_id": "call_42",
      "content": "{\"passed\":14,\"failed\":1}"
    }]
  }
]

That check covers an in-process translation function. I did not send this payload to Anthropic. The useful invariant is concrete: the result still points to the same tool use after the format change.

The Argument That Disappears

For the failure case, I changed the requested model to openai/fixture-down. The local server returned a JSON error with status 429. The call included:

litelm.completion(
    "openai/fixture-down",
    messages=messages,
    fallbacks=["openai/fixture-model"],
    num_retries=0,
    api_key="fixture-key",
    api_base=local_base_url,
)

The script observed:

fallback probe: RateLimitError, 1 HTTP request, no fallback attempted

It also asserted that the outgoing request omitted fallbacks. In _prepare_call, the adapter removes fallbacks, cache, caching and retry_strategy. It maps num_retries or max_retries to the SDK client's retry count, which defaults to zero in this path.

A caller can therefore pass familiar arguments without receiving their familiar policy. The package describes a limited routing and formatting surface in its README. I still prefer a visible warning or rejection for unsupported policy arguments: silent acceptance makes an import-only migration look safer than the actual contract.

A successful tool exchange and a failed fallback expectation can coexist in the same adapter.

For an application that already owns failover, this separation is useful. You can keep a narrow transport layer and test its payloads. For an application that relies on a completion library to choose another provider after a failure, replacing the import also requires moving that decision somewhere explicit.

The migration check I trust is the one above: make the first route fail and count the requests. A healthy tool-call round trip gives evidence about message compatibility. The failing route exposes who owns recovery.