Skip to content

Derive LiteLLM reasoning capabilities from local metadata - #2240

Open
Selene29 wants to merge 1 commit into
repowise-dev:mainfrom
Selene29:fix/litellm-model-capabilities
Open

Selene29 wants to merge 1 commit into
repowise-dev:mainfrom
Selene29:fix/litellm-model-capabilities

Conversation

@Selene29

Copy link
Copy Markdown
Contributor

Summary

  • Build LiteLLM model options from the loaded local model metadata.
  • Advertise exact reasoning efforts from explicit levels and capability flags.
  • Use the same metadata interpretation for discovery and runtime validation.

Problem

LiteLLM discovery offered a fixed low, medium, and high set for reasoning models, while execution used a separate capability lookup. The two paths could disagree about which effort values a model accepts.

Catalog construction also consulted per-model helpers and included non-text model categories. This change keeps enumeration local, filters those categories, and treats explicit negative or always-on metadata as authoritative.

Testing

  • 6 LiteLLM provider tests passed
  • Ruff lint and format checks passed
  • Whitespace checks passed

Catalog construction called reasoning helpers for every model and offered
a fixed low/medium/high set. Read the loaded model_cost metadata when
building model options, derive effort choices from explicit levels and
capability flags, and filter known non-text model categories.

Honor explicit negative capability metadata and thinking-always-on
constraints. Prefer local metadata for runtime validation, retaining the
existing capability lookup fallback when metadata is absent.

Validation:
- 6 LiteLLM provider tests passed with SDK calls mocked.
- Ruff lint/format checks and git diff --check passed.
@RaghavChamadiya

Copy link
Copy Markdown
Member

Thanks @Selene29, reading the loaded catalog once is the right direction, and keeping mode: responses visible is correct. LiteLLM's acompletion path detects that mode and routes through its Responses API bridge, so those models are compatible with this provider's call.

There is still one discovery/runtime split in the reasoning path. _litellm_model_options derives modes only from model_cost.get(model_id), while _litellm_supported_reasoning_modes falls through get_model_info() and supports_reasoning() when that direct row has no decisive fields. This disagrees in the LiteLLM 1.90.2 catalog we lock today: eight current model_list entries, including vercel_ai_gateway/deepseek/deepseek-r1 and vercel_ai_gateway/xai/grok-3-mini, have no direct reasoning fields but supports_reasoning() returns true. Discovery offers only auto; runtime accepts low, medium, and high. There are also 122 model_list IDs absent from model_cost, so the catalog-only branch is not a total mapping.

Please make an indecisive or missing catalog row take the same fallback in both paths, and add a regression with a model present in model_list but absent from model_cost where get_model_info or supports_reasoning supplies the capability. The existing agreement test puts the same metadata in both maps, so it cannot catch this case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants