81e748b455
Some checks failed
Release & Docker / release (push) Has been cancelled
## Release v0.51.247 — Release HO (stage-q19) Backend correctness fix. ### Fixed | Issue | Author | Fix | |-------|--------|-----| | #3505 | @franksong2702 | **Reasoning effort is coerced to a level the active model/provider actually supports** before each request, instead of being sent verbatim and rejected. `openai-codex` `gpt-5` no longer gets `max` (→ `xhigh`); `o1`/`o3`/`o4` clamp to `low`/`medium`/`high`. Coercion only steps *down* (never escalates); `none`/unset preserved. The capability filter is applied across heuristic / models.dev / Copilot / LM Studio paths. | This is the narrow, correct fix for the detection gap that #3431 tried to address by removing the chip-visibility gate (which we shelved). The chip-visibility gate is **untouched** (Codex confirmed) — `get_reasoning_status`/`_applyReasoningChip` still hide the chip for unconfirmed models. ### Review fix absorbed (Codex + self-flagged) The first cut **dropped** a configured effort for *unrecognized* models, because capability detection returns `[]` for both "known-unsupported" and "simply-unknown" (custom providers, aggregator-rewritten ids, new releases) — that's a behavior change vs master (which sent it verbatim) and would silently disable reasoning. Fixed: an **empty** capability set now **preserves** the configured effort (provider stays the final authority; worst case = the same rejected request master already produces, i.e. no regression). Known-bad clamps return *non-empty* filtered sets, so they still degrade correctly. Nathan chose this "preserve-for-unknown" behavior. + regression test. ### Gate - Full pytest suite: **7548 passed, 0 failed** - ruff: CLEAN · 48 reasoning tests pass (incl. preserve-for-unknown + codex-clamp + never-escalate) - Codex (regression): SHIP-ONLY-WITH-FIXES (unknown-model drop) → fixed → **SAFE TO SHIP** - Verified empirically: gpt-5/codex max→xhigh, o3 max/xhigh→high, unknown high→high (preserved), none/unset preserved Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
133 lines
4.3 KiB
Python
133 lines
4.3 KiB
Python
"""Tests for model-aware reasoning effort chip visibility."""
|
|
|
|
from api import config as cfg
|
|
|
|
|
|
def test_cursor_acp_models_do_not_support_reasoning_effort_levels():
|
|
assert cfg.resolve_model_reasoning_efforts(
|
|
"cursor/composer-2.5",
|
|
provider_id="cursor-acp",
|
|
) == []
|
|
|
|
|
|
def test_openai_codex_gpt5_supports_reasoning_effort_levels():
|
|
efforts = cfg.resolve_model_reasoning_efforts(
|
|
"gpt-5.5",
|
|
provider_id="openai-codex",
|
|
)
|
|
assert "medium" in efforts
|
|
assert "high" in efforts
|
|
assert "xhigh" in efforts
|
|
assert "max" not in efforts
|
|
|
|
|
|
def test_openai_codex_prefixed_gpt5_supports_reasoning_effort_levels():
|
|
efforts = cfg.resolve_model_reasoning_efforts(
|
|
"@openai-codex:gpt-5.5",
|
|
provider_id="openai-codex",
|
|
)
|
|
assert "medium" in efforts
|
|
assert "high" in efforts
|
|
assert "xhigh" in efforts
|
|
assert "max" not in efforts
|
|
|
|
|
|
def test_openai_codex_max_effort_is_clamped_before_streaming():
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"max",
|
|
"gpt-5.5",
|
|
provider_id="openai-codex",
|
|
) == "xhigh"
|
|
|
|
|
|
def test_unsupported_xhigh_degrades_to_high_not_disabled():
|
|
# o1/o3/o4 on openai-codex cap at low/medium/high. A configured xhigh (or
|
|
# max) must clamp DOWN to the highest supported level (high), not silently
|
|
# disable reasoning by returning "".
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"xhigh",
|
|
"o3-mini",
|
|
provider_id="openai-codex",
|
|
) == "high"
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"max",
|
|
"o3-mini",
|
|
provider_id="openai-codex",
|
|
) == "high"
|
|
|
|
|
|
def test_coerce_never_escalates_above_configured_effort():
|
|
# A supported lower effort is returned verbatim; coercion only degrades.
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"low",
|
|
"gpt-5.5",
|
|
provider_id="openai-codex",
|
|
) == "low"
|
|
|
|
|
|
def test_coerce_preserves_effort_for_unrecognized_model():
|
|
# #3505 review: resolve_model_reasoning_efforts() returns [] for BOTH
|
|
# known-unsupported AND simply-unrecognized models (custom providers,
|
|
# aggregator-rewritten ids, brand-new releases). Coercion must NOT silently
|
|
# drop a configured effort just because we don't recognize the model — that
|
|
# would be a behavior change vs sending it verbatim (master). Preserve the
|
|
# configured level for an empty/unknown capability set; the provider stays
|
|
# the final authority. The known-bad CLAMP paths return a NON-empty set, so
|
|
# they are unaffected (covered by the openai-codex tests above).
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"high",
|
|
"some-unknown-model-xyz",
|
|
provider_id="some-custom-provider",
|
|
) == "high"
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"max",
|
|
"brand-new-model-2099",
|
|
provider_id="some-custom-provider",
|
|
) == "max"
|
|
# 'none' / unset still pass through unchanged for unknown models.
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"none", "some-unknown-model-xyz", provider_id="custom"
|
|
) == "none"
|
|
assert cfg.coerce_reasoning_effort_for_model(
|
|
"", "some-unknown-model-xyz", provider_id="custom"
|
|
) == ""
|
|
|
|
|
|
def test_github_copilot_gpt5_supports_reasoning_effort_levels():
|
|
efforts = cfg.resolve_model_reasoning_efforts(
|
|
"gpt-5.5",
|
|
provider_id="github-copilot",
|
|
)
|
|
assert "medium" in efforts
|
|
assert "high" in efforts
|
|
|
|
|
|
def test_openrouter_anthropic_models_keep_reasoning_effort_levels():
|
|
efforts = cfg.resolve_model_reasoning_efforts(
|
|
"anthropic/claude-sonnet-4.5",
|
|
provider_id="openrouter",
|
|
)
|
|
assert "medium" in efforts
|
|
assert "high" in efforts
|
|
|
|
|
|
def test_non_reasoning_http_models_hide_reasoning_effort_levels():
|
|
assert cfg.resolve_model_reasoning_efforts(
|
|
"meta-llama/llama-3.1-8b-instruct",
|
|
provider_id="openrouter",
|
|
) == []
|
|
|
|
|
|
def test_get_reasoning_status_includes_supported_efforts(monkeypatch):
|
|
monkeypatch.setattr(
|
|
cfg,
|
|
"resolve_model_reasoning_efforts",
|
|
lambda *a, **k: ["low", "medium", "high"],
|
|
)
|
|
status = cfg.get_reasoning_status(
|
|
model_id="gpt-5.5",
|
|
provider_id="openai-codex",
|
|
)
|
|
assert status["supported_efforts"] == ["low", "medium", "high"]
|
|
assert status["supports_reasoning_effort"] is True
|