Release v0.51.247 — Release HO (stage-q19) (#3521)
Some checks failed
Release & Docker / release (push) Has been cancelled
Some checks failed
Release & Docker / release (push) Has been cancelled
## Release v0.51.247 — Release HO (stage-q19) Backend correctness fix. ### Fixed | Issue | Author | Fix | |-------|--------|-----| | #3505 | @franksong2702 | **Reasoning effort is coerced to a level the active model/provider actually supports** before each request, instead of being sent verbatim and rejected. `openai-codex` `gpt-5` no longer gets `max` (→ `xhigh`); `o1`/`o3`/`o4` clamp to `low`/`medium`/`high`. Coercion only steps *down* (never escalates); `none`/unset preserved. The capability filter is applied across heuristic / models.dev / Copilot / LM Studio paths. | This is the narrow, correct fix for the detection gap that #3431 tried to address by removing the chip-visibility gate (which we shelved). The chip-visibility gate is **untouched** (Codex confirmed) — `get_reasoning_status`/`_applyReasoningChip` still hide the chip for unconfirmed models. ### Review fix absorbed (Codex + self-flagged) The first cut **dropped** a configured effort for *unrecognized* models, because capability detection returns `[]` for both "known-unsupported" and "simply-unknown" (custom providers, aggregator-rewritten ids, new releases) — that's a behavior change vs master (which sent it verbatim) and would silently disable reasoning. Fixed: an **empty** capability set now **preserves** the configured effort (provider stays the final authority; worst case = the same rejected request master already produces, i.e. no regression). Known-bad clamps return *non-empty* filtered sets, so they still degrade correctly. Nathan chose this "preserve-for-unknown" behavior. + regression test. ### Gate - Full pytest suite: **7548 passed, 0 failed** - ruff: CLEAN · 48 reasoning tests pass (incl. preserve-for-unknown + codex-clamp + never-escalate) - Codex (regression): SHIP-ONLY-WITH-FIXES (unknown-model drop) → fixed → **SAFE TO SHIP** - Verified empirically: gpt-5/codex max→xhigh, o3 max/xhigh→high, unknown high→high (preserved), none/unset preserved Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
This commit is contained in:
@@ -3,6 +3,11 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [v0.51.247] — 2026-06-03 — Release HO (stage-q19 — coerce reasoning effort to model-supported levels)
|
||||
|
||||
### Fixed
|
||||
- A globally-configured reasoning effort (`agent.reasoning_effort`) is now **coerced to the closest level the active model/provider actually supports** before each request, instead of being sent verbatim and rejected. For example `openai-codex` `gpt-5` rejects `max` (now degraded to `xhigh`) and `o1`/`o3`/`o4` only accept `low`/`medium`/`high` (so `max`/`xhigh` degrade to `high`). Coercion only ever steps *down* to a supported level (never escalates), and `none`/unset are preserved. The model/provider effort-capability filter is applied consistently across the heuristic, models.dev metadata, GitHub Copilot, and LM Studio detection paths. (#3505, @franksong2702)
|
||||
|
||||
## [v0.51.246] — 2026-06-03 — Release HN (stage-q18 — WebUI rename syncs to agent state.db)
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -2234,6 +2234,28 @@ def _candidate_supports_reasoning(candidate: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _filter_reasoning_efforts_for_provider(
|
||||
efforts: list[str],
|
||||
model_id: str,
|
||||
provider_id: str,
|
||||
) -> list[str]:
|
||||
"""Apply provider/model quirks to otherwise valid reasoning effort levels."""
|
||||
normalized = [
|
||||
str(eff).strip().lower()
|
||||
for eff in efforts
|
||||
if str(eff).strip().lower() in VALID_REASONING_EFFORTS
|
||||
]
|
||||
normalized = list(dict.fromkeys(normalized))
|
||||
provider = _resolve_provider_alias(str(provider_id or "").strip().lower())
|
||||
bare = _strip_provider_hint_for_reasoning(model_id).lower().rsplit("/", 1)[-1]
|
||||
if provider == "openai-codex":
|
||||
if bare.startswith(("o1", "o3", "o4")):
|
||||
return [eff for eff in normalized if eff in {"low", "medium", "high"}]
|
||||
if bare.startswith("gpt-5"):
|
||||
return [eff for eff in normalized if eff != "max"]
|
||||
return normalized
|
||||
|
||||
|
||||
def _heuristic_reasoning_efforts(model_id: str, provider_id: str) -> list[str]:
|
||||
"""Fallback when hermes_cli is unavailable."""
|
||||
model = _strip_provider_hint_for_reasoning(model_id).lower()
|
||||
@@ -2244,7 +2266,9 @@ def _heuristic_reasoning_efforts(model_id: str, provider_id: str) -> list[str]:
|
||||
if provider == "openai-codex" and bare.startswith(("gpt-5", "o1", "o3", "o4")):
|
||||
if bare.startswith(("o1", "o3", "o4")):
|
||||
return ["low", "medium", "high"]
|
||||
return list(VALID_REASONING_EFFORTS)
|
||||
return _filter_reasoning_efforts_for_provider(
|
||||
list(VALID_REASONING_EFFORTS), model, provider
|
||||
)
|
||||
if provider in {"copilot", "github-copilot"}:
|
||||
if bare.startswith(("gpt-5", "o1", "o3", "o4")):
|
||||
if bare.startswith(("o1", "o3", "o4")):
|
||||
@@ -2298,7 +2322,9 @@ def _models_dev_reasoning_efforts(model_id: str, provider_id: str) -> list[str]
|
||||
|
||||
supports_reasoning = getattr(capabilities, "supports_reasoning", None)
|
||||
if supports_reasoning is True:
|
||||
return list(VALID_REASONING_EFFORTS)
|
||||
return _filter_reasoning_efforts_for_provider(
|
||||
list(VALID_REASONING_EFFORTS), model, provider
|
||||
)
|
||||
if supports_reasoning is False:
|
||||
return []
|
||||
return None
|
||||
@@ -2338,7 +2364,9 @@ def resolve_model_reasoning_efforts(
|
||||
return _heuristic_reasoning_efforts(hinted_model, provider)
|
||||
else:
|
||||
if provider in {"copilot", "github-copilot"}:
|
||||
return github_model_reasoning_efforts(hinted_model)
|
||||
return _filter_reasoning_efforts_for_provider(
|
||||
github_model_reasoning_efforts(hinted_model), hinted_model, provider
|
||||
)
|
||||
|
||||
if provider == "lmstudio":
|
||||
probe_base = resolved_base_url or _get_provider_base_url(provider)
|
||||
@@ -2348,11 +2376,16 @@ def resolve_model_reasoning_efforts(
|
||||
return []
|
||||
level_opts = [opt for opt in normalized if opt in VALID_REASONING_EFFORTS]
|
||||
if level_opts:
|
||||
return list(dict.fromkeys(level_opts))
|
||||
return _filter_reasoning_efforts_for_provider(
|
||||
level_opts, hinted_model, provider
|
||||
)
|
||||
if set(normalized).issubset({"off", "on"}):
|
||||
return []
|
||||
return []
|
||||
|
||||
# _models_dev_reasoning_efforts already applies the provider/model filter
|
||||
# internally, so it is returned as-is here (filtering again would be
|
||||
# redundant — the filter is idempotent but the double pass obscures flow).
|
||||
metadata_efforts = _models_dev_reasoning_efforts(hinted_model, provider)
|
||||
if metadata_efforts is not None:
|
||||
return metadata_efforts
|
||||
@@ -2360,6 +2393,57 @@ def resolve_model_reasoning_efforts(
|
||||
return _heuristic_reasoning_efforts(hinted_model, provider)
|
||||
|
||||
|
||||
def coerce_reasoning_effort_for_model(
|
||||
effort: str | None,
|
||||
model_id: str | None = None,
|
||||
provider_id: str | None = None,
|
||||
base_url: str | None = None,
|
||||
) -> str:
|
||||
"""Return the closest supported effort for the target model/provider."""
|
||||
raw = str(effort or "").strip().lower()
|
||||
if not raw:
|
||||
return ""
|
||||
if raw == "none":
|
||||
return "none"
|
||||
if raw not in VALID_REASONING_EFFORTS:
|
||||
return ""
|
||||
supported = resolve_model_reasoning_efforts(
|
||||
model_id,
|
||||
provider_id=provider_id,
|
||||
base_url=base_url,
|
||||
)
|
||||
# An empty list is ambiguous: resolve_model_reasoning_efforts() returns []
|
||||
# both for models KNOWN not to support reasoning AND for models we simply
|
||||
# don't recognize (custom providers, aggregator-rewritten ids, brand-new
|
||||
# releases). Coercion exists to avoid sending a level a KNOWN-incompatible
|
||||
# model rejects (e.g. openai-codex gpt-5 'max', o1/o3/o4 above 'high') —
|
||||
# those paths return a NON-empty clamped set, so the degrade ladder below
|
||||
# still applies. When the set is empty we can't tell "unsupported" from
|
||||
# "unknown", so preserve the user's configured effort verbatim (the prior
|
||||
# behavior) rather than silently disabling reasoning — the provider stays
|
||||
# the final authority. Worst case is the same rejected request that master
|
||||
# already produces, i.e. no regression. (#3505 review)
|
||||
if not supported:
|
||||
return raw
|
||||
if raw in supported:
|
||||
return raw
|
||||
# Degrade to the closest *lower* supported level instead of silently
|
||||
# disabling reasoning. e.g. max -> xhigh -> high, or xhigh -> high when the
|
||||
# target model caps below the configured effort. Never escalate.
|
||||
ladder = list(VALID_REASONING_EFFORTS) # ascending: minimal..max
|
||||
try:
|
||||
raw_idx = ladder.index(raw)
|
||||
except ValueError:
|
||||
return raw
|
||||
for level in reversed(ladder[:raw_idx]): # strictly lower, highest first
|
||||
if level in supported:
|
||||
return level
|
||||
# raw is below every supported level (shouldn't happen for a non-empty set
|
||||
# that excludes raw, but be safe): preserve the configured effort rather
|
||||
# than blank it.
|
||||
return raw
|
||||
|
||||
|
||||
def get_reasoning_status(
|
||||
*,
|
||||
model_id: str | None = None,
|
||||
|
||||
@@ -36,6 +36,8 @@ from api.config import (
|
||||
resolve_custom_provider_connection,
|
||||
model_with_provider_context,
|
||||
load_settings,
|
||||
parse_reasoning_effort,
|
||||
coerce_reasoning_effort_for_model,
|
||||
)
|
||||
from api.helpers import redact_session_data, _redact_text
|
||||
from api.compression_anchor import is_context_compression_marker, visible_messages_for_anchor
|
||||
@@ -5099,10 +5101,15 @@ def _run_agent_streaming(
|
||||
# `/reasoning <level>`) and hand the parsed dict to AIAgent. When
|
||||
# the key is absent or invalid, pass None → agent uses its default.
|
||||
try:
|
||||
from api.config import parse_reasoning_effort as _parse_reff
|
||||
_effort_cfg = _cfg.get('agent', {}) if isinstance(_cfg, dict) else {}
|
||||
_effort_raw = _effort_cfg.get('reasoning_effort') if isinstance(_effort_cfg, dict) else None
|
||||
_reasoning_config = _parse_reff(_effort_raw)
|
||||
_effort = coerce_reasoning_effort_for_model(
|
||||
_effort_raw,
|
||||
resolved_model,
|
||||
provider_id=resolved_provider,
|
||||
base_url=resolved_base_url,
|
||||
)
|
||||
_reasoning_config = parse_reasoning_effort(_effort)
|
||||
except Exception:
|
||||
_reasoning_config = None
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ def test_models_dev_false_suppresses_prefix_heuristic(monkeypatch):
|
||||
) == []
|
||||
|
||||
|
||||
def test_codex_gpt55_uses_models_dev_including_xhigh(monkeypatch):
|
||||
def test_codex_gpt55_uses_models_dev_excluding_unsupported_max(monkeypatch):
|
||||
_install_fake_models_dev(
|
||||
monkeypatch,
|
||||
lambda provider, model: SimpleNamespace(supports_reasoning=True),
|
||||
@@ -91,8 +91,8 @@ def test_codex_gpt55_uses_models_dev_including_xhigh(monkeypatch):
|
||||
result = cfg.resolve_model_reasoning_efforts(
|
||||
"gpt-5.5", provider_id="openai-codex"
|
||||
)
|
||||
assert result == list(cfg.VALID_REASONING_EFFORTS)
|
||||
assert "xhigh" in result
|
||||
assert "max" not in result
|
||||
|
||||
|
||||
def test_codex_metadata_false_returns_empty(monkeypatch):
|
||||
|
||||
@@ -17,6 +17,8 @@ def test_openai_codex_gpt5_supports_reasoning_effort_levels():
|
||||
)
|
||||
assert "medium" in efforts
|
||||
assert "high" in efforts
|
||||
assert "xhigh" in efforts
|
||||
assert "max" not in efforts
|
||||
|
||||
|
||||
def test_openai_codex_prefixed_gpt5_supports_reasoning_effort_levels():
|
||||
@@ -26,6 +28,69 @@ def test_openai_codex_prefixed_gpt5_supports_reasoning_effort_levels():
|
||||
)
|
||||
assert "medium" in efforts
|
||||
assert "high" in efforts
|
||||
assert "xhigh" in efforts
|
||||
assert "max" not in efforts
|
||||
|
||||
|
||||
def test_openai_codex_max_effort_is_clamped_before_streaming():
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"max",
|
||||
"gpt-5.5",
|
||||
provider_id="openai-codex",
|
||||
) == "xhigh"
|
||||
|
||||
|
||||
def test_unsupported_xhigh_degrades_to_high_not_disabled():
|
||||
# o1/o3/o4 on openai-codex cap at low/medium/high. A configured xhigh (or
|
||||
# max) must clamp DOWN to the highest supported level (high), not silently
|
||||
# disable reasoning by returning "".
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"xhigh",
|
||||
"o3-mini",
|
||||
provider_id="openai-codex",
|
||||
) == "high"
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"max",
|
||||
"o3-mini",
|
||||
provider_id="openai-codex",
|
||||
) == "high"
|
||||
|
||||
|
||||
def test_coerce_never_escalates_above_configured_effort():
|
||||
# A supported lower effort is returned verbatim; coercion only degrades.
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"low",
|
||||
"gpt-5.5",
|
||||
provider_id="openai-codex",
|
||||
) == "low"
|
||||
|
||||
|
||||
def test_coerce_preserves_effort_for_unrecognized_model():
|
||||
# #3505 review: resolve_model_reasoning_efforts() returns [] for BOTH
|
||||
# known-unsupported AND simply-unrecognized models (custom providers,
|
||||
# aggregator-rewritten ids, brand-new releases). Coercion must NOT silently
|
||||
# drop a configured effort just because we don't recognize the model — that
|
||||
# would be a behavior change vs sending it verbatim (master). Preserve the
|
||||
# configured level for an empty/unknown capability set; the provider stays
|
||||
# the final authority. The known-bad CLAMP paths return a NON-empty set, so
|
||||
# they are unaffected (covered by the openai-codex tests above).
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"high",
|
||||
"some-unknown-model-xyz",
|
||||
provider_id="some-custom-provider",
|
||||
) == "high"
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"max",
|
||||
"brand-new-model-2099",
|
||||
provider_id="some-custom-provider",
|
||||
) == "max"
|
||||
# 'none' / unset still pass through unchanged for unknown models.
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"none", "some-unknown-model-xyz", provider_id="custom"
|
||||
) == "none"
|
||||
assert cfg.coerce_reasoning_effort_for_model(
|
||||
"", "some-unknown-model-xyz", provider_id="custom"
|
||||
) == ""
|
||||
|
||||
|
||||
def test_github_copilot_gpt5_supports_reasoning_effort_levels():
|
||||
|
||||
@@ -361,6 +361,10 @@ class TestStreamingReasoningWiring:
|
||||
"api/streaming.py must import parse_reasoning_effort to translate "
|
||||
"config.yaml agent.reasoning_effort into AIAgent reasoning_config"
|
||||
)
|
||||
assert 'coerce_reasoning_effort_for_model' in src, (
|
||||
"api/streaming.py must clamp/drop unsupported model-specific effort "
|
||||
"levels before sending reasoning_config to the provider"
|
||||
)
|
||||
assert "reasoning_config" in src and "'reasoning_config' in _agent_params" in src, (
|
||||
"api/streaming.py must guard the reasoning_config kwarg with "
|
||||
"inspect.signature so older hermes-agent builds don't TypeError"
|
||||
|
||||
Reference in New Issue
Block a user