Release v0.51.247 — Release HO (stage-q19) (#3521)
Some checks failed
Release & Docker / release (push) Has been cancelled

## Release v0.51.247 — Release HO (stage-q19)

Backend correctness fix.

### Fixed
| Issue | Author | Fix |
|-------|--------|-----|
| #3505 | @franksong2702 | **Reasoning effort is coerced to a level the active model/provider actually supports** before each request, instead of being sent verbatim and rejected. `openai-codex` `gpt-5` no longer gets `max` (→ `xhigh`); `o1`/`o3`/`o4` clamp to `low`/`medium`/`high`. Coercion only steps *down* (never escalates); `none`/unset preserved. The capability filter is applied across heuristic / models.dev / Copilot / LM Studio paths. |

This is the narrow, correct fix for the detection gap that #3431 tried to address by removing the chip-visibility gate (which we shelved). The chip-visibility gate is **untouched** (Codex confirmed) — `get_reasoning_status`/`_applyReasoningChip` still hide the chip for unconfirmed models.

### Review fix absorbed (Codex + self-flagged)
The first cut **dropped** a configured effort for *unrecognized* models, because capability detection returns `[]` for both "known-unsupported" and "simply-unknown" (custom providers, aggregator-rewritten ids, new releases) — that's a behavior change vs master (which sent it verbatim) and would silently disable reasoning. Fixed: an **empty** capability set now **preserves** the configured effort (provider stays the final authority; worst case = the same rejected request master already produces, i.e. no regression). Known-bad clamps return *non-empty* filtered sets, so they still degrade correctly. Nathan chose this "preserve-for-unknown" behavior. + regression test.

### Gate
- Full pytest suite: **7548 passed, 0 failed**
- ruff: CLEAN · 48 reasoning tests pass (incl. preserve-for-unknown + codex-clamp + never-escalate)
- Codex (regression): SHIP-ONLY-WITH-FIXES (unknown-model drop) → fixed → **SAFE TO SHIP**
- Verified empirically: gpt-5/codex max→xhigh, o3 max/xhigh→high, unknown high→high (preserved), none/unset preserved

Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
This commit is contained in:
nesquena-hermes
2026-06-03 19:21:26 -07:00
committed by GitHub
parent 772a5c17ed
commit 81e748b455
6 changed files with 173 additions and 8 deletions

View File

@@ -3,6 +3,11 @@
## [Unreleased]
## [v0.51.247] — 2026-06-03 — Release HO (stage-q19 — coerce reasoning effort to model-supported levels)
### Fixed
- A globally-configured reasoning effort (`agent.reasoning_effort`) is now **coerced to the closest level the active model/provider actually supports** before each request, instead of being sent verbatim and rejected. For example `openai-codex` `gpt-5` rejects `max` (now degraded to `xhigh`) and `o1`/`o3`/`o4` only accept `low`/`medium`/`high` (so `max`/`xhigh` degrade to `high`). Coercion only ever steps *down* to a supported level (never escalates), and `none`/unset are preserved. The model/provider effort-capability filter is applied consistently across the heuristic, models.dev metadata, GitHub Copilot, and LM Studio detection paths. (#3505, @franksong2702)
## [v0.51.246] — 2026-06-03 — Release HN (stage-q18 — WebUI rename syncs to agent state.db)
### Fixed

View File

@@ -2234,6 +2234,28 @@ def _candidate_supports_reasoning(candidate: str) -> bool:
return False
def _filter_reasoning_efforts_for_provider(
efforts: list[str],
model_id: str,
provider_id: str,
) -> list[str]:
"""Apply provider/model quirks to otherwise valid reasoning effort levels."""
normalized = [
str(eff).strip().lower()
for eff in efforts
if str(eff).strip().lower() in VALID_REASONING_EFFORTS
]
normalized = list(dict.fromkeys(normalized))
provider = _resolve_provider_alias(str(provider_id or "").strip().lower())
bare = _strip_provider_hint_for_reasoning(model_id).lower().rsplit("/", 1)[-1]
if provider == "openai-codex":
if bare.startswith(("o1", "o3", "o4")):
return [eff for eff in normalized if eff in {"low", "medium", "high"}]
if bare.startswith("gpt-5"):
return [eff for eff in normalized if eff != "max"]
return normalized
def _heuristic_reasoning_efforts(model_id: str, provider_id: str) -> list[str]:
"""Fallback when hermes_cli is unavailable."""
model = _strip_provider_hint_for_reasoning(model_id).lower()
@@ -2244,7 +2266,9 @@ def _heuristic_reasoning_efforts(model_id: str, provider_id: str) -> list[str]:
if provider == "openai-codex" and bare.startswith(("gpt-5", "o1", "o3", "o4")):
if bare.startswith(("o1", "o3", "o4")):
return ["low", "medium", "high"]
return list(VALID_REASONING_EFFORTS)
return _filter_reasoning_efforts_for_provider(
list(VALID_REASONING_EFFORTS), model, provider
)
if provider in {"copilot", "github-copilot"}:
if bare.startswith(("gpt-5", "o1", "o3", "o4")):
if bare.startswith(("o1", "o3", "o4")):
@@ -2298,7 +2322,9 @@ def _models_dev_reasoning_efforts(model_id: str, provider_id: str) -> list[str]
supports_reasoning = getattr(capabilities, "supports_reasoning", None)
if supports_reasoning is True:
return list(VALID_REASONING_EFFORTS)
return _filter_reasoning_efforts_for_provider(
list(VALID_REASONING_EFFORTS), model, provider
)
if supports_reasoning is False:
return []
return None
@@ -2338,7 +2364,9 @@ def resolve_model_reasoning_efforts(
return _heuristic_reasoning_efforts(hinted_model, provider)
else:
if provider in {"copilot", "github-copilot"}:
return github_model_reasoning_efforts(hinted_model)
return _filter_reasoning_efforts_for_provider(
github_model_reasoning_efforts(hinted_model), hinted_model, provider
)
if provider == "lmstudio":
probe_base = resolved_base_url or _get_provider_base_url(provider)
@@ -2348,11 +2376,16 @@ def resolve_model_reasoning_efforts(
return []
level_opts = [opt for opt in normalized if opt in VALID_REASONING_EFFORTS]
if level_opts:
return list(dict.fromkeys(level_opts))
return _filter_reasoning_efforts_for_provider(
level_opts, hinted_model, provider
)
if set(normalized).issubset({"off", "on"}):
return []
return []
# _models_dev_reasoning_efforts already applies the provider/model filter
# internally, so it is returned as-is here (filtering again would be
# redundant — the filter is idempotent but the double pass obscures flow).
metadata_efforts = _models_dev_reasoning_efforts(hinted_model, provider)
if metadata_efforts is not None:
return metadata_efforts
@@ -2360,6 +2393,57 @@ def resolve_model_reasoning_efforts(
return _heuristic_reasoning_efforts(hinted_model, provider)
def coerce_reasoning_effort_for_model(
effort: str | None,
model_id: str | None = None,
provider_id: str | None = None,
base_url: str | None = None,
) -> str:
"""Return the closest supported effort for the target model/provider."""
raw = str(effort or "").strip().lower()
if not raw:
return ""
if raw == "none":
return "none"
if raw not in VALID_REASONING_EFFORTS:
return ""
supported = resolve_model_reasoning_efforts(
model_id,
provider_id=provider_id,
base_url=base_url,
)
# An empty list is ambiguous: resolve_model_reasoning_efforts() returns []
# both for models KNOWN not to support reasoning AND for models we simply
# don't recognize (custom providers, aggregator-rewritten ids, brand-new
# releases). Coercion exists to avoid sending a level a KNOWN-incompatible
# model rejects (e.g. openai-codex gpt-5 'max', o1/o3/o4 above 'high') —
# those paths return a NON-empty clamped set, so the degrade ladder below
# still applies. When the set is empty we can't tell "unsupported" from
# "unknown", so preserve the user's configured effort verbatim (the prior
# behavior) rather than silently disabling reasoning — the provider stays
# the final authority. Worst case is the same rejected request that master
# already produces, i.e. no regression. (#3505 review)
if not supported:
return raw
if raw in supported:
return raw
# Degrade to the closest *lower* supported level instead of silently
# disabling reasoning. e.g. max -> xhigh -> high, or xhigh -> high when the
# target model caps below the configured effort. Never escalate.
ladder = list(VALID_REASONING_EFFORTS) # ascending: minimal..max
try:
raw_idx = ladder.index(raw)
except ValueError:
return raw
for level in reversed(ladder[:raw_idx]): # strictly lower, highest first
if level in supported:
return level
# raw is below every supported level (shouldn't happen for a non-empty set
# that excludes raw, but be safe): preserve the configured effort rather
# than blank it.
return raw
def get_reasoning_status(
*,
model_id: str | None = None,

View File

@@ -36,6 +36,8 @@ from api.config import (
resolve_custom_provider_connection,
model_with_provider_context,
load_settings,
parse_reasoning_effort,
coerce_reasoning_effort_for_model,
)
from api.helpers import redact_session_data, _redact_text
from api.compression_anchor import is_context_compression_marker, visible_messages_for_anchor
@@ -5099,10 +5101,15 @@ def _run_agent_streaming(
# `/reasoning <level>`) and hand the parsed dict to AIAgent. When
# the key is absent or invalid, pass None → agent uses its default.
try:
from api.config import parse_reasoning_effort as _parse_reff
_effort_cfg = _cfg.get('agent', {}) if isinstance(_cfg, dict) else {}
_effort_raw = _effort_cfg.get('reasoning_effort') if isinstance(_effort_cfg, dict) else None
_reasoning_config = _parse_reff(_effort_raw)
_effort = coerce_reasoning_effort_for_model(
_effort_raw,
resolved_model,
provider_id=resolved_provider,
base_url=resolved_base_url,
)
_reasoning_config = parse_reasoning_effort(_effort)
except Exception:
_reasoning_config = None

View File

@@ -80,7 +80,7 @@ def test_models_dev_false_suppresses_prefix_heuristic(monkeypatch):
) == []
def test_codex_gpt55_uses_models_dev_including_xhigh(monkeypatch):
def test_codex_gpt55_uses_models_dev_excluding_unsupported_max(monkeypatch):
_install_fake_models_dev(
monkeypatch,
lambda provider, model: SimpleNamespace(supports_reasoning=True),
@@ -91,8 +91,8 @@ def test_codex_gpt55_uses_models_dev_including_xhigh(monkeypatch):
result = cfg.resolve_model_reasoning_efforts(
"gpt-5.5", provider_id="openai-codex"
)
assert result == list(cfg.VALID_REASONING_EFFORTS)
assert "xhigh" in result
assert "max" not in result
def test_codex_metadata_false_returns_empty(monkeypatch):

View File

@@ -17,6 +17,8 @@ def test_openai_codex_gpt5_supports_reasoning_effort_levels():
)
assert "medium" in efforts
assert "high" in efforts
assert "xhigh" in efforts
assert "max" not in efforts
def test_openai_codex_prefixed_gpt5_supports_reasoning_effort_levels():
@@ -26,6 +28,69 @@ def test_openai_codex_prefixed_gpt5_supports_reasoning_effort_levels():
)
assert "medium" in efforts
assert "high" in efforts
assert "xhigh" in efforts
assert "max" not in efforts
def test_openai_codex_max_effort_is_clamped_before_streaming():
assert cfg.coerce_reasoning_effort_for_model(
"max",
"gpt-5.5",
provider_id="openai-codex",
) == "xhigh"
def test_unsupported_xhigh_degrades_to_high_not_disabled():
# o1/o3/o4 on openai-codex cap at low/medium/high. A configured xhigh (or
# max) must clamp DOWN to the highest supported level (high), not silently
# disable reasoning by returning "".
assert cfg.coerce_reasoning_effort_for_model(
"xhigh",
"o3-mini",
provider_id="openai-codex",
) == "high"
assert cfg.coerce_reasoning_effort_for_model(
"max",
"o3-mini",
provider_id="openai-codex",
) == "high"
def test_coerce_never_escalates_above_configured_effort():
# A supported lower effort is returned verbatim; coercion only degrades.
assert cfg.coerce_reasoning_effort_for_model(
"low",
"gpt-5.5",
provider_id="openai-codex",
) == "low"
def test_coerce_preserves_effort_for_unrecognized_model():
# #3505 review: resolve_model_reasoning_efforts() returns [] for BOTH
# known-unsupported AND simply-unrecognized models (custom providers,
# aggregator-rewritten ids, brand-new releases). Coercion must NOT silently
# drop a configured effort just because we don't recognize the model — that
# would be a behavior change vs sending it verbatim (master). Preserve the
# configured level for an empty/unknown capability set; the provider stays
# the final authority. The known-bad CLAMP paths return a NON-empty set, so
# they are unaffected (covered by the openai-codex tests above).
assert cfg.coerce_reasoning_effort_for_model(
"high",
"some-unknown-model-xyz",
provider_id="some-custom-provider",
) == "high"
assert cfg.coerce_reasoning_effort_for_model(
"max",
"brand-new-model-2099",
provider_id="some-custom-provider",
) == "max"
# 'none' / unset still pass through unchanged for unknown models.
assert cfg.coerce_reasoning_effort_for_model(
"none", "some-unknown-model-xyz", provider_id="custom"
) == "none"
assert cfg.coerce_reasoning_effort_for_model(
"", "some-unknown-model-xyz", provider_id="custom"
) == ""
def test_github_copilot_gpt5_supports_reasoning_effort_levels():

View File

@@ -361,6 +361,10 @@ class TestStreamingReasoningWiring:
"api/streaming.py must import parse_reasoning_effort to translate "
"config.yaml agent.reasoning_effort into AIAgent reasoning_config"
)
assert 'coerce_reasoning_effort_for_model' in src, (
"api/streaming.py must clamp/drop unsupported model-specific effort "
"levels before sending reasoning_config to the provider"
)
assert "reasoning_config" in src and "'reasoning_config' in _agent_params" in src, (
"api/streaming.py must guard the reasoning_config kwarg with "
"inspect.signature so older hermes-agent builds don't TypeError"