Spent a couple hours debugging a polyglot bench setup against a local vllm endpoint before figuring out that benchmark.py doesn't accept the --model-settings-file flag that aider's main cli (and the adv-model-settings docs) use. The bench flag is --read-model-settings, defined in benchmark/benchmark.py L206.
The bench's own README (line 85) names --read-model-settings correctly but links straight to the adv-model-settings docs which use --model-settings-file throughout. Users following the link reasonably assume the flag is the same and silently load no settings. Qwen-family AWQ in particular hangs in long-decode loops without the min_p / top_p knobs from the YAML, so the failure mode looks like "bench sits there for an hour" rather than a clear error.
Possibly related to #2766 (open since Jan 2025) where the same wall got hit and never resolved.
Three options by effort:
- one-line callout in adv-model-settings docs + bench README naming the flag rename
- alias
--model-settings-file in benchmark.py so both flags work
- rename
--read-model-settings to --model-settings-file in benchmark.py with a deprecation alias
fyi benchmark.py also sets LONG_TIMEOUT = 24 * 60 * 60 which is why this looks like a silent hang and not a fast failure. Related but separate footgun.
Not blocking. Can send a PR for #1 if useful.
Spent a couple hours debugging a polyglot bench setup against a local vllm endpoint before figuring out that
benchmark.pydoesn't accept the--model-settings-fileflag that aider's main cli (and the adv-model-settings docs) use. The bench flag is--read-model-settings, defined inbenchmark/benchmark.pyL206.The bench's own README (line 85) names
--read-model-settingscorrectly but links straight to the adv-model-settings docs which use--model-settings-filethroughout. Users following the link reasonably assume the flag is the same and silently load no settings. Qwen-family AWQ in particular hangs in long-decode loops without themin_p/top_pknobs from the YAML, so the failure mode looks like "bench sits there for an hour" rather than a clear error.Possibly related to #2766 (open since Jan 2025) where the same wall got hit and never resolved.
Three options by effort:
--model-settings-filein benchmark.py so both flags work--read-model-settingsto--model-settings-filein benchmark.py with a deprecation aliasfyi
benchmark.pyalso setsLONG_TIMEOUT = 24 * 60 * 60which is why this looks like a silent hang and not a fast failure. Related but separate footgun.Not blocking. Can send a PR for #1 if useful.