Fix issue 3355 assertion error

[marksverdhei]

Summary

This PR fixes issue #3355, which caused an AssertionError when running evaluations with OpenAI-compatible endpoints when API responses have "content": null. This happens with vllm if using a reasoning model (GPT-OSS-20b) and running out of tokens before the non-think response is generated. It is solved by treating the response as an empty string while logging a warning.

Reproducing the error

# start a local vllm instance of gpt-oss-20b
vllm serve openai/gpt-oss-20b --port 7000

# Example run, not a real eval. Breaks before PR, works after
lm_eval --model local-chat-completions --tasks nortruthfulqa_gen_nob_p0 --model_args base_url=http://localhost:7000/v1/chat/completions,num_concurrent=1,max_retries=3,tokenized_requests=False,model="openai/gpt-oss-20b" --apply_chat_template --limit 0.001

Details

The issue occurred in lm_eval/models/utils.py:541 in the Collator.get_original() method:

assert all(cov)  # AssertionError here

The problem: When parse_generations() returned None values (from API responses with "content": null), the original code would skip these values:

# OLD CODE - BUGGY
if generated_text is not None:
    res.append(generated_text)  # Only appends non-None values

This caused a mismatch:

• The Collator tracked that it had N requests to process (_size = N)
• But the result list res only had M items (where M < N, skipping the None values)
• When get_original() tried to restore the original order, not all positions were filled
• The assertion assert all(cov) failed because some positions remained uncovered

Based on real API logs from the issue, this happens when vllm OpenAI-compatible endpoint return:

{
  "choices": [{
    "index": 0,
    "message": {
      "content": null,                    // ← NULL content!
      "reasoning_content": "The user..."  // ← Actual text here
    }
  }]
}

Note that caching still only happens for successful results (None values not cached)

Resolves #3355

*this PR was created with the assistance of AI tools

[marksverdhei]

The PR has been out for two weeks now without any response, I hope this can be resolved soon 🙏

[marksverdhei]

@baberabb please let me know if there's anything I can do to make the process / reviewing easier or faster.

[baberabb]

@baberabb please let me know if there's anything I can do to make the process / reviewing easier or faster.

Sorry for the delay. LGTM!