Qwen3-VL and Gemma3 (VLM) ignore `shift_labels`

[qgallouedec]

System Info

transformers main (9810159), torch 2.13.0, python 3.13

Who can help?

@zucchini-nlp

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import torch
from transformers import AutoModelForImageTextToText

for name in ["trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration", "trl-internal-testing/tiny-Qwen3VLForConditionalGeneration", "trl-internal-testing/tiny-Gemma3ForConditionalGeneration"]:
    model = AutoModelForImageTextToText.from_pretrained(name)
    input_ids = torch.randint(10, 100, (1, 16))
    ignore_all = torch.full_like(input_ids, -100)  # every target masked
    print(name, model(input_ids=input_ids, labels=input_ids, shift_labels=ignore_all).loss.item())

Outcome

tiny-Qwen2_5_VLForConditionalGeneration nan
tiny-Qwen3VLForConditionalGeneration 11.953
tiny-Gemma3ForConditionalGeneration 12.539

Expected behavior

All nan.

Qwen3VLForConditionalGeneration.forward calls self.loss_function(logits=logits, labels=labels, vocab_size=...) without **kwargs (modeling_qwen3_vl.py:1376), so shift_labels is silently ignored and the loss shifts labels locally. Qwen2.5-VL passes **kwargs. This breaks sequence/context parallel training (accelerate CP, Ulysses SP, DeepSpeed ALST), where the pre-shifted labels are the only correct ones: each shard's last token silently gets no target. Same call in modular_qwen3_vl.py:921 and modeling_qwen3_vl_moe.py:1560. Gemma3ForConditionalGeneration is the other case: it shifts labels by hand with accepts_loss_kwargs = False instead of going through self.loss_function, so it cannot take shift_labels at all.