mtmd: support DeepSeek-V4-Flash-Vision-Exp - #28133
Conversation
github-actions
Bot
added
mtmd
Corrects a wrong claim that reached a public announcement (#1122, since withdrawn), the compose headers, both registry notes, BENCHMARKS.md and the learnings file. What was published: "the name says Vision, the weights do not" — that DeepSeek-V4-Flash-Vision-Exp has no vision capability. What is true: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp is a genuine VLM. Tagged image-text-to-text, config.json carries a full vision block (vision_dim 1024, vision_n_layers 32, vision_patch_size 14), and the safetensors index holds 263 vision tensors (vision.patch_embed.*, vision.blocks.0-31.*). It ships inference/vision.py and an image processor. The unsloth GGUF carries ZERO of them, so llama-server returns `image input is not supported` and the slug is text-only IN PRACTICE. Why no projector exists yet: llama.cpp support was opened upstream 2026-09-01 as ggml-org/llama.cpp#28133 (mtmd: support DeepSeek-V4-Flash-Vision-Exp, @ngxson, +498/-5, labels mtmd + conversion). Its notes confirm the text model already works out of the box -- matching what we see -- and what it adds is the CONVERSION that emits an mmproj. 23 HF repos checked for this model: none carries an mmproj, because until that merges the conversion cannot produce one. Tracked in docs/UPSTREAM.md as a re-test trigger; author discloses ~90% AI-generated code and testing on a slice of the weights, so merge is not assured. vision_capable: false remains correct for this GGUF -- only the recorded REASON was wrong. The methodology error, kept in learnings: the GGUF gate was sound for the question it asked ("can this GGUF serve images?") and its answer was right, later confirmed by the server's own runtime rejection. The error was over-generalising it to "this model has no vision" without reading the source model card. A second error compounded it -- the first withdrawal notice claimed vision was architecturally integral so no companion mmproj could exist, which #28133 disproves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…le too Missed in 4dd88b3 -- the profile's vision_capable comment still said the MODEL had no vision. The value stays false (correct for this GGUF); the reason now points at ggml-org/llama.cpp#28133 and the missing conversion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
I think this would break when we have image + text in the same ubatch because of how we handle the the kv-cache i.e. positions are conflated with cells |
|
ah right, but even with pure image batch, like two images it would not work? |
|
if 2 consecutive images (no text in-between), then they should look exactly like one single image? because input images are already padded, they are aligned to compress ratio, and input batch will contain only embeddings |
|
/bot review |
Automated code reviewStatic review of PR #28133 (mtmd: DeepSeek-V4-Flash-Vision-Exp) Overall this is a clean, conventional addition: no new ggml ops, tensor names go through Verify (correctness) (point 1) (point 2) The KV-cache/positions question raised in the PR comments (image embeddings + text in the same ubatch, positions conflated with cells) is still open and affects the core premise of this design; please settle it in the discussion before merge. Will slow the review (point 3) (point 4) The block-layout arithmetic ( (point 5) (point 6) (point 7) Nits (point 8) Typo in (point 9) (point 10) Testing and process notes All numeric validation was done against a small slice of the weights. The sentinel/layout bit-exactness is convincing, but please run at least one end-to-end sanity check with the full mmproj + text model on a real image before merge. The AI disclosure is filled in (~90% AI-generated). Per This review was generated automatically by pi coding agent using |
|
Ah I think M-rope wouldn't work, but this model does not seem to use that. So maybe it's ok |
So we don't need to resolve this for now? llama.cpp/src/llama-kv-cache-dsv4.cpp Line 1610 in 1b89a43 |
|
@ggerganov since they're not using M-RoPE I would consider this resolved for now, unless you see some other issue with this (apart from not being able to run unified kv-cache) |
|
It should probably be OK. Maybe we can add an assert in |
4 participants
Overview
Important
Text model need to be reconverted, need to apply this: #28154
Add support for https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp (text model is already supported out of the box)
Most of the things are trivial, with only one small catch: because the text model "compress" a block of 4 tokens, the image need to be aligned to this boundary (left-padded with a learned embedding token)
We assume the whole prompt is always fed into
mtmd_tokenizefor now, so that both the text and image can be counted and thus, the padding can also be counted correctly. For example: if we have 5 text tokens followed by an image, then 3 pad tokens must be added (so that image started at 5+3 = 8, aligned to 4)Testing
I only tested with a small slice of the real weight
Requirements