webui : send both backend_sampling == false/true by ggerganov · Pull Request #18781 · ggml-org/llama.cpp

ggerganov · 2026-01-12T13:22:00Z

I believe the webui does not send backend_sampling == false in the request when the checkbox is unchecked. The goal is when the server is started with -bs to be able to disable backend sampling from the client.

Also I think the checkbox is currently not synchronized with the server argument like the rest of the sampling parameters. But not sure how to fix this of if it is supported for checkboxes.

allozaur · 2026-01-12T13:24:28Z

I will take a closer look at this

allozaur

@ggerganov I've added the parameter sync for the setting

@reeselevine

* model, mtmd: fix gguf conversion for audio/vision mmproj (ggml-org#21309) * fix gguf conversion for audio/vision mmproj * fix test * tests: allow exporting graph ops from HF file without downloading weights (ggml-org#21182) * tests: allow exporting graph ops from HF file without downloading weights * use unique_ptr for llama_context in HF metadata case * fix missing non-required tensors falling back to type f32 * use unique pointers where possible * use no_alloc instead of fixing f32 fallback * fix missing space * ggml-webgpu: add vectorized flash attention (ggml-org#20709) * naive vectorized version * add vectorized flash attention * update vec version * remove unused path and shader * remove unused helper functions * add comments * remove pad path * ggml-webgpu: fix flash-attn vec nwg=1 path and tighten vec specialization * change back to vec4 * enable multi split * enable vec path when: - Q->ne[1] < 20 - Q->ne[0] % 32 == 0 - V->ne[0] % 4 == 0 - K->type == f16 * update flast_attn_vec_split.wgsl to reduce redundant workgroup barrier usage and use select * enable vec path for q4 and q8 * flash-attn vec nwg=1 fast path (skip tmp/reduce staging) * use packed f16 K loads in flash-attn vec split * use packed f16 K loads in flash-attn vec split on host side * tune flash-attn vec f16 VEC_NE by head dim * cleanup * cleanup * keep host side clean * cleanup host side * change back to original host wait/submit behavior * formatting * reverted param-buffer pool r ecfactor * add helper functions * ggml-webgpu: move flash-attn vec pipeline caching back into shader lib * ggml-webgpu: remove duplicate functions * ggml-webgpu: reserve flash-attn vec scratch in dst buffer allocation * ggml-webgpu: revert unrelated change * ggml-webgpu: revert deleted comment * disable uniformity check * remove unnecessary change * Update ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_vec_split.wgsl * Update ggml/src/ggml-webgpu/ggml-webgpu.cpp --------- Co-authored-by: Reese Levine <reeselevine1@gmail.com> * tests : add unit test coverage for llama_tensor_get_type (ggml-org#20112) * Add unit test coverage for llama_tensor_get_type * Fix merge conflicts, add more schemas * clang formatter changes * Trailing whitespace * Update name * Start rebase * Updating files with upstream changes prior to rebase * Changes needed from rebase * Update attn_qkv schema, change throw behaviour * Fix merge conflicts * White space * Update with latest changes to state counters * Revert accidental personal CLAUDE.md changes * Change quotation mark * Reuse metadata.name since we have it * Move test-only stuff out of llama-quant.cpp * Hide the regex functionality back in llama-quant.cpp, use a unique pointer to a new struct 'compiled_tensor_type_patterns' which contains the patterns * cont : inital deslop guidelines * Cleanup based on review comments * Continue cleanup * Small cleanup * Manually set proper ordering of tensors, mostly applies to gemma * Formatting * Update tests/test-quant-type-selection.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Fix merge conflicts --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix: gemma 4 template (ggml-org#21326) * [HIP] Bump ROCm version to 7.2.1 (ggml-org#21066) Bump ROCm version on Linux from 7.2 to 7.2.1 Add gfx1102 target Delete LLVM workaround since ROCm 7.2.1 has fix for ROCm 7.2 perf regression ROCm/rocm-systems#2865 --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * ci : add AMD ZenDNN label to PR labeler (ggml-org#21345) * ci : add AMD CPU label to PR labeler Add automatic labeling for PRs that modify AMD CPU (ZenDNN) backend files * ci : rename label AMD CPU to AMD ZenDNN in labeler config Co-authored-by: Aaron Teo <taronaeo@gmail.com> --------- Co-authored-by: Aaron Teo <taronaeo@gmail.com> * (revert) kv-cache : do not quantize SWA KV cache (ggml-org#21332) This reverts commit 17193cc. * chat : avoid including json in chat.h (ggml-org#21306) * rpc : reuse compute graph buffers (ggml-org#21299) Reuse the buffer for the ggml context which is used for creating the compute graph on the server side. This partially addresses a memory leak created by the CUDA backend due to using buffer addresses as cache keys. ref: ggml-org#21265 ref: ggml-org#20315 * vocab: fix Gemma4 tokenizer (ggml-org#21343) * seems to work * fix case with new line Co-authored-by: sayap <sokann@gmail.com> * gemma 4: fix pre tok regex --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: sayap <sokann@gmail.com> * ggml-zendnn : add MUL_MAT_ID op support for MoE models (ggml-org#21315) * ggml-zendnn : add MUL_MAT_ID op support for MoE models - Add MUL_MAT_ID op acceleration for Mixture-of-Experts models - MUL_MAT_ID op fallback to CPU backend if total experts > 32 - Point ZenDNN lib to latest bits ZenDNN-2026-WW13 * ggml-zendnn : add braces to sgemm failure condition for consistency Co-authored-by: Aaron Teo <taronaeo@gmail.com> --------- Co-authored-by: Aaron Teo <taronaeo@gmail.com> * fix: add openssl to nix dependencies (ggml-org#21353) (ggml-org#21355) * HIP: build eatch ci build test for a different architecture (ggml-org#21337) This helps improve our chances of finding build failures before the release workflow builds for all architectures. * fix: remove stale assert (ggml-org#21369) * ci: add more binary checks (ggml-org#21349) * jinja: coerce input for string-specific filters (ggml-org#21370) * docs: Update build.md: HSA_OVERRIDE_GFX_VERSION clarification (ggml-org#21331) The `HSA_OVERRIDE_GFX_VERSION` variable can be used in ROCm to override an unsupported target architecture with a similar but supported target architecture. This does not and has never worked on Windows. I think the clarification could avoid driving Windows people towards this solution that does not work. * docker : bump cuda12 to 12.9.1 (ggml-org#20920) Co-authored-by: M1DNYT3 <m1dnyt3@MacBookPro.lan> Co-authored-by: CISC <CISC@users.noreply.github.com> * common : fix tool call type detection for nullable and enum schemas (ggml-org#21327) * common : fix tool call type detection for nullable and enum schemas * common, tests : fix grammar delegation for nullable/enum schemas and add tests Fix enum type inference to scan all enum values (not just index 0) so schemas like {"enum": [0, "celsius"]} correctly detect string type. Fix schema_delegates in peg-parser to handle nullable type arrays (["string", "null"]) and typeless enum schemas in raw mode, allowing the tagged parser to use raw text instead of JSON-formatted strings. Add test cases for Qwen3-Coder (TAG_WITH_TAGGED format): - nullable string ["string", "null"] - nullable string with null first ["null", "string"] - nullable integer ["integer", "null"] - enum without explicit type key * common/parser: fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers (ggml-org#21230) * Fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers * Rename * Update common/chat-auto-parser-generator.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * server: save and clear idle slots on new task (`--clear-idle`) (ggml-org#20993) * server: clear idle slots KV from VRAM (LLAMA_KV_KEEP_ONLY_ACTIVE) * server: move idle slot KV clearing to slot release The save "cost" is now paid by the finishing request. * server: add --kv-clear-idle flag, enable by default * server: skip clearing last idle slot, clear on launch * server: test --no-kv-clear-idle flag * server: simplify on-release clearing loop * server: remove on-release KV clearing, keep launch-only * cont : clean-up * tests: update log strings after --clear-idle rename * tests: use debug tags instead of log message matching * test: fix Windows CI by dropping temp log file unlink --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ci: Add Windows Vulkan backend testing on Intel (ggml-org#21292) * experimenting CI * Experimenting CI fix for MinGW * experimenting CI on Windows * modified script for integration with VisualStudio * added proxy handling * adding python version for Windows execution * fix iterator::end() dereference * fixed proxy handling * Fix errors occurring on Windows * fixed ci script * Reverted to master * Stripping test items to simplify Windows test * adjusting script for windows testing * Changed shell * Fixed shell * Fixed shell * Fix CI setting * Fix CI setting * Fix CI setting * Experimenting ci fix * Experimenting ci fix * Experimenting ci fix * Experimenting ci fix * experimenting fix for unit test error * Changed to use BUILD_LOW_PERF to skip python tests * Fix CI * Added option to specify Ninja generator * Reverted proxy related changes * ggml-webgpu: move from parameter buffer pool to single buffer with offsets (ggml-org#21278) * Work towards removing bitcast * Move rest of existing types over * Add timeout back to wait and remove synchronous set_tensor/memset_tensor * move to unpackf16 for wider compatibility * cleanup * Remove deadlock condition in free_bufs * Start work on removing parameter buffer pools * Simplify and optimize further * simplify profile futures * Fix stride * Try using a single command buffer per batch * formatting * llama: add custom newline split for Gemma 4 (ggml-org#21406) * llama-model: read final_logit_softcapping for Gemma 4 (ggml-org#21390) * common : respect specified tag, only fallback when tag is empty (ggml-org#21413) Signed-off-by: Adrien Gallouët <angt@huggingface.co> * server: Fix undefined timing measurement errors in server context (ggml-org#21201) Co-authored-by: Dan Hoffman <dhoffman@cyket.net> * common : add gemma 4 specialized parser (ggml-org#21418) * common : add gemma4 dedicated parser * cont : add '<|tool_response>' as eog * cont : emit JSON from Gemma4 tool call AST * cont : more fixes * cont : refactor convert function * cont : refine rules and mapping * cont : add more tests * cont : clean up * cont : remove autoparser gemma4 implementation * cont : more cleanup * cont : rename gemma4.jinja to match the others * cont : add custom template to support interleaved thinking * cont : preserve reasoning in model turns * cont : fix initializer error * cont : fix unused vars * cont : fix accidental static * cont : fix specialized_template signature * fix extra semicolon * remove debug line and extra space [no ci] * ci: fix vulkan workflow referencing non-existent action (ggml-org#21442) * ci: lower cuda12 floor to 12.8.1 for broader host compatibility (ggml-org#21438) Co-authored-by: M1DNYT3 <m1dnyt3@MacBookPro.lan> * server : fix logging of build + system info (ggml-org#21460) This PR changes the logging that occurs at startup of llama-server. Currently, it is redundant (including CPU information twice) and it is missing the build + commit info. * ci : use default RISE RISC-V Runners (ggml-org#21263) * model : add HunyuanOCR support (ggml-org#21395) * HunyuanOCR: add support for text and vision models - Add HunyuanOCR vision projector (perceiver-based) with Conv2d merge - Add separate HUNYUAN_OCR chat template (content-before-role format) - Handle HunyuanOCR's invalid pad_token_id=-1 in converter - Fix EOS/EOT token IDs from generation_config.json - Support xdrope RoPE scaling type - Add tensor mappings for perceiver projector (mm.before_rms, mm.after_rms, etc.) - Register HunYuanVLForConditionalGeneration for both text and mmproj conversion * fix proper mapping * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> * Update tools/mtmd/clip.cpp Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> * address comments * update * Fix typecheck * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * llama : correct platform-independent loading of BOOL metadata (ggml-org#21428) * model-loader : fix GGUF bool array conversion * model-loader : fix remaining GGUF bool pointer uses * hexagon: slight optimization for argosrt output init (ggml-org#21463) * sycl : handle other FA case (ggml-org#21377) * convert : set "add bos" == True for Gemma 4 (ggml-org#21500) * convert : set "add bos" == True for Gemma 4 * cont : handle old GGUFs * docs: add hunyuan-ocr gguf, also add test [no ci] (ggml-org#21490) * server : handle unsuccessful sink.write in chunked stream provider (ggml-org#21478) Check the return value of sink.write() in the chunked content provider and return false when the write fails, matching cpp-httplib's own streaming contract. This prevents logging chunks as sent when the sink rejected them and properly aborts the stream on connection failure. * convert : fix block_ff_dim retrieval for lfm2 (ggml-org#21508) * vocab : add byte token handling to BPE detokenizer for Gemma4 (ggml-org#21488) * llama-bench: add `-fitc` and `-fitt` to arguments (ggml-org#21304) * llama-bench: add `-fitc` and `-fitt` to arguments * update README.md * address review comments * update compare-llama-bench.py * [CUDA ] Write an optimized flash_attn_stream_k_fixup kernel (ggml-org#21159) * Write an optimized flash_attn_stream_k_fixup kernel Write a specialized and more optimized kernel for cases where nblocks_stream_k is multiple of ntiles_dst. Make nblocks_stream_k to multiple of ntiles_dst if nblocks_stream_k > 2 * ntiles_dst * Use the new kernel only for nblocks_stream_k_raw > 4 * ntiles_dst to make sure we have enough concurrency on GPUs * Address review comments * Address review comments * Revert variable names to original * cli: fix stripping of \n in multiline input (ggml-org#21485) * llama-cli: fix stripping of \n in multiline input * Change & string to string_view * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Fix EditorConfig linter error --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * ggml: add Q1_0 1-bit quantization support (CPU) (ggml-org#21273) * ggml: add Q1_0 and Q1_0_g128 1-bit quantization support (CPU) * add generic fallback for x86 * remove Q1_0 (group size 32) * rename Q1_0_g128 => Q1_0 * fix Q1_0 LlamaFileType Enum * Fix trailing spaces; add generic fallback for othre backends * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix /r/n spacing + arch-fallback --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * ggml-webgpu: Add the support of `MUL_MAT_ID` (ggml-org#21147) * Add mul_mat_id support to WebGPU * Apply suggestion from @reeselevine --------- Co-authored-by: Reese Levine <reeselevine1@gmail.com> * docs: fix typo in build.md (emdawbwebgpu -> emdawnwebgpu) (ggml-org#21518) * [SYCL] Add Q8_0 reorder optimization (~3x tg speedup on Intel Arc) (ggml-org#21527) Extend the existing reorder optimization to Q8_0. The reorder separates scale factors from weight data for coalesced memory access -- was implemented for Q4_0/Q4_K/Q6_K but Q8_0 was missing. On Arc Pro B70 (Xe2), Q8_0 tg goes from 4.88 to 15.24 t/s (3.1x) on Qwen3.5-27B. BW utilization: 21% -> 66%. The key fix beyond the kernels: Q8_0 was missing from the type check in ggml_backend_sycl_buffer_init_tensor() that allocates the extra struct carrying the reorder flag -- so the optimization was silently skipped. AI (Claude) was used to assist with root cause investigation and writing the kernel code. All code was human-reviewed and tested on real hardware. Fixes: ggml-org#21517 * Fix rtl text rendering (ggml-org#21382) * Fix Arabic RTL text rendering in web UI - Add dir='auto' attributes to markdown containers and blocks - Implement post-processing to add dir='auto' to all text elements - Replace directional CSS properties with logical properties for proper RTL list alignment - Ensure bidirectional text support for mixed Arabic/English content * Clean up commented duplicate function Remove the commented-out duplicate transformMdastNode function that was left over from refactoring. * Fix Arabic RTL text rendering in web UI - Add dir='auto' attributes to markdown containers and blocks - Implement post-processing to add dir='auto' to all text elements - Replace directional CSS properties with logical properties for proper RTL list alignment - Minor code formatting improvements This ensures bidirectional text support for mixed Arabic/English content in the llama.cpp web UI. * Implement rehype plugin for comprehensive RTL text support - Add rehypeRtlSupport plugin that applies dir='auto' to all elements with children - Replace DOMParser-based approach with efficient HAST tree processing - Remove hardcoded element lists for better maintainability - Ensure proper bidirectional text rendering for mixed RTL/LTR content * Fix RTL text rendering with rehype plugin and cleanup * fix: prettier formatting * fix: Detect streaming state in reasoning content blocks (ggml-org#21549) * ggml-cuda : fix CDNA2 compute capability constant for gfx90a (MI210) (ggml-org#21519) GGML_CUDA_CC_CDNA2 was set to 0x910 Fix by setting the constant to 0x90a to match the actual gfx90a ISA. * webui : store reasoning_content so it is sent back in subsequent requests (ggml-org#21249) * vulkan: add FA dequant for q4_1, q5_0, q5_1, iq4_nl (ggml-org#21029) Add dequantize4() implementations for Q4_1, Q5_0, Q5_1, and IQ4_NL in the flash attention base shader. Register them in the shader generator, pipeline creation, and enable in the scalar/coopmat1 FA support check. * ggml: Vulkan build, Linux -- output error string for errno on fork failure (ggml-org#20868) (ggml-org#20904) * ggml : deprecate GGML_OP_ADD1 (ggml-org#21363) * ggml : deprecate GGML_OP_ADD1 * cont : remove tests * cont : re-enable vulkan check * server : fix restore for checkpoints with pos_min == 0 (ggml-org#21510) * llama: remove per-arch tensor name lists (ggml-org#21531) * unicode : add custom Qwen2 regex handler to fix segfault on long input (ggml-org#21257) * unicode : add custom Qwen2 regex handler to fix segfault on long input std::regex uses recursive backtracking internally, which causes a stack overflow (segfault) when tokenizing long sequences of repeated characters (e.g. 43K 'A's). The Qwen2 tokenizer regex differs from Llama3 only in the digit pattern (\p{N} vs \p{N}{1,3}), so it was falling through to the std::regex fallback path instead of using a custom handler. Add unicode_regex_split_custom_qwen2() following the established pattern used by gpt2, llama3, kimi_k2, and afmoe custom handlers. Closes: ggml-org#21113 * cont : remove TODO comment * cont : update comment to reflect original regex * use the correct regex in the comment this time... [no ci] --------- Co-authored-by: Aldehir Rojas <hello@alde.dev> * llama-server: fix model params not propagated (ggml-org#21509) Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * CUDA: check for buffer overlap before fusing (ggml-org#21566) * CUDA: check for buffer overlap before fusing * use ggml_cuda_check_fusion_memory_ranges * ggml-webgpu: parameterize submission size and add iOS specific limits (ggml-org#21533) * Work towards removing bitcast * Move rest of existing types over * Add timeout back to wait and remove synchronous set_tensor/memset_tensor * move to unpackf16 for wider compatibility * cleanup * Remove deadlock condition in free_bufs * Start work on removing parameter buffer pools * Simplify and optimize further * simplify profile futures * Fix stride * Try using a single command buffer per batch * formatting * Add parameters for different browsers in-flight submissions * Update handling of batch size too * Throttle ios as much as possible * Increase timeout for llvm-pipe testing * kv-cache : support attention rotation for heterogeneous iSWA (ggml-org#21513) * kv-cache : support attention rotation for heterogeneous iSWA * cont : remove assert * gguf-py : fix missing comma after bad merge in tensor-mapping (ggml-org#21558) This commit adds a missing comma in the vision encoder attention qkv block. The motivation for this change is that without the comma there will be a string concatenation of the Kimi-K2.5 and the Nemotron Nano v2 VL tensor mappings which will be broken. * ggml-cuda: ds_read_b128 for q4_0 and q4_1 mmq kernels (ggml-org#21168) * ds_read_b128 for q4_0 and q4_1 mmq kernels Current for loop generates ds_read_b32 instructions with hip compiler, the new solution generates ds_read_b128 instructions for the same operation, saving some LDS bandwidth. Tested on MI50 and RX6800XT, its faster on both. * Vectorized lds load update: used ggml_cuda_get_max_cpy_bytes and ggml_cuda_memcpy_1 functions for generic implementation * Explicit for loop in mmq, renamed vec into tmp * Fixed max_cpy usage in the loading loop * Fixed typo in q4_1 kernel * Update ggml/src/ggml-cuda/mmq.cuh Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Update ggml/src/ggml-cuda/mmq.cuh Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Update ggml/src/ggml-cuda/mmq.cuh Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Renoved trailing white line 500 * Update mmq.cuh removed other whitelines * Remove trailing whitespaces --------- Co-authored-by: iacopPBK <iacopPBK@users.noreply.github.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> Co-authored-by: iacopPBK <iacop@deneb.com> * CUDA: make cuda graphs props check faster (ggml-org#21472) * CUDA: compute fast hash instead of expensive props check * use seen node * use memcp * devops: kleidiai: provide KleidiAI-Enabled ARM Release Artifact (ggml-org#21259) * Unified macOS release setup with strategy-matrix block * Added KleidiAI arm64 macOS release definition Change-Id: I05520889ffc646488a178d06817a17f29274465a Signed-off-by: Martin Klacer <martin.klacer@arm.com> * webui: fix syntax highlighting lost after streaming for non-common languages (ggml-org#21206) * webui: fix syntax highlighting lost for non-common languages after streaming rehype-highlight uses lowlight internally, which only bundles 37 "common" languages. The streaming code path uses highlight.js directly (192 languages), so languages like Haskell highlight correctly while streaming but lose all color once the code block closes. Pass the full lowlight language set to rehype-highlight so both paths support the same languages. * webui: rebuild static files after rebase * model : support step3-vl-10b (ggml-org#21287) * feat: support step3-vl-10b * use fused QKV && mapping tensor in tensor_mapping.py * guard hardcoded params and drop crop metadata * get understand_projector_stride from global config * img_u8_resize_bilinear_to_f32 move in step3vl class * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix the \r\n mess * add width and heads to MmprojModel.set_gguf_parameters --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * chore: Remove legacy files (ggml-org#21606) * chore: Update labeler to have separate labels for `server/webui` and `server` changes (ggml-org#21567) * tests : remove obsolete .mjs script (ggml-org#21615) * parser: fix MiniMax handling (ggml-org#21573) * examples : disable cb_eval callback for --save-logits (ggml-org#21553) This commit updates the debug example to not create the base_callback_data. The motivation for this is when using `--save-logits`, which is used by examples/model-conversion scripts, we often don't care about the tensor outputs and they just add noise to the output. This changes is quiet by default we can always remove --save-logits to get the tensor outputs when debugging. * gemma : perform per-layer projections in the first layer (ggml-org#21612) * gemma : reduce graph splits by keeping per-layer ops in the input layer * gemma : put the per-layer proj in the first layer * cont : move the projection before the layer loop * metal: Q1_0 backend (ggml-org#21528) * initial Q1_0 Metal backend * tuning q1_0 metal kernels * add Q1_0 to test-backend-ops * add Q1_0<->F32 copy test * Apply suggestions from code review Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * webgpu : Query for adapter support when registering WebGPU backend (ggml-org#21579) * kv-cache : extend cache quantization checks (ggml-org#21586) to also check for enabled flash attention, instead of just auto. * Propose fix a couple of typos (ggml-org#21581) Signed-off-by: John E <jeis4wpi@outlook.com> * webui : send both backend_sampling == false/true (ggml-org#18781) * webui : send both backend_sampling == false/true * feat: Parameter sync --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> * vocab : remove </s> eog token if gemma4 (ggml-org#21492) * server: respect the ignore eos flag (ggml-org#21203) * fix: free ctx_copy in ggml_opt_free to plug per-training-session leak (ggml-org#21592) * fix: free ctx_copy in ggml_opt_free to plug per-training-session leak ggml_opt_alloc populates opt_ctx->ctx_copy via a free+init pair every time the allocated graph shape changes. The last ctx_copy from the final ggml_opt_alloc call survives until ggml_opt_free is invoked, but ggml_opt_free was only freeing ctx_static and ctx_cpu, never ctx_copy. Each opt_ctx lifetime therefore leaks the final per-batch context — ~900 KB for a typical GNN training session in sindarin-pkg-tensor, surfaced via AddressSanitizer. ctx_copy is nullptr-initialized and ggml_free() handles NULL safely, so the new release is guard-free. * Update ggml/src/ggml-opt.cpp Co-authored-by: Johannes Gäßler <johannesg@5d6.de> --------- Co-authored-by: realorko <realorko@nowhere.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * CUDA: also store `node->src->data` ptrs for equality check (ggml-org#21635) * CUDA: also store node->src->data ptrs for equality check * address review comments * common : skip non-primary GGUF split files when selecting model (ggml-org#21633) We should not assume files are listed in order. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * vulkan: unify type macros to use Vx instead of _VECx (ggml-org#21605) * ci: drop v5 `all:` composition from labeler.yml (ggml-org#21627) actions/labeler@v6 removed the `all:` / `any:` composition keys. The `server/webui` and `server` entries used `all:` to combine `any-glob-to-any-file` with negated `all-globs-to-all-files`, which now errors on every PR with: Unknown config options were under "changed-files": all Flatten both entries to a single `any-glob-to-any-file`. PRs touching both webui and other server files will now receive both labels instead of only `server/webui`. Co-authored-by: Marxist-Leninist <noreply@users.noreply.github.com> * sycl : add flash-attn support for head size 512 (ggml-org#21654) * sycl : add flash-attn support for head size 512 This patch extends the SYCL Flash Attention implementation to support head sizes (DKQ/DV) of 512. Changes: - Added DKQ/DV 512 cases to both tile and vector Flash Attention kernels. - Updated kernel selection logic to allow vector kernels for head sizes up to 512 (previously 256). - Removed unused/redundant AMD and RDNA-specific configuration functions in `fattn-tile.hpp`. - Refactored `ggml_backend_sycl_buffer_init_tensor` to use a switch statement for clearer tensor extra buffer initialization. - Added necessary template instances for the new 512 head size across various quantization types. * remove defunct mxfp4 reorder from setting buffer type * webui: Add option to pre-encode conversation for faster next turns (ggml-org#21034) * server : fix grammar commandline args (ggml-org#21543) Co-authored-by: AUTOMATIC <-> * fix: Model Selector choice sync (ggml-org#21628) * metal : add missing mm-id specializations for q1_0 (ggml-org#21662) * jinja : support ensure_ascii=true, string repetition and int/float self-filtering (ggml-org#21623) * feat: jinja engine improvements for reka-edge Port three Jinja engine improvements needed for the reka-edge model: 1. Python-style string repetition ("ab" * 3 → "ababab") 2. ensure_ascii=true support for tojson filter (escapes non-ASCII to \uXXXX) 3. int() builtin on value_int_t (identity, needed for Reka Edge template) * fix: escape invalid utf8 bytes when ensure_ascii=true The json_ensure_ascii_preserving_format function does not correctly handle an edge case where if UTF-8 parsing fails, it adds the non-ascii character back to the output as a raw byte. This commit fixes that by adding the unicode standard replacement character \\ufffd to the output instead. This is the standard behavior for various programming languages like Python, Rust, Go, etc. * chore: address PR comments 1. Add todo comment for supporting string repetition for array/tuples 2. Add support for float identity operation 3. Move invalid ascii test case to test_fuzzing * chore: accept suggestion for common/jinja/value.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * vocab: add gemma4 tokenizer tests, fix edge case (ggml-org#21534) * YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests! * Remove unnecessary hash from update script. * minor: move constant * mtmd: support dots.ocr (ggml-org#17575) * convert gguf * clip impl * fix conversion * wip * corrections * update docs * add gguf to test script * model: fix multimodal padding token for gemma3n/gemma4 (ggml-org#21625) * model: fix multimodal padding token for gemma3n/gemma4 * nits * common : simplify autoparser tagged parser rules (ggml-org#21216) * common : simplify autoparser tagged parser rules * cont : remove upper limit on optional args * cont : revert changes to parsing at the end * cont : undo arbitrary ordering of optional args * cont : fix uninitialized required parameters * revert to simplify merge * re-apply patches * restore flexible optional arg ordering tests * common : fix ambiguous grammar rule in gemma4 (ggml-org#21661) * common : fix ambiguous grammar rule in gemma4 * cont : fix missing comma... * webui: add "Send message on Enter" setting (ggml-org#21577) * webui: make Enter to send chat a setting * Shorten description * Use isMobile hook from $lib/hooks * Rebuild static output * requirements : update transformers to 5.5.1 (ggml-org#21617) * requirements : update transformers to 5.5.0 This commit updates the transformers dependency to version 5.5.0. The motivation for this is that transformers 5.5.0 includes support for Gemma4 and is required to be able to convert Gemma4 models. This is also causing issues for user of gguf-my-repo. Refs: https://huggingface.co/spaces/ggml-org/gguf-my-repo/discussions/202 * fix huggingface_hub version * set version of transformers to 5.5.0 * convert : add ty ignore directives to convert_hf_to_gguf.py This commit adds `ty: ignore` directives to transformers tokenizers field/methods to avoid type check errors. There might be better ways to handle this and perhaps this can be done in a follow up commit. The motivation for this is that it looks like in transformers 5.5.0 AutoTokenizer.from_pretrained can return generic tokenizer types or None and the type checker now produces an error when the conversion script accesses field like tokenizer.vocab. * convert : add ty ignore to suppress type check errors * convert : remove incorrect type ignores * convert : fix remaining python checks I was running a newer version of ty locally but I've switched to version 0.0.26 which is what CI uses and I was then able to reproduce the errors. Sorry about the noise. * update transformers version to 5.5.1 * ggml : check return value of CUB calls used in argsort and top-k (they all return cudaError_t) (ggml-org#21676) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * ggml: backend-agnostic tensor parallelism (experimental) (ggml-org#19378) * ggml: backend-agnostic tensor parallelism * support for GPT-OSS, Qwen 3 MoE * partial Vulkan fix * add support for 4/8 GPUs * unconditional peer access * re-use buffers + ggml contexts * fix output pattern * NCCL support * GGML: HIP: add RCCL support * Remove shfl and AllReduce from backend interface * move allocation workaround out of ggml-alloc.c * 2d tensor set/get support * Fix the seg fault without NCCL * Apply suggestion from JohannesGaessler * support for tensor dims % n_devs != 0 * fix view_offs scaling * arbitrary num. of GPUs/tensor split * fix compilation * better granularity estimate * Support device-specific host buffer types if all underlying backends expose the same type. This allows using pinned memory instead of pageable memory for CUDA. Fix compilation errors. * partial Qwen 3 Next support * Fix qwen3 30b (ggml-org#8) * Fix crash with Qwen-30B-A3B Q4_0 Qwen-30B-A3B Q4_0 has an intermediate dimension of 768. Using a granularity of 256 forces an uneven split between GPUs, which is not supported by the current implementation. * Decide block size based on tensor quantization type * Fix crashes due to KV cache serialization (ggml-org#9) KV cache serialization requires non-zero offsets on the tensor. Add support in the meta backend to set/get a tensor with a non-zero offset. * metal : fix build (ggml-org#7) * static memory allocations, fix usage count * fix tensor granularity * more even memory distribution * use BF16 for allreduce * rebase fixup * better error message for unsupported architectures * Fix device mismatch during scatter of allReduce. (ggml-org#11) There is a mismatch between the dst buffer device and the backend device, causing the use of sync copies * Enable the previous allreduce implementation. It is better in both perf and stability (ggml-org#12) * delay AllReduce for Moe for less I/O * build : clean-up compile warnings * backend : move most of the meta backend API to ggml-backend-impl.h * cont : hide unused public API in the implementation * llama : use llama_device + remove ggml_backend_dev_is_meta() * ggml-backend : remove unused alloc include * minor : remove regex include * ggml : introduce ggml-ext.h for staging new APIs * rebase fixup * fix tests * llama : more robust logic for determining Meta devices (ggml-org#16) * llama : more robust logic for determining Meta devices * cont : fix devs size check Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * cont : fix log type Co-authored-by: Johannes Gäßler <johannesg@5d6.de> --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * disable roundtrip for meta backend * fix arch selection * Qwen 3.5 support * fix Gemma 4 MoE * fix OpenVino, SYCL * fix test-llama-archs for CPU-only builds * Fix Qwen 3.5 MoE * disable meta backend tests for WebGPU * tests : filter CPU-based devices from the Meta backend tests (ggml-org#17) * meta : formatting, naming, indentation (ggml-org#18) * formatting : llama-model.cpp * formatting : ggml-ext.h * formatting : ggml-backend-meta.cpp * meta : add TODO * add documentation * better error messages * fix GPT-OSS --------- Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * HIP: add CDNA4 (gfx950) architecture support for MI350X/MI355X (ggml-org#21570) Add AMD Instinct MI350X/MI355X (gfx950, CDNA4) support: - vendors/hip.h: Add CDNA4 preprocessor define for __gfx950__ - common.cuh: Add GGML_CUDA_CC_CDNA4 and GGML_CUDA_CC_IS_CDNA4 macros - mma.cuh: Route CDNA4 to compatible MFMA instructions: * f32 matmul: mfma_f32_16x16x4f32 (xf32 variant unavailable on gfx950) * bf16 matmul: mfma_f32_16x16x16bf16_1k (same as CDNA3) * int8 matmul: mfma_i32_16x16x32_i8/32x32x16 (same as CDNA3) - mmq.cuh: Include CDNA4 in stream-k kernel dispatch CDNA4 is largely compatible with CDNA3 except: - No xf32 MFMA (mfma_f32_16x16x8_xf32) — routes to f32 path - Different FP8 format (e4m3fn vs e4m3_fnuz) — not changed here Tested on AMD Instinct MI355X (gfx950), ROCm 7.0.1: - Build: compiles cleanly with -DAMDGPU_TARGETS=gfx950 - llama-bench (Qwen2.5-1.5B Q4_K_M, single GPU): * f16+FA: 40,013 tok/s prefill, 254 tok/s decode * q8_0+FA: functional - Flash attention: works correctly - MMQ: works correctly with stream-k dispatch Co-authored-by: Andy Luo <andyluo7@users.noreply.github.com> * CUDA: fuse muls (ggml-org#21665) * common : add fluidity to the progress bar (ggml-org#21671) Signed-off-by: Adrien Gallouët <angt@huggingface.co> * vulkan: Support Q1_0 (ggml-org#21539) * vulkan: Support Q1_0 * use get_dm * docs : fix broken link to ggml-openvino in OPENVINO.md (ggml-org#21709) * common : enable reasoning budget sampler for gemma4 (ggml-org#21697) * fix: enable reasoning budget sampler for gemma4 Add thinking_start_tag and thinking_end_tag to common_chat_params_init_gemma4(). Without these, the reasoning budget sampler never activates for gemma4. Make the newline after "thought" optional in the PEG parser to handle budget=0 (sampler forces end tag before the newline). Add test case for empty thinking block. Fixes ggml-org#21487 * use p.space() instead of p.optional(p.literal("\n")) in gemma4 thought parser * webui: Static build output improvements (ggml-org#21667) * refactor: Build improvements * chore: Formatting + package lock update * common: mark --split-mode tensor as experimental (ggml-org#21684) * common : fix when loading a cached HF models with unavailable API (ggml-org#21670) Signed-off-by: Adrien Gallouët <angt@huggingface.co> * server : ignore --alias when using --models-preset (ggml-org#21380) I'm not sure what the purpose of keeping `--alias` was when using `--models-preset`, but the result is really weird, as shown in the following logs: $ build/bin/llama-server --models-preset preset.ini --alias "Gemma 4 E4B UD Q8_K_XL" ... init: using 31 threads for HTTP server srv load_models: Loaded 2 cached model presets srv load_models: Loaded 1 custom model presets from preset.ini main: failed to initialize router models: alias 'Gemma 4 E4B UD Q8_K_XL' for model 'angt/test-split-model-stories260K:F32' conflicts with existing model name So I propose to simply ignore `--alias` too in this case. With this commit, the server starts in routing mode correctly. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * ggml-webgpu: address quantization precision and backend lifecycle managment (ggml-org#21521) * ggml(webgpu): fix the busy-polls in Emscripten in the waitAny after ggml-org#20618, and remove the busy webgpu log * Merge with upstream * Fix GET_ROWS packed integer NaN when using f16 as memory buffer in shader quants * Update Unary wgsl EXP and EXPM1 for f16 stability * Fix GET_ROWS IQ4_XS strcut for NaN f16 canonicalization * Fix numerical percision for unary sqrt when working with f16 * Fix NaN canonicalization for packed integers using f16 * Update err threshold for binary div ops when using f16 * backend: Keep one Dawn/WebGPU instance alive for the lifetime of the static backend * clean: uncomment existing code logs * clean: clean the unncessary debug info * Refactor and generalize dequant helpers * Remove deprecated quant structs * Refactor shader defines to reduce repetition * Remove error override for F16 type * fix: fix the accidential removal of the proper initialization of ctx * clean: clean legacy and format code * fix: did not modify tests ops --------- Co-authored-by: Jeremy J. Hartmann <jeremy@mtion.tv> * ggml-webgpu: support non-square subgroup matrix configs for Intel GPUs (ggml-org#21669) * model : make Gemma 4 shared-KV tail attn_k tensors optional on load (ggml-org#21739) * common : add callback interface for download progress (ggml-org#21735) Signed-off-by: Adrien Gallouët <angt@huggingface.co> * common : better align to the updated official gemma4 template (ggml-org#21704) * hexagon: improved Op queuing, buffer and cache management (ggml-org#21705) * hexagon: introduce op request batching and rewrite buffer managment The host now prepares batches of requests and dispatches them via a single dspqueue message. Buffers are mapped explicitly by NPU while processing batches. * hex-dma: disable l2 bypass since to work around new issue due to no flushes between Ops * hex-utils: add explicit l2flush and l2clear helpers * hex-opreq: use fine-grain per tensor l2 management * hex-opreq: avoid redundant invalidates for tensors we already flushed * hex-opreq: update debug messages * htp-opreq: reuse ops_context * hex-opreq: do not flush or invalidate cache lines beyond buffer boundry * hex-opreq: fix errors in log message * Revert "hex-opreq: do not flush or invalidate cache lines beyond buffer boundry" This reverts commit 8b7f0a55a750a6430ce4eb1874c7feb3d720056d. * hexagon: limit l2 flushes to 1MB which covers l2 cache * hex-opreq: limit cache flush to 4MB Looks like 4MB cont. vitual space should cover the 1MB cache. * hexagon: drop cache flush size to 2MB * hex-opreq: start reworking opreq packing * hex-opreq: introduce new way of packing opbatch where tensors are stored separately * hex-opreq: add a simple fastrpc call to force unmap all buffers * hex-l2flush: somehow 2MB does not seem robust, also cleanup step size to use line-size * hex-opreq: bump opreq batch size to 256 * hex-mm: place src1 spad at the top of vtcm for easy reuse * hex-ops: introduce internal types and disable src1 reuse for now Nothing new just formalizing the repack / qyn.quant types we've been using. * htp-opreq: use tensor pointers instead of copies * hex-opreq: introduce more robust way for tracking vtcm/spad reuse This removes the SKIP_QUANTIZE flag that became fragile with the addition of HMX and other ops. * hex-cumsum: fix error post opreq merge * hex-opreq: move request batch handling into the session Prepping everything for using dspqueue buffers and doing that inside the session is much cleaner. * hex-mm: yet another fix for src1 reuse when we're mixing hmx/hvx * hex-bufs: introduce pinned mmapings and use non-pinned ones for model buffers * hex-buf: add support for allocating shared/pinned buffer for opreqs * hex-opbatch: make opbatches configurable * hex-naming: better name for ggml_hexagon_shared_buffer * hex-naming: add session->c_name() helper * hex-opbatch: start using shm but still copy for now * hex-opbatch: use shared buffer for packing opbatch * hex-opbatch: beter naming for opbatch related classes and code * hex-opbatch: reuse batched tensors with same data/dims/strides * hex-opbatch: update logging * hex-opbatch: add support for vmem limit for op batching * hex-opbatch: update htp side to properly support dynamic mmap/unmap * hex-opbatch: add OB and OQ params for run-completion script and fix the asserts in batch processing * hex-opbatch: fixed src1 handling in act ops * hex-act: fix empty src1 handling in swiglu and friends Simplify preamble macro while at it * hex-mm: minor fix vtcm and dma handling in matmul cleaning up some left-overs from merges * hex-opbatch: allocate extra 1KB for dspqueue overhead * hexagon: fix softmax for non-aligned tensors and cleanup vtcm alloc * hex-mm: properly handle hmx_disabled flag * hex-ops: update comments * hex-ops: add debug output for get/set-rows * hex-mmap: optimize un/mapping of buffers * hex-opreq: global cache flush and invalidate beyond 128KB threshold * hex-ops: add super simple opfilter regex for debugging If an Op matches the regex hex backend will reject it. * hex-opbatch: wireup newer ops missed in merge and update main switch to detect this in future * hexagon: improved vtcm acquision to remove inter-op overhead Fully compatible with QNN-HTP coex * hex-mm: fixed hvx fallback path * hex-mm: lower the vmem threshold a bit further to ~3GB * hexagon: update debug & error logs This also fixes an issue with newer llvm merging repack and non-repack functions. We use those pointer to distinguish between buffer types. * hexagon: move ops context into main context Just a cleanup. We don't need separate contexts at this point. * hex-opbatch: cleanup naming and headers for opbatch and related descriptors * hex-fa: it's now better to enable FA during TG to reduce graph splits * hexagon: remove GGML_HEXAGON_EXPERIMENTAL env var It's no longer useful. Please use more flexible GGML_HEXAGON_OPFILTER to disable Ops if needed for debugging or validation. * hexagon: fixed editorconfig check * Update ggml/src/ggml-hexagon/ggml-hexagon.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Trivikram Reddy <tamarnat@qti.qualcomm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * hexagon: add support for linux on snapdragon (ggml-org#21707) * hexagon: add support for debian on ex2 * hexagon: add -fvectotize to c/c++ cmake flags * hexagon: remove trailing white space * update onboarding steps * hexagon: update linux setup documentation * hexagon: update intallation scripts * Hexagon: update docs * hexagon: update onboarding scripts --------- Co-authored-by: Zack Li <zackli@qti.qualcomm.com> * fix: Fix broken structured output when using $refs in json_schema (ggml-org#21699) * CUDA: also store node->src ne/nb for graph equality (ggml-org#21736) * py : Bump typer to latest to fix huggingface_hub issue (ggml-org#21701) * ggml : fix a few instances of missing GGML_TYPE_Q1_0 cases (ggml-org#21716) * TP: fix Qwen 3 Next data split (ggml-org#21732) * opencl: add basic support for q5_k (ggml-org#21593) * opencl: add general q5_k mv * opencl: add flattened Q5_K mv and general Q5_K mm * opencl: fix Q5_K unit tests --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co> Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> Signed-off-by: Martin Klacer <martin.klacer@arm.com> Signed-off-by: John E <jeis4wpi@outlook.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> Co-authored-by: Ruben Ortlam <rortlam@redhat.com> Co-authored-by: Zheyuan Chen <sephirotheca17@gmail.com> Co-authored-by: Reese Levine <reeselevine1@gmail.com> Co-authored-by: Bartowski <3266127+bartowski1182@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Slobodan Josic <127323561+slojosic-amd@users.noreply.github.com> Co-authored-by: Vishal Singh <vishal@zettabolt.com> Co-authored-by: Aaron Teo <taronaeo@gmail.com> Co-authored-by: Radoslav Gerganov <rgerganov@gmail.com> Co-authored-by: sayap <sokann@gmail.com> Co-authored-by: Tillerino <Tillerino@users.noreply.github.com> Co-authored-by: uvos <carl@uvos.xyz> Co-authored-by: Aaron Teo <aaron.teo1@ibm.com> Co-authored-by: jeromew <jerome.wagner@m4x.org> Co-authored-by: M1DNYT3 <42499082+M1DNYT3@users.noreply.github.com> Co-authored-by: M1DNYT3 <m1dnyt3@MacBookPro.lan> Co-authored-by: CISC <CISC@users.noreply.github.com> Co-authored-by: Samanvya Tripathi <samanu09@gmail.com> Co-authored-by: Yes You Can Have Your Own <188969017+yychyo@users.noreply.github.com> Co-authored-by: Masato Nakasaka <masato.nakasaka@intel.com> Co-authored-by: Aman Gupta <amangupta052@gmail.com> Co-authored-by: SamareshSingh <97642706+ssam18@users.noreply.github.com> Co-authored-by: Adrien Gallouët <angt@huggingface.co> Co-authored-by: Dan Hoffman <43101339+thedanhoffman@users.noreply.github.com> Co-authored-by: Dan Hoffman <dhoffman@cyket.net> Co-authored-by: Aldehir Rojas <hello@alde.dev> Co-authored-by: Nicholas Sparks <157740354+nisparks@users.noreply.github.com> Co-authored-by: ddh0 <chemist-mulches-39@icloud.com> Co-authored-by: Ludovic Henry <ludovic@rivosinc.com> Co-authored-by: Richard Davison <richard.davison1@gmail.com> Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> Co-authored-by: anchortense <daniel.redshaw@uqconnect.edu.au> Co-authored-by: Yarden Tal <yardent@qti.qualcomm.com> Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> Co-authored-by: lainon1 <271530700+lainon1@users.noreply.github.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Bipin Yadav <83943505+bipinyadav3175@users.noreply.github.com> Co-authored-by: Pasha Khosravi <khosravipasha@users.noreply.github.com> Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com> Co-authored-by: Dmytro Romanov <casteldazur@gmail.com> Co-authored-by: PMZFX <georgiopapairo@gmail.com> Co-authored-by: Kabir08 <62639358+Kabir08@users.noreply.github.com> Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> Co-authored-by: Antoine Viallon <antoine@lesviallon.fr> Co-authored-by: mkoker <132301062+mkoker@users.noreply.github.com> Co-authored-by: Tom Overlund <tomov@dilacero.org> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com> Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com> Co-authored-by: iacopPBK <iacopogiottorossi@gmail.com> Co-authored-by: iacopPBK <iacopPBK@users.noreply.github.com> Co-authored-by: iacopPBK <iacop@deneb.com> Co-authored-by: Martin Klacer <martin.klacer@arm.com> Co-authored-by: Hamish M. Blair <hmblair@stanford.edu> Co-authored-by: forforever73 <63285796+forforever73@users.noreply.github.com> Co-authored-by: Erik Scholz <Green-Sky@users.noreply.github.com> Co-authored-by: John Eismeier <42679190+jeis4wpi@users.noreply.github.com> Co-authored-by: Yuri Khrustalev <ykhrustalev@users.noreply.github.com> Co-authored-by: RealOrko <45273739+RealOrko@users.noreply.github.com> Co-authored-by: realorko <realorko@nowhere.com> Co-authored-by: Marxist-Leninist <31905382+Marxist-Leninist@users.noreply.github.com> Co-authored-by: Marxist-Leninist <noreply@users.noreply.github.com> Co-authored-by: Akarshan Biswas <akarshan@menlo.ai> Co-authored-by: AUTOMATIC1111 <16777216c@gmail.com> Co-authored-by: Kwa Jie Hao <31984694+kwajiehao@users.noreply.github.com> Co-authored-by: JvM <mourix@live.nl> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com> Co-authored-by: Andy Luo <andyluo7@users.noreply.github.com> Co-authored-by: Jeff Bolz <jbolz@nvidia.com> Co-authored-by: Belem Zhang <belem.zhang@intel.com> Co-authored-by: Berk Idem <55372926+berkidem@users.noreply.github.com> Co-authored-by: Chen Yuan <constant.chen@uwaterloo.ca> Co-authored-by: Jeremy J. Hartmann <jeremy@mtion.tv> Co-authored-by: Rithik Sharma <rithiksh02@gmail.com> Co-authored-by: MoonRide303 <130458190+MoonRide303@users.noreply.github.com> Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> Co-authored-by: Trivikram Reddy <tamarnat@qti.qualcomm.com> Co-authored-by: Todor Boinovski <todorb@qti.qualcomm.com> Co-authored-by: Zack Li <zackli@qti.qualcomm.com> Co-authored-by: Galunid <karolek1231456@gmail.com> Co-authored-by: shaofeiqi <shaoqi@qti.qualcomm.com>

* webui : send both backend_sampling == false/true * feat: Parameter sync --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

…tion-Id (#25760) * server: return 400 instead of 500 on validation error with X-Conversation-Id set_req() attaches the spipe as soon as the header is present, before the request body is parsed. When params validation throws, set_next() never runs and next_orig stays empty, so on_complete() called it and crashed with std::bad_function_call, turning the prepared 400 JSON into a generic 500. on_complete() now treats an empty next_orig as "streaming never started" and evicts the session installed by set_req(), so a failed request leaves nothing behind for discovery or replay. This also covers valid requests that carry the header but do not stream, which previously left an empty finalized session in the map until the GC TTL. * ui: do not send the backend_sampling placeholder On a fresh profile the syncable settings hold the empty string placeholder meaning "let the server decide". Every neighbor field goes through the hasValue() guard that filters it, except backend_sampling, which sent the placeholder verbatim and made every default settings completion fail validation. Guard the field with hasValue() like its neighbors. hasValue(false) is true, so an explicit false still reaches the server and the intent of #18781 (send both true and false) is preserved. Only the placeholder is filtered.

* sync : ggml * opencl: transpose q4_K noshuffle scales for coalesced reads (#25805) * opencl: read/write MoE dp4a activation tiles to local memory as 128-bit (vectorized LD/ST perf opt) for Adreno GPUs (#25810) * opencl: read MoE dp4a activation tile as 128-bit local loads * opencl: vectorize MoE dp4a activation staging as 128-bit loads * opencl: load and use `kernel_gemm_moe_q6_k_f32_ns` from bin kernel lib (#25797) * llama-quant : exclude i32 ffn_gate_tid2eid routing table from quantization (#25787) DeepSeek-V4's ffn_gate_tid2eid tensor is an i32 token-id -> expert-id index table, not weights. It was never added to the name-based exclusion list alongside ffn_gate_inp.weight, so llama-quantize tries to quantize it and fails since i32 cannot convert to a float type. Fixes ggml-org/llama.cpp#25754 Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com> * model: rotate injected K/V cache for DFlash (#25823) * dflash: rotate injected K/V cache when using K/V quantization * Update src/models/dflash.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * clearer format * remove trailing whitespace --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * opencl: Support broadcast for Adreno MUL_MAT and honor `view_offs` for Adreno Q8_0 MUL_MAT for llama-server multi-stream (#25910) * opencl: handle broadcast for adreno gemm/gemv_noshuffle * opencl: honor view_offs for adreno noshuffle gemm/gemv * opencl: general GEMM/GEMV support broadcast * opencl: remove unnecessary tests * opencl: remove unnecessary comments --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * ui: enable the agentic flow when only the JS sandbox is active (#25865) The agentic gate counted MCP servers, builtin and custom tools but not frontend tools, so with the JS sandbox as the only tool source the agentic flow was skipped, no tools field reached the server and the chat template rendered without the tool system prompt. The sandbox is fully client-side: frontendTools derives from the Developer settings toggle alone, counting it in the gate restores that single source of truth. * UI: fix Settings/Display tool call content toggle (#25783) * ui: fix Show tool call in progress toggle ignored The showToolCallInProgress setting was disconnected from the render path during the agentic content rework: getDefaultExpanded() returns a hardcoded false for tool call sections and an unconditional effect auto-expands the currently executing tool call regardless of the setting. Drive default expansion of all tool call section types from the setting and remove the now redundant auto-expand effect. Manual toggling still takes precedence over the default in both directions. * ui: rename Show tool call in progress to Always show tool call content The previous name suggested symmetry with Show thought in progress, which only applies while inference is running, but tool call content stays expanded after completion. Rename the label, the settings key and the syncable server key to alwaysShowToolCallContent. The synced parameter never worked under its previous name so no migration is needed. * ui: fix collapsed user bubble with markdown rendering (#25869) Edge paragraph margins are now zeroed at the source in markdown-content.css, but the user bubble and the system message still carried the -my-4 compensation for them. The uncompensated negative margins shrank the wrapper 2rem below its content, collapsing single-line user bubbles into a scrollable sliver and skewing the system message expand threshold. * llama_dsv4: write only used rows in state (#25325) * llama_dsv4: write only used rows in state * add TODO about conflating token pos with kv rows * ui: Sidebar Conversations Bulk Action + Improved Settings logic/UI (#25815) * feat: WIP * feat: Replace conversation rename flow with unified AlertDialog component * feat: Add radio group component and consolidate title generation settings * refactor: Remove JS Sandbox global toggle and migrate legacy user state * chore: Formatting * refactor: Cleanup Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> * refactor: Cleanup * refactor: Marquee selection hook * feat: UI improvements * refactor: Bulk db operations * fix: optimize bulk conversation deletion to handle ancestor chains * refactor: remove pairedKey mechanism from settings system * fix: remove redundant onclick handler from dialog cancel button * chore: pin @lucide/svelte to exact version * feat: Run JavaScript tool disabled by default * fix: correct active conversation deletion tracking in bulk delete * feat: improve shift-key multi-selection support in sidebar via keyboard * refactor: Retrieve JS Tool enabling via Developer Settings * nits: sync, dialog wording, cycle guard, and lockfile follow-ups - Restore titleGenerationUseLLM registry entry so it syncs across devices again - Mention fork cascade in the bulk delete confirmation dialog - Clear newParent on cycle guard break so children never point at a deleted conversation - Align @lucide/svelte in package-lock.json with the exact pin in package.json --------- Co-authored-by: Pascal <admin@serveurperso.com> * hexagon: add CLAMP op (#25934) * CUDA: vectorize same-type get_rows with int4 copy (#25929) k_get_rows_float did a scalar one-element-per-thread copy and recomputed the row-invariant work (index load, fast_div_modulo, src/dst row pointers) for every element. Hoist that out of the per-element loop, and add a vectorized path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the contiguous same-type (no-cast) case. The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at runtime on 16-byte alignment of the base pointers and all row strides and on ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small single-row gather can drop below the device CU count and regress; an occupancy gate keeps those on the block-rich scalar path. On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288) drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan backend, with no regression on the small conv-state gather; total get_rows -27%. test-backend-ops GET_ROWS passes (47/47). Assisted-by: Claude Opus 4.8 * ggml-openvino: Add GGML_BACKEND_DL_IMPL invocation for OpenVINO backend (#25795) This adds the missing `GGML_BACKEND_DL_IMPL()` macro invocation, that other backends have. Fixes #25586 for me * vulkan: Refactor vk_queue to use per-instance mutexes and unique handles (#23570) * Refactor vk_queue to use per-instance mutexes and unique handles * integrates VK_KHR_internally_synchronized_queues, abstracting the queue submission into a polymorphic interface that completely bypasses host-side mutex locking when driver-side synchronization is supported * fix compilation error * fix duplicate pNext chain for VkPhysicalDeviceInternallySynchronizedQueuesFeaturesKHR * add fallback defines for VK_KHR_internally_synchronized_queues * add null checks for queues in vk_device_struct destructor * use unique_ptr for outer queues to enforce exclusive ownership and optimize lifetime * use static constexpr for eInternallySynchronizedKHR * add lock guard to ggml_vk_create_aliased_queue for thread safety * initialize sync_query_features.internallySynchronizedQueues to VK_FALSE * reuse sync_query_features for internallySynchronizedQueues and simplify chaining * refactor internallySynchronizedQueues detection * fix internallySynchronizedQueues query guard * use eInternallySynchronizedKHR constant * fix self-referential alias for eInternallySynchronizedKHR * use macro for eInternallySynchronizedKHR fallback * fix internallySynchronizedQueues query timing in ggml-vulkan.cpp to prevent device creation mismatch * reset sync_query_features.pNext before reusing in device creation chain, also removed the redundant second probe call * refactor internally synchronized queues detection to use chained feature query and avoid redundant API calls * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * rename sync_enable_features to internally_synchronized_queues_features * queue_flags is still computed before has_internally_synchronized_queues is set * fix trailing whitespace * replace eInternallySynchronizedKHR macro with static constexpr * preserve source queue semantics in single-queue aliased transfer queue * vulkan: fix cmd_pool access via pointer for compute_queue unique_ptr * vulkan: lock queue during debug label emission when not internally synchronized --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * server : properly handle null llama_context (#25868) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * server: return 400 instead of 500 on validation error with X-Conversation-Id (#25760) * server: return 400 instead of 500 on validation error with X-Conversation-Id set_req() attaches the spipe as soon as the header is present, before the request body is parsed. When params validation throws, set_next() never runs and next_orig stays empty, so on_complete() called it and crashed with std::bad_function_call, turning the prepared 400 JSON into a generic 500. on_complete() now treats an empty next_orig as "streaming never started" and evicts the session installed by set_req(), so a failed request leaves nothing behind for discovery or replay. This also covers valid requests that carry the header but do not stream, which previously left an empty finalized session in the map until the GC TTL. * ui: do not send the backend_sampling placeholder On a fresh profile the syncable settings hold the empty string placeholder meaning "let the server decide". Every neighbor field goes through the hasValue() guard that filters it, except backend_sampling, which sent the placeholder verbatim and made every default settings completion fail validation. Guard the field with hasValue() like its neighbors. hasValue(false) is true, so an explicit false still reaches the server and the intent of #18781 (send both true and false) is preserved. Only the placeholder is filtered. * common: resolve draft repo to its requested sidecar (#25955) With -hfd pointing to a repo shipping speculative sidecars, the draft resolved to the main model of that repo, since find_best_model() excludes sidecar files, and the explicit draft plan suppressed the sidecar discovery on the -hf repo. The draft plan already discovers its sidecars, they were just never consumed. Wire them as the draft, following the fallback pattern of the main plan, so this now works as expected: llama-server -hf repo -hfd repo --spec-type draft-dflash * kleidiai : warn once when a weight type has no KleidiAI kernel (#25701) * cuda: add sqrt_softplus in topk-moe for dsv4 (#25896) * hexagon: check tensor type when reusing descriptors (#25968) * mtmd : use align_corners for qwen3vl vision position embedding interpolation (#25781) The Qwen3-VL learned position embedding is interpolated to the runtime patch grid with the default bilinear+antialias (align_corners=False) sampling, while the transformers reference uses align_corners=True (torch.linspace(0, side-1, T)). The mismatch scales grounding coordinates about the image center, growing with image size and per-axis for non-square images (see #16880). * convert: fix handle HunyuanVL XD-RoPE config (#25514) Signed-off-by: wendadawen <wendadawen@qq.com> * Add support for Laguna XS.2 & M.1 (#25165) * llama-arch: fix DeepSeek4 APE tensor op (#25945) * cuda: GET_ROWS quants (#25962) * cuda: add k-quant support to GET_ROWS Device-side embedding lookups require GET_ROWS to handle the k-quants used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without it the backend rejects the op and the scheduler falls back to the host, copying the full embedding matrix back on every token in single-device graphs. Factor the super-block dequantizers out of the dequantize_block kernels in convert.cu into shared device functions in dequantize.cuh and reuse them from a new k_get_rows_kq kernel : one thread block dequantizes one (dst row, super-block) pair with the existing thread layouts, 32 threads for q4_K and 64 for the other k-quants. Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left as a TODO. * cuda: add i-quant support to GET_ROWS Extends the shared super-block dequantizers to the nine i-quants and reuses them from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernels. supports_op gates the k-quant and i-quant path on ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its own (QK4_NL sub-blocks). mxfp4 is left as a TODO. * cuda: add mxfp4 support to GET_ROWS Moves the mxfp4 dequantizer into the shared super-block helpers and reuses it from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernel. mxfp4 joins the ne0 % QK_K gate in supports_op since its 32-value sub-blocks do not guarantee QK_K-aligned rows on their own. This closes GET_ROWS type coverage on CUDA: every quantized GGML type now takes the direct device path. * cuda: gate the GET_ROWS row size only for 32-value sub-block types Address review from @pwilkin: the i-quant commit replaced the return shared by the whole supported type cascade, so f16/f32/bf16/i32 and the legacy quants also inherited the ne0 % QK_K == 0 gate and any row size that is not a multiple of 256 fell back to the scheduler. Split the cascade: unconditional support is restored everywhere, the gate stays only on iq4_nl and mxfp4 whose 32-value sub-blocks do not guarantee the QK_K super-blocks the kernel iterates on. * webgpu : add CONV_2D_DW (depthwise conv2d) kernel (#25847) * webgpu : add CONV_2D_DW (depthwise conv2d) kernel Implement GGML_OP_CONV_2D_DW for the WebGPU backend, ported from the Vulkan backend's conv2d_dw.comp. Assisted-by: Claude Opus-4.8 * Remove unnecessary comments in webgpu support * update supported ops tables, triggered by adding webgpu CONV_2D_DW * ci : fix SYCL package shared library lookup (#25987) * ggml: enable PowerPC backend variants on AIX (#25983) * ggml: enable PowerPC backend variants on AIX Allow the PowerPC CPU backend variants to be built on AIX by extending the platform check in the CMake configuration. This reuses the existing PowerPC backend implementations without changing their behavior. Also fix a missing semicolon in the PowerPC Q0 matmul implementation. * Fix missing semicolon in sgemm.cpp * Fix DeepSeek4 crafted template (#25414) * chat: fix DS4 template to explicitly follow reference behavior * Support DeepSeekv4 flag (`drop_reasoning`). * fix: hook DS3.2 parser for DS4 as well * fix: add tool result reordering * fix: post-merge * common: infer the speculative type from the draft repo sidecars (#25989) With -hfd pointing to a repo that ships mtp-/dflash-/eagle3- sidecars and no --spec-type given, the draft resolved to a full model while the sidecar was the intended draft. When the speculative types are still at their default, discover the sidecars of the draft repo, pick the first available following the existing mtp > dflash > eagle3 priority, and set the corresponding type, so this now works without any extra flag: llama-server -hf repo:Q3_K_M -hfd repo:Q8_0 An explicit --spec-type disables the inference, and a draft repo without sidecars keeps resolving to a full model as before. * minor: fix reasoning preserve var for DS4 [no ci] (#25999) * feat(ui): add symbolic math support to JS sandbox via nerdamer (#25948) * feat(ui): add symbolic math support to JS sandbox via nerdamer Preload nerdamer (with decimal.js) in the sandboxed worker, exposing the `nerdamer` global for symbolic computation: simplify, expand, factor, diff, integrate, solve, laplace, ilt, limit, partfrac, gcd/lcm, roots, coefficients, and more. Mirrors the math.js integration pattern from the feature/sandbox-symbolic-math branch, but uses nerdamer for a lighter, more focused symbolic math engine. * Update sandbox-harness.ts * docs(ui): update sandbox tool description with detailed nerdamer usage guide * Clarify nerdamer usage in sandbox tool description Updated the description of the sandbox tool to clarify usage of nerdamer. * ui: build nerdamer sandbox prelude from vendored source Replace the vendored all.min.js with the readable nerdamer-prime source and its two bundled deps (big-integer, decimal.js), licenses included. A vite plugin bundles and minifies them at build time with the upstream esbuild flags, exposed as virtual:nerdamer and imported lazily on first sandbox use. The vendors package.json pins commonjs so the project level type: module does not break esbuild format detection. The harness gains a CSP removing network egress from the worker, and browser tests cover the prelude, exact arithmetic, the fetch block and the timeout. Upstream snapshot: together-science/nerdamer-prime@1936145 * feat(ui): make symbolic math (nerdamer) a user-toggleable setting - Add SYMBOLIC_MATH_ENABLED setting key and registry entry (checkbox, default false) - Convert SANDBOX_TOOL_DEFINITION to buildSandboxToolDefinition(includeSymbolicMath) so the tool description includes/excludes nerdamer API docs dynamically - Cache sandbox harness per variant ('nerdamer' / 'plain') for instant toggle - Deprecate SANDBOX_TOOL_DEFINITION constant alias for backward compatibility - Update tools store to pass symbolic math config into tool definition * docs(ui): tell LLM to list nerdamer functions first, do not guess * test(ui): enable symbolic math in sandbox tests via settingsStore config * style(ui): fix formatting for tools.svelte.ts --------- Co-authored-by: Pascal <admin@serveurperso.com> * mtmd: use RAII for setting and resetting non-causal attention (#25723) * mtmd: use RAII for setting and resetting non-causal attention * mtmd: drop dependency on <optional> * mtmd: shorten class and variable names * hexagon: activation ops update (#25974) * hex-geglu: optimized all-in-one geglu microkernel * hex-geglu: enable non-contiguous src and strided DMA * hex-act: enable non-contiguous srs and strided DMA for rest of ACT ops * hex-act: generalize GLU per-thread functions via DEFINE_GLU_PER_THREAD macro * hexagon: move UNARY_SILU and UNARY_GELU to unary-ops * hex-act: replace the generic ops_context scratchpad usage with a local htp_vtcm_layout computation per act op. --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * CUDA: Improve NVFP4 W4A4 activation quantization (#25730) * Squash history before conflict-resolution during rebase on master WIP commit Add 32-byte loads, restore per-block amax Use nvfp4x4 intrinsic when available Fuse per-channel amax and quantization kernels Do pointer arithmetic only once on x Remove unnecessary ternary in the load We assert on host side that ne00 is 64-aligned Add back scale-search, but optimize it with intrinsics Code cleanup Make scale in MMQ-epilogue NVFP4-specific/restrictive for now Remove unneeded include, add comment Fix trailing whitespace Guard __builtin_align__(32) struct to NVIDIA Seems like HIP doesn't have this available, see https://github.com/ggml-org/llama.cpp/actions/runs/29438651734/job/87431623001 * compiler massaging to avoid unnecessary LDCs * kvalues_mxfp4 -> kvalues_nvfp4 in quantize_mmq_nvfp4 * Always pass in src1_scale.ptr * Extract ggml_cuda_is_aligned helper * ui: Add a "Default" option for the reasoning selector (#25846) * ui: add Default reasoning option that defers to the server The webui always injected enable_thinking, overriding the chat template default and the --reasoning flag, breaking models that reason unconditionally (e.g. Gemma 4 E4B) on a fresh client. Default sends nothing so the server decides, Off and effort levels force the value as before. All choices are remembered. Also remove the boolean thinking API from the conversations store and drop ChatFormReasoningEffortSubmenu.svelte (dead code). * ui: close the whole menu tree on reasoning level selection The reasoning levels were raw buttons inside the SubContent, so selecting one only closed the submenu via manual state while the root dropdown stayed open. DropdownMenu.Item closes the full tree on select like the sibling entries and brings native keyboard navigation. * ui: prevent the add menu tooltip from flashing when the dropdown closes * contrib: allow all AI-generated code in general (#26012) * conversion: fix non-MoE NomicBert GGUF conversion error (#25996) * metal : add f16 type support to leaky relu (#25981) * contrib: fix leftovers from the AI usage policy update (#26030) * args: refactor mlock/mmap/directio into load-mode (#20834) * args: overhaul mmap/mlock/dio into single arg Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update docs with llama-gen-docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: satisfy code quality Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * args: make the `+` sign an actual modifier now Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: general code clean up + comments Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: fix deprecated flags support Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: quick sanity check Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: sync llama-bench argument parsing Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * fix: bugfix variable behaviour + llama-bench lm column size Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: inverse commands should do the opposite instead of doing nothing Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix incorrect dash Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix missing modifiers for deprecated flags Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * llama: switch back to thread_local Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: switch back to single enum Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update arg docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: fix missing `mlock` from llama_load_mode_from_str + cleanup llama-bench Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * llama: fix mlock not activating Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: add deprecation warning when old and new flags are combined Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: cont add comment for todo in the future Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: sync with upstream Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: re-sync with upstream again Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * CUDA: fix external compilation of q1_0 MMQ (#25778) * hexagon: fix Windows crash when op_poll is enabled (#26029) * hexagon: further improved pipeline of the core bits (L2, DMA, MM, FA) (#26049) * hex-l2: use dirty ranges for flushing * hex-l2: simplify range based flush logic * hex-l2: optimize dirty range scans * hex-hvx: support for reduce_max_i32 * hex-mm: optimize fused MUL_MAT+ADD to use vtcm for bias when it fits * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation (round2) * hmx-mm: optimize output proc by tiling (col-chunking) * hex-fa: start the next q dmas a bit earlier * hex-fa: prefetch Q even earlier * hvx-fa: optimize softmax to keep things in hvx registers * hex-fa: hoist const register init in softmax loop * hmx-fa: kick off next-qkv DMAs before o-proc * hmx-fa: hoist various checks out of the inner loop * hmx-fa: adjust the cost model to better balance softmax work across hvx threads * hmx-fa: overlap diag rescale build with last HMX task * hmx-fa: optimize idx update in output proc * hmx-fa: unroll the softmax loops for improved perf * hmx-fa: overlap qk-dot with softmax, double-buffer p and s tiles * hex-trace: double the default number of trace entries * hex-trace: add trace events for opbatch and buffer mgmt * hex-trace: overhaul tracing to simplify runtime event handling and support opbatch stats * hex-trace: replace ascii timeline diagram with pipeline bubbles detector * hex-trace: handle missing start/stop events * hex-dma: always log stop/start trace events even for dummy dmas * hex-scripts: fix flake warnings * vendor: update subprocess.h (#26061) * cohere2 moe template parser: enforce JSON schema for text responses if a response schema is provided (#26018) * UI: Fix settings precedence, Factory < Admin (--ui-config-file) < Users (Settings panel) (#26002) * skill: create `add-new-model` and `code-review` (#26042) * skill: add new model * add common pitfalls * add code review skill * nits * add codeowners * add security review section * mention about skills in agents.md * add design review * exclude ggml-gh-bot * Apply suggestions from code review Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> * conversion-time weight modification * nits --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * opencl: do not treat NULL-mask flash attention as causal (#25771) * opencl: cache compiled cl_program binaries on disk (#26050) * HIP: remove rocWMMA FlashAttention (#26046) * llama: various bug fixes (#26051) * server: support "reasoning_effort": "none" in OAI API (#26045) * support "reasoning_effort": "none" in OAI API * handle reasoning.effort: "none" in OAI responses API * clarify non-"none" values of reasoning_effort have no effect * use json_value instead of body.at * ui: fix MCP server display name conflicts in tools lists (#26011) * ui: fix MCP server display name conflicts in tools lists Tool groups were keyed by display label so two servers reporting the same name broke the keyed each blocks and only one was visible. Key rendering, expand state and toggles by the stable server id instead, and suffix duplicate labels with a counter in config order. * ui: customizable MCP server display name with autofill Add a display name field to the MCP server form, add and edit alike. The custom name takes precedence over the server-reported one, so two servers reporting the same name can be told apart; clearing the field returns to the automatic label. In the add dialog a debounced preview handshake prefills the field with the server-reported name: a manual edit freezes the autofill, stale responses are discarded, failures stay silent, and an unedited prefill is not persisted so the label keeps following the server. * ui: fix recursive fetch passthrough in the client test setup The original fetch was captured inside beforeEach, where it is the previous test's spy since vi.spyOn returns the existing one, so the default passthrough recursed on itself for any URL outside the mocked set. Capture the real fetch once at module load. * model: add GLM 5.2 Indexer support (#25407) * Start building graph - reuse deepseek32 * Enable kv cache and rotation for glm_dsa architecture Just follow Deepseek 3.2 for now. * Reuse prev_top_k for "shared" indexer layers * GLM 5.2 uses LLAMA_ROPE_TYPE_NORM for the indexer. This is transformers' `apply_rotary_pos_emb_interleave` * Default indexer types to GLM pattern Previous converted GGUFs like https://huggingface.co/unsloth/GLM-5.2-GGUF write indexer weights to _all_ layers, even if they are only required for "full" types. This PR relies on a new key "%s.attention.indexer.types"; if absent, it will use the default GLM 5.2 schedule as defined in https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json#L26. Note that conversion is not saving this key yet. * Save indexer types to gguf, restore on load * Use ggml_lightning_indexer when cparams.fused_lid Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * GLM 5 and 5.1 use full indexers Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * Fix indentation * Ensure array is zero-filled * Prefer explicit std::fill * Assert prev_top_k exists for shared indexer --------- Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * ui: remove render effects (#26083) * ui: remove viewport fade in and smooth autoscroll bottom snap fadeInView mounted every message and markdown block at opacity 0 and relied on an IntersectionObserver to reveal it. When the observer never fires (blocks mounted offscreen during long agentic loops) the content stays invisible forever while still present in the DOM. Remove the action, its orphaned isElementInViewport util and all call sites: blocks now render visible immediately. AutoScrollController.scrollToBottom defaulted to behavior smooth and is invoked every 100 ms while streaming. Each tick restarts an easing animation toward a moving scrollHeight, producing a random elastic bump of a few pixels when the user reaches the bottom and autoscroll reengages. Default to instant scrolling; the user facing scroll down button keeps its smooth behavior. * ui: skip rendering of offscreen chat messages via content-visibility Apply content-visibility auto with contain-intrinsic-size to chat messages so the browser skips layout and paint for messages outside the viewport. The DOM stays complete: component state, find-in-page, text selection, and the mutation based autoscroll are unaffected, and browsers without support simply ignore the properties. * ui: remove conversation switch fade Switching conversations faded the message list out and in over 500 ms plus a 300 ms route delay, deferring the message refresh behind two requestAnimationFrame calls. Remove the fade, its navigation hooks and dead state, and refresh messages directly so switching is only bound by actual render time. * ui: describe present behavior in comments and drop unused parameter * ui: anchor the context gauge popup to the form with plain CSS The stats card was portaled to body and repositioned in script on every ancestor scroll event, trailing the page by one frame while streaming. Render it as an absolutely positioned sibling of the input box inside the already relative form, so nothing runs during scroll. The card sits just above the dial, centered on it and overlapping the textarea, from a single measurement of the dial center and top taken when it opens; the dial and the card share the same positioning frame, so the values stay exact while the card is open. Mouse pointers open on hover with a grace delay to reach the card, touch pointers toggle on tap, any press outside the card and the dial closes it, and Enter and Space toggle from the keyboard. The card lives outside the input box because its overflow-hidden and backdrop-filter would clip any positioned descendant. * ui: extract context gauge popup constants and relocate its state store Move the placement values and the close grace delay to lib/constants/context-gauge-popup.ts, matching the auto-scroll constants layout, and move the popup state module from the component folder to lib/stores where runes modules live in this codebase. * ui: declare the context gauge popup card ref as $state * ui: reduce per-token render cost when streaming (#26053) * performance harness - the empirical root Assisted-by: Claude Opus 4.8 * 210.36ms -> 2.67ms per streamed token Assisted-by: Claude Opus 4.8 * 11.58ms -> 0.62ms per streamed token Assisted-by: Claude Opus 4.8 * 22.02ms -> 3.33ms per streamed token Assisted-by: Claude Opus 4.8 * 3.07ms -> 1.36ms per streamed token at 40 messages Assisted-by: Claude Opus 4.8 --------- Co-authored-by: Zach Winter <dmtommy@icloud.com> * tests: synchronize save-load-state generation (#26056) * common : add support for multiple end sequences in the reasoning budget sampler (#25544) * common : extract trie/ac to a separate file * common : support multiple token sequences in the reasoning budget sampler * common/trie : return matched word index * common/trie : rename "word" to "pattern" * common/reasoning-budget : expose matched end sequence * common/sampling : replay end sequence when reasoning budget is done * cont : update to use multiple end sequences * cont : clean up * Update ggml/src/gguf.cpp : Defined virtual keyword for destructor of gguf_writer_base (#25867) Without a virtual destructor, deleting a derived object through a base-class pointer only invokes the base destructor, skipping the derived one. * vendor : update cpp-httplib to 0.51.0 (#26067) Signed-off-by: Adrien Gallouët <angt@huggingface.co> * server : add missing task parameters(adaptive_target, adaptive_decay) in generation_settings (#25830) These two parameters were overlooked when task parameters were being JSONized within `generation_settings` and have been added. A regression test has been added to prevent the problem from recurring and it passes. Fixes #25803 * server: add format arg to datetime tool (#26117) * common : skip empty implicit default preset (#25643) The INI parser creates an implicit default section for top-level metadata. After reserved keys such as `version` are skipped, that section can have no model options but was still added and exposed in router mode. Skip only the empty implicit default while preserving real default presets, named presets, and the global `[*]` settings. Signed-off-by: JS van Dijk <267467744+hogeheer499-commits@users.noreply.github.com> Co-authored-by: JS van Dijk <267467744+hogeheer499-commits@users.noreply.github.com> * hexagon: partial im2col support (#26007) * hexagon: add IM2COL op Add Hexagon IM2COL support targeting only patch-embedding convolutions. * hexagon: im2col refactor and cleanup * hex-im2col: instrument and update im2col. * hex-im2col: add local htp_vtcm_layout computation. * server: support MCP stdio (#26062) * move server_pipe to common * init impl * vendor: update subprocess.h * add server_mcp_stdio * stderr drain * server_mcp_transport * server_mcp_stdio is now framing-only, no json * internal/mcp-stdio: integration + tests + fixes (#26075) * server-mcp: harden transport and wire up the tool integration Builds on the transport/manager architecture (server_mcp_transport + server_pipe) with the hardening and integration the draft did not yet have. Hardening: * Reader and stderr pumps are polled (running-aware) instead of blocking on a read that only ends at EOF. subprocess_terminate() SIGKILLs only the direct child, so a grandchild the MCP server spawned that inherited the pipe would otherwise keep the write end open and hang teardown (both warmup shutdown at startup and process shutdown). The writer is likewise non-blocking + polled. * Windows: resolve the command through PATHEXT so "npx" (npm ships npx.cmd, never npx.exe) spawns, matching POSIX's PATH search; and enumerate the parent environment as UTF-8 (GetEnvironmentStringsW) instead of the active code page. * server_pipe gains an opt-in max_size (default unbounded, so the router's streaming use is unchanged); the MCP reply queue uses it so a server that streams unsolicited notifications between requests cannot grow it without bound. Integration: * --mcp-servers-config / --mcp-servers-json flags; enabling MCP restricts default CORS to localhost, same as --tools. * MCP tools are exposed through /tools (and chat-completions) as <server>_<tool>, skipping names that collide with a built-in or another MCP tool. * Manager lifecycle wired into llama_server(): warmup at start, shutdown() from the signal handler before the HTTP server drains, blocking teardown in clean_up(). * SIGPIPE ignored so a child dying mid-write yields EPIPE rather than killing us. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com> * server-mcp: add MCP test suite with grandchild deadlock regression test 21 tests over the /tools endpoint: tool discovery/invocation, timeouts, crash recovery and respawn cooldown, warmup partial failure, malformed and batched notification+response output, tool-definition shape, and prompt shutdown during a slow call. The last test spawns an MCP server that leaves a grandchild inheriting its stdout/stderr and asserts the server both starts and stops promptly. Verified it fails (5s SIGKILL fallback on a deadlocked reader-join) when the pump is made to ignore the running flag, and passes with the polled reader. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com> * clean up * clean up 2 * even stricter life cycle * nits * nits 2 --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> * fix some edge cases * fix last_error data race * fix response schema + docs * server: fix MCP zombie leak and timeout-induced transport teardown join_pumps() never reaped the child, leaking one zombie per spawn: call subprocess_join() before subprocess_destroy(). A per-call timeout permanently closed from_server and got a healthy transport evicted: add close_on_stop to server_pipe::read() and pass false from send_rpc(), where should_stop is a per-request deadline and a late reply is already skipped on id mismatch. Also drop the unreachable disconnect cancellation in server_mcp_tool::invoke(): support_stream is false, st is always null. (cherry picked from commit e6de1ec043174fd0570b1e60d47f06c7c19d620d) Assisted-by: Claude Opus 4.8 * server: make MCP test fixtures JSON-RPC 2.0 compliant Add the missing notification guard to mcp_malformed_server.py and mcp_burst_server.py (the latter treated id 0 as a notification and replied to unknown ones; its notification table is now unused). Return -32602 instead of -32601 for unknown tools: tools/call is a valid method, the tool name is the invalid parameter. Also fix the test module docstring: tools are named <server>_<tool>. (cherry picked from commit 74a08e8c311dabf3b49d06cc6d754b0097ae7a38) Assisted-by: Claude Opus 4.8 --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Pascal <admin@serveurperso.com> * common : use-after-free when loading LoRA adapter fails (#25611) * ggml-webgpu: Fix WASM compilation with OpenMP (#25943) * Fix emscripten compilation with openmp * Separate wasm job to its own workflow * Add flags necessary for newer emsdk * Just disable openmp * Update triggers * ui: fix context gauge card regressions and land at the conversation end (#26099) The context gauge card starts monitoring like the dial does, because its own processing state instance only follows the live stream while its monitoring flag is set. It also gets back the text-sm and ring classes the removed hover card wrapper used to inject, which restores its layout. Routing to a conversation now lands at the bottom instantly and keeps the pin one frame at a time until the page height settles, since content-visibility size realizations and syntax highlight passes grow the page without DOM mutations. * opencl: fix fused RMS norm mul view offset (#26085) * model: Add MiniMax-M3 (MSA: MiniMax Sparse Attention) support (#24908) * Add preliminary MiniMax-M3 support Text-only port that re-uses existing components: MiniMax-M2 style GQA with per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and routed/shared experts, and swigluoai activation. Sparse attention is not yet supported (dense fallback); vision tower and MTP heads are dropped. * MiniMax-M3 vision tower (mmproj + clip graph) * Delete m3_vision_ref.py * Update clip.cpp * MSA * Update constants.py * Update minimax.py * Cache creation. Working withotu flash attention * Added flash attention for sparse layers * Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx * Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking * Implement sparse attention calc out of stock ops. * Fix a cache allocation and cont issue * Fixed -fa auto crash, flagged debug spots * Delete vocab.json * Delete model.safetensors.index.json * Delete generation_config.json * Delete Minimax directory * Handled multi stream case to fall back on Dense Attention * Development scaffolding cleanup. No functional change to the decode or 4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the selection-parity validation. * Remove redundant comment from minimax-m3.cpp * Changed 3 Gelu Ops for vision into Gelu_erf ops * Assert that n_kv is multiple of 128 * Rename MSA index tensors to indexer convention Note: All GGUFs generated before this change will need to be regenerated. * Fix incorrect Assert * Review driven changes (#3) * Remove comment from conversion minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespaces from constants.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Tighten comment in minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * inherit MiniMax-M3 from MiniMax-M2 * drop dead text_config fallbacks * Add indexer writer methods * Reuse LLM_FFN_SWIGLU_OAI_MOE * Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention * Fix conversion error /gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-kv-cache.cpp Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove Whitespace in Update src/llama-model.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * remove multimodal code upon maintainer request. Will be made as a separate PR * Whitespace clean in tensor_mapping.py * Log cache size on launch, block ctx shift, support prompt caching Log indexer cache size on launch Disallow ctx shift Support prompt caching * Update minimax-m3.cpp * Optimize implementation, add multi stream support. Fully rewrote minimax-m3.cpp for speed and buffer size gains: Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3] Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill Decode: ~25 nodes/layer vs ~50, no per-group concats/conts Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token) In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support. * set default cache type to F32 * Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in * remove F16 downcasts in MSA attention, force F32 indexer score accum * Add Minimax eos to llama vocab * Guard edge case where idx cache can become stale after a tail trim * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Update llama-kv-cache.cpp * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Review driven changes * style fix * indexer hparams are required * fix tests * fix lint --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> * mtmd: add GLM-5.2-Vision (#26126) Co-authored-by: Eric Hartford <eric@quixi.ai> * common: add `subproc.h` wrapper, disabled on android/ios (#26102) * add common/subproc.h|cpp * add compile flag LLAMA_SUBPROCESS * disabled by default on android and ios * test-jinja: use common subproc * mtmd: disable video if subproc is not set * disable subproc on wasm * make is_created atomic * migrate server-mcp * ui: detect the conversation import format from file contents (#26121) * ui: detect the conversation import format from file contents iOS resolves every accept entry to a UTI and has none for ".jsonl", so the picker greyed out exported conversations. Drop the accept filter and pick the parser from the file contents: ZIP magic bytes, then a first "session" record for JSONL, otherwise the legacy JSON format. Also remove the unused importConversations() picker and an orphan doc comment, and cover each format with unit tests. * ui: report what a conversation import actually wrote The import summary echoed the selection back, so re-importing conversations already in the database claimed success while nothing was written and only a console warning said otherwise. Return the imported and skipped conversations from the database layer, list the written ones in the summary, and count the rest in a toast. * ui: name the literals of the JSONL conversation format Introduce SessionRecordType and SESSION_HARNESS, and reuse the existing NEWLINE constant, so the record format lives in one place. This also covers the writer side, which predates the import path under review and carried the same literals: an enum stated by the reader alone lets the two sides drift. Values are unchanged, so an export stays byte identical. * Keep Minimax's indexer tensors at F32 for speed and accuracy (#26144) * Keep Minimax's indexer tensors at F32 for speed and accuracy * name -> new_name * ui: fix system message edit box not expanding to fit content (#26006) The in-conversation system-message edit textarea had a fixed min-height and no auto-resize, so long messages were crammed to 2 lines. Reused existing autoResizeTextarea helper on input and when the editor opens, and added max-height to cap growth. * mtmd: fix android build (#26150) * mtmd: Add Vision Support for Minimax-M3 (#25113) * Add preliminary MiniMax-M3 support Text-only port that re-uses existing components: MiniMax-M2 style GQA with per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and routed/shared experts, and swigluoai activation. Sparse attention is not yet supported (dense fallback); vision tower and MTP heads are dropped. * MiniMax-M3 vision tower (mmproj + clip graph) * Delete m3_vision_ref.py * Update clip.cpp * MSA * Update constants.py * Update minimax.py * Cache creation. Working withotu flash attention * Added flash attention for sparse layers * Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx * Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking * Implement sparse attention calc out of stock ops. * Fix a cache allocation and cont issue * Fixed -fa auto crash, flagged debug spots * Delete vocab.json * Delete model.safetensors.index.json * Delete generation_config.json * Delete Minimax directory * Handled multi stream case to fall back on Dense Attention * Development scaffolding cleanup. No functional change to the decode or 4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the selection-parity validation. * Remove redundant comment from minimax-m3.cpp * Changed 3 Gelu Ops for vision into Gelu_erf ops * Assert that n_kv is multiple of 128 * Rename MSA index tensors to indexer convention Note: All GGUFs generated before this change will need to be regenerated. * Fix incorrect Assert * Review driven changes (#3) * Remove comment from conversion minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespaces from constants.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Tighten comment in minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * inherit MiniMax-M3 from MiniMax-M2 * drop dead text_config fallbacks * Add indexer writer methods * Reuse LLM_FFN_SWIGLU_OAI_MOE * Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention * Fix conversion error /gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-kv-cache.cpp Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove Whitespace in Update src/llama-model.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update minimax_m3.cpp Rewrite code comment based on feedback and to better reflect the actual architecture, and reuse existing build_vit * Rename minimax_m3.cpp to minimax-m3.cpp * Update CMakeLists.txt * Remove debug code from clip.cpp * Update clip.cpp * Update comments in tools/mtmd/models/minimax-m3.cpp * Permute Q/K at conversion, drop precomputed sin/cos * Log cache size on launch, block ctx shift, support prompt caching Log indexer cache size on launch Disallow ctx shift Support prompt caching * Update minimax-m3.cpp * Optimize implementation, add multi stream support. Fully rewrote minimax-m3.cpp for speed and buffer size gains: Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3] Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill Decode: ~25 nodes/layer vs ~50, no per-group concats/conts Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token) In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support. * set default cache type to F32 * Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in * remove F16 downcasts in MSA attention, force F32 indexer score accum * Add Minimax eos to llama vocab * Guard edge case where idx cache can become stale after a tail trim * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Update llama-kv-cache.cpp * Update llama-kv-cache.h * Change resize Pad to none, resize alg to Bicubic Pillow * Review driven changes * Update llama-kv-cache.cpp * rm unrotated pos_t * fused rope w + pad * rename merge --> merger for consistency * add review skill for mtmd * graph should use hparams n_merge * fix lint --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> * ui: Fix symbolic math tool JS sandbox prompt (#26131) * Update sandbox.ts * Update sandbox.ts * Update sandbox.ts * Update sandbox.ts * Revise nerdamer description in sandbox constants Updated NERDAMER_DESCRIPTION to clarify usage and warnings. * server + ui: fix stream routes for model names containing a slash (#26137) * server + ui: refactor resumable stream routes to query string conv_id The conversation id can embed a model name containing slashes (ggml-org/...) in router mode, which the decoded path splits before the :conv_id param is captured, so stop and resume never matched the session. Move the id to the conv_id query string on the public routes and on the internal router -> child hop, where slashes survive encoding. Handlers are unchanged since query and path params land in the same map. Add a regression test with a slashed model name. * server: move stream route docs to server-stream.h Address review: ngxson wants the main server.cpp registration code kept clean and simple, with route-level explanations living in the header. Move the query string rationale and the lookup ownership note next to the handler declarations in server-stream.h, and shorten the wiring comment to a pointer. * server: cancel a pending request when its stream is stopped during model load The conversation was registered in the conv map only after the blocking autoload wait, so a stop issued while the model loaded found nothing to cancel and the request went on to generate an orphan once the load ended. Register the conversation before the wait and give the entry a ticket: a stop erases the entry, and the parked request checks its ticket after the wait and aborts with 400 instead of starting. A newer request on the same conversation replaces the entry, so only the stopped request is cancelled. Add a regression test that stops during the load window. * server + ui: resume a stream after a page reload during model load A pending request died with the client socket when the page was reloaded while its model was loading, so no session ever existed and the conversation had nothing to recover. A session request that waited for a load now detaches from the client socket and reaches the child regardless, the session buffer receives the generation, and the resume route answers 503 while the owner is loading so the client retries instead of dropping its state. The WebUI persists the pending stream at send time, quietly polls on 503, and attaches once the session exists. Add a regression test that drops the client during the load window. * ui: show the model load progress again after a page refresh The resume wait was invisible, so a conversation refreshed while its model was loading showed nothing until the first byte. On a 503 from the resume probe, mark the conversation as loading again so the assistant row persisted at send time renders the processing info, and target the model frozen in the persisted stream state for the progress, since the row has no model yet and the dropdown may not be restored. * fix CI * fix CI bis * args: add `-lm mlock` where it mlocks but doesnt mmap (#26135) * arg: add `-lm mlock` where it mlocks but doesnt mmap Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: rm unwanted docs changes Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: revert auto-formatting Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix automated review point 3 Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: revert the meaning of --mlock to non-mmap'ed mlock Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: remove extra changes from `llama-gen-docs` Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * ggml-cpu: Enable BF16 tiled gemm optimization on PowerPC (#26068) * docs: add exception about weight folding (#26168) * docs: add exception about weight folding * add example * common: fix explicit -md precedence over draft sidecar resolution (#26165) * common: fix explicit -md precedence over draft sidecar resolution Follow-up of #25955, an explicit --model-draft file given with -hfd was silently overridden by the sidecar resolution of the draft repo, and its path was never resolved to a local file. An explicit draft file selection now disables the sidecar resolution, so the manual CLI configuration wins over the automatic one. * common: apply the -hfd tag to the sidecar resolution The sidecar selection was anchored on the primary of the draft plan, so a tag without a matching full model aborted the whole plan, and the sidecar quant silently followed the default model pick. The tag now anchors the sidecar directly: exact tag match first, then closest quant to the tag, and a requested sidecar resolves even when no full model matches the tag. A wired draft sidecar also counts as an explicit draft, so the main plan no longer downloads a second one. * common: promote speculative load logs from trace to info Show the loaded draft model and the MTP draft context at the default verbosity, for consistency with the mmproj and primary logs. Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * tests : remove unnecessary sync in test-save-load-state (#26166) * ggml : adjust logic for offloading ops to weight's backend (#25832) * ggml : adjust logic for offloading ops to weight's backend * llama : dsv4 graph fixes * sycl(build): parallelize ocloc invocations (#25903) * fit : count nextn (MTP) blocks in n_gpu_layers so front layers stay on GPU (#26177) * model: Add support for Nanbeige4.2 (#25994) * support nanbeige4.2 model * fix * fix flake8 Lint check * fix loop bound check and drop redundant head_dim --------- Co-authored-by: root <lizongqiang@kanzhun.com> * common : add common_print_available_devices() (#26170) Signed-off-by: Adrien Gallouët <angt@huggingface.co> * mtmd: support MiMo-V2.5 audio input (RVQ-based model) (#26190) * gguf converter for mimo audio * fix conv * cpp impl * nits * nits 2 * Disable -ffast-math on HIP (#25495) * contrib : add guideline about the "merge ready" label (#26178) * contrib : add guideline about the "merge ready" label * cont : add ref [no ci] * spec: add eagle3-v3 support for gpt-oss model (#25794) * ggml-metal: FWHT kernel for metal backend (#25924) * metal fwht wip * shape guard and formatting * formatting * Formatting and typos Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * fix narrowing issue Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * cont : minor style --------- Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * server : add extra trace log for prompt similarity (#26218) * sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path (#25880) * sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path The scale was uploaded with an async memcpy sourced from a stack local. On the in-order queue that copy is ordered behind the K/V staging kernels; once n_kv is large enough (>= ~26k observed on Arc Pro B70) the staging outlives the host stack frame and the copy reads recycled memory, feeding the SDPA a garbage scale. Output then collapses to a single repeated token and the KV cache is poisoned for the rest of the session. Short contexts win the race by accident, and test-backend-ops caps FLASH_ATTN_EXT at kv=1024, which is why CI never caught it. The previous device_count > 1 wait_and_throw() gate (and reverting it, PR #25741) fixes the symptom only by keeping the frame alive across the copy at the cost of a host sync on every FA call. Fix: cache one device scalar per (device, value) -- the scale is constant per model -- and upload it synchronously once. The single-device fast path (no per-call host sync) is then safe: every device-side hazard already serializes on the in-order queue. The multi-GPU conservative wait is kept unchanged. Also: - GGML_SYCL_FA_ONEDNN_MAX_KV env (0 = unlimited): optional n_kv ceiling that routes very long sequences to the native FA kernel. - test-backend-ops: FLASH_ATTN_EXT F16 cases up to kv=65536 (Qwen3.6-27B geometry hsk=hsv=256 GQA 6, and hsk=128 GQA 4), closing the kv=1024 blind spot. Note the race itself needs a live multi-op pipeline to reproduce; single-op runs pass even on broken builds. Verified on Arc Pro B70 (bmg_g31), Qwen3.6-27B Q4_K, -c 131072: output byte-identical at temp 0 to the native FA path through 32k-deep prefill, with prefill depth-flat at 820-840 t/s (vs 340-350 native at 32k depth). Assisted-by: Claude Fable 5 * sycl: handle GGML_SYCL_FA_ONEDNN_MAX_KV like the other runtime env vars and document it Review feedback on #25880: - read the variable once at backend init into g_ggml_sycl_fa_onednn_max_kv via ggml_sycl_get_env, and print it in the startup env listing (-lv 4 shows it) - document GGML_SYCL_FA_ONEDNN and GGML_SYCL_FA_ONEDNN_MAX_KV in the SYCL.md runtime table Also trim the added FLASH_ATTN_EXT cases to kv={4096,16384}: the 32768/65536 shapes exceed the legacy NMSE threshold on both the oneDNN and native kernels (long-sequence fp16 accumulation drift, present before this PR) and would fail CI for an unrelated reason. Assisted-by: Claude Fable 5 * sycl: clarify GGML_SYCL_FA_ONEDNN_MAX_KV default is disabled Assisted-by: Claude Fable 5 * sycl: state default behavior of GGML_SYCL_FA_ONEDNN_MAX_KV explicitly Assisted-by: Claude Fable 5 * Update ggml/src/ggml-sycl/fattn-onednn.cpp Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> * sycl: write the SDPA scale from a kernel instead of caching it The per-(device, value) scale cache was a function-local static unordered_map with no synchronization, so concurrent backend instances could access and rehash it at the same time. Write the scalar with a single_task instead. The value is captured into the command, so no host memory has to outlive the call -- which is what the use-after-return fix needed in the first place. That removes the shared container, the leaked device allocation and the string key, and it also closes the remaining async-memcpy-from-a-stack-local on the first flash-attention call. Ordering does not rely on timing: the queue is created with sycl::property::queue::in_order and the dnnl stream wraps that same queue, so the write completes before the SDPA reads the scalar. The multi-GPU wait_and_throw() branch is unchanged. Also drop the <cstdlib> include, which is unused. Assisted-by: Claude Opus 5 --------- Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> * common/chat: add specialized minimax m3 parser (#26210) * spec: add DSpark speculative decoding (#25173) * spec: add DSpark speculative decoding DSpark (DeepSpec, 2026) on top of the merged DFlash drafter. It reuses the DFlash encoder/decoder graph, target feature extraction and KV-cache injection, and the verify/accept path unchanged; the draft model is a new "dspark" arch adding a low-rank Markov head (markov_w1/w2) and an optional (unused here) confidence head. No new public APIs. The proposal is the only change: the block is anchor-first (position 0 already predicts the first draft) and the decoder graph applies a semi-autoregressive, previous-token conditioned logit bias in-graph, chained per block position: logits'(i) = logits(i) + markov_w2 . markov_w1[prev(i)] prev(0) = the block's anchor token, prev(i>0) = argmax(logits'(i-1)) vectorized across all blocks in the batch; the anchors are fed through a dedicated graph input (token 0 of every block). Greedy stays lossless (verify unchanged, same as DFlash). - new arch "dspark" (llama_model_dspark : llama_model_dflash, reuses the graph, loads the markov/confidence tensors; shares the target's embed/lm_head). - Qwen3DSparkModel converter. - new spec type "draft-dspark" (common_speculative_impl_draft_dspark : common_speculative_impl_draft_dflash, overrides draft() only: submits whole anchor-first blocks and greedily reads back the biased logits). * spec: read draft block size in the dflash impl * docs: add DSpark section to speculative.md * spec: keep dspark block size read in the dspark impl * dspark : add TODOs for incomplete parts - confidence head is loaded but not used yet - confidence-scheduled prefix pruning is not implemented - the in-graph Markov chain…

…tion-Id (ggml-org#25760) * server: return 400 instead of 500 on validation error with X-Conversation-Id set_req() attaches the spipe as soon as the header is present, before the request body is parsed. When params validation throws, set_next() never runs and next_orig stays empty, so on_complete() called it and crashed with std::bad_function_call, turning the prepared 400 JSON into a generic 500. on_complete() now treats an empty next_orig as "streaming never started" and evicts the session installed by set_req(), so a failed request leaves nothing behind for discovery or replay. This also covers valid requests that carry the header but do not stream, which previously left an empty finalized session in the map until the GC TTL. * ui: do not send the backend_sampling placeholder On a fresh profile the syncable settings hold the empty string placeholder meaning "let the server decide". Every neighbor field goes through the hasValue() guard that filters it, except backend_sampling, which sent the placeholder verbatim and made every default settings completion fail validation. Guard the field with hasValue() like its neighbors. hasValue(false) is true, so an explicit false still reaches the server and the intent of ggml-org#18781 (send both true and false) is preserved. Only the placeholder is filtered.

* webui : send both backend_sampling == false/true * feat: Parameter sync --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

ggerganov requested a review from allozaur as a code owner January 12, 2026 13:22

loci-dev mentioned this pull request Jan 12, 2026

UPSTREAM PR #18781: webui : send both backend_sampling == false/true auroralabs-loci/llama.cpp#896

Open

github-actions Bot added examples server labels Jan 12, 2026

allozaur added the server/webui label Mar 17, 2026

ggerganov mentioned this pull request Mar 31, 2026

arg: fix incorrect default for backend sampling #21210

Closed

webui : send both backend_sampling == false/true

684818b

allozaur force-pushed the gg/webui-bs-send-false branch from c1c42f1 to 684818b Compare April 8, 2026 13:39

allozaur requested a review from a team as a code owner April 8, 2026 13:39

allozaur removed examples server labels Apr 8, 2026

feat: Parameter sync

12929c2

allozaur approved these changes Apr 8, 2026

View reviewed changes

allozaur requested a review from ServeurpersoCom April 8, 2026 13:49

ServeurpersoCom approved these changes Apr 8, 2026

View reviewed changes

allozaur merged commit 4a05e0c into master Apr 8, 2026
5 of 6 checks passed

allozaur deleted the gg/webui-bs-send-false branch April 8, 2026 14:37

Navigation Menu

Uh oh!

webui : send both backend_sampling == false/true - #18781

webui : send both backend_sampling == false/true#18781
allozaur merged 2 commits into
masterfrom
gg/webui-bs-send-false

ggerganov commented Jan 12, 2026

Uh oh!

allozaur commented Jan 12, 2026

Uh oh!

allozaur left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ggerganov commented Jan 12, 2026

Uh oh!

allozaur commented Jan 12, 2026

Uh oh!

allozaur left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants