Feature Request: server in Router Mode: Multiple presets for the same model (loaded once)

[rtpt-erikgeiser]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

It would be great if we could have multiple presets in router that reference the same model as in the same model process where the model is only loaded once. Of course, that does not work for all settings but it would be specifically useful to be able to offer a reasoning and a non-reasoning version with different sampling parameters for a single model that is only loaded once. This means the router would only have to override request parameters when forwarding the request to a model runner process.

There is already an issue to implement such presets in the web UI (#22523), but I think it would be better to add them to the server/router so that other apps and harnesses can utilize them, because many harnesses do not allow overriding request parameters.

Motivation

Most users are likely constrained by RAM, and being able to offer a reasoning and a non-reasoning model (for apps that don't support request parameter overrides) without having to load the same model twice would be great. Also being able to offer multiple appropriately named presets with different sampling parameters (e.g. for coding vs creative writing) would make for a good user experience.

Possible Implementation

The config could look like this:

[Qwen3.6-35B-A3B-Instruct]
model = /path/to/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf
ctx-size = 1048576
parallel = 4
mmproj = /path/to/mmproj-F16.gguf
load-on-startup = on
temp = 0.7
top-p = 0.8
min-p = 0
top-k = 20
presence-penalty = 1.5
repeat-penalty = 1.0
reasoning = off

[Qwen3.6-35B-A3B-Thinking]
template = Qwen3.6-35B-A3B-Instruct  # <- reference the original preset
# override parameters of the original preset:
temp = 0.6
top-p = 0.95
min-p = 0
top-k = 20
presence-penalty = 0.0
repeat-penalty = 1.0
reasoning = on
chat-template-kwargs = {"preserve_thinking": true}
# parallel = 2 <- this would not work

Without having looked into the current implementation and as far as I understand the current router mode, it could be possible to check if the requested model is based on a template and then forward the request to the preset configured in template while overriding the specified parameters (unless the user already overrode them in the request).

[ngxson]

IIRC there was a discussion in the past about extend a preset in a OOP inheritance way:

[model0]

[model1]
extend = model0

[model2]
extend = model1

But IIRC the proposed implementation was too buggy and at the same time over-complicated.

[rtpt-erikgeiser]

The specific implementation was too buggy and over-complicated, or do you think implementing this would be over-complicated in general?

[ngxson]

the specific implementation is over-complicated (I don't remember which PR)

this feature itself is not complicated, inheritance is a straight-forward concept

[ngxson]

note that the feature I'm mentioning about has nothing to do with web UI, I think there is a mix up between preset (a server-specific feature) vs web UI feature in the PR description

[ngxson]

It would be great if we could have multiple presets in router that reference the same model as in the same model process where the model is only loaded once.

multi-preset, same model, same process --> complicated, we don't current accept such feature

multi-preset, same model, one preset per process --> very simple to implement

[rtpt-erikgeiser]

• One preset per process would simply unclutter the config.
• Multiple presets for the same model and the same process would allow for massive VRAM saving and this is what I am proposing. I'm sure that wanting to serve a model with and without reasoning is a very common usecase and it is incredibly wasteful when the same model has to be loaded twice (if VRAM allows this at all).

What I had in mind was something like the following:

• Request comes in, model name refers to a non-extended preset -> forward request to process
• Request comes in, model name refers to an extended preset -> override parameters in request (if they are not already explicitly set) -> forward request to the process of the non-extended preset

[ngxson]

an extended preset -> override parameters in request

this is the most complicated and fragile part. if the derived preset has something like different n_ctx or n_parallel, you have no choice but to fallback to one preset per process

---

otherwise, it should probably better to be implemented on top of the alias system. but in either ways, this has nothing to do with preset or router mode

for example, note that we are taking about single-model mode (no preset)

llama-server -m ...
  --alias "model1,model2"
  --alias-override 'model1={"temperature":0,...}'

where the override is a plain simple JSON that will get overlaid on top of the incoming request.

then get defined in preset:

model = ....
alias = model1,model2
alias-override = model1={"temperature":0,...}

but obviously it's not as simple as it looks, basically each endpoint may need a specific override, and then have to define nesting override

if you have a more simple and fool-proof solution, it will be welcomed. but IMO with the current state of the code, it will be quite fragile and may require a big refactoring first.

[CatKasha]

can i open separate issue for "copy preset args to another preset" in models-preset ini? i thought about this a week ago but i got lazy
and for second feature i saw something similar here #20004

[rtpt-erikgeiser]

this is the most complicated and fragile part. if the derived preset has something like different n_ctx or n_parallel, you have no choice but to fallback to one preset per process

Yes I know, this would work only for the parameters that can now already be specified in the request which excludes (n_ctx or n_parallel).

Personally, I would try to avoid something like alias-override = model1={"temperature":0,...} because mixing language (ini and JSON) is always a bit awkward. But that's just the config format, for the implementation I agree that the server should overlay a JSON over the request. Ideally in a way that the code only overrides parameters that were not explicitly set in the request already.

[MaxKruse]

Personally, I would try to avoid something like alias-override = model1={"temperature":0,...} because mixing language (ini and JSON) is always a bit awkward. But that's just the config format, [...]

on that part, that already happens inherently because of the kwargs string needing to be json. I feel the preset.ini system as a whole is working, but rather fragile in its design as of now.

I fear even discussing about changing the preset system to use a well-defined json would be out of scope/too much?

Example

I know i know, too much and out of scope and all that, and extra complexity that is probably not worth it to keep in sync with other parts of the codebase. Just a throwaway idea

{
    "$schema": "http://github.com/ggml/server-router-schema/somewhere.yaml",
    "router": {
        "globalConfig": {
            "parallel": 4,
            "spec-type": "draft-mtp,ngram-mod,ngram-map-k",
            // ... and so forth, akin to [*] in the ini
        },
        "models": [
            {
                "baseName": "Qwen3.5 35B", // Would result in "Qwen3.5 35B" for the default preset
                "config": {
                    "ctx-size": 8192,
                    "model": "/mnt/models/qwen3.5-35b/qwen3.5-35b.gguf",
                    "temperature": 0.7,
                    "reasoning": "on"
                },
                "presets": [
                    {
                        "name": "Thinking", // Would Result in "Qwen3.5 35B - Thinking"
                        "config": { // Would keep the same model in memory as the base model, since it doesn't specify any load-time parameters
                            "temperature": 0.6,
                            "top_p": 0.9,
                            "reasoning": "off" // unsure if thats actually a load-time parameter or not?
                        }
                    },
                    {
                        "name": "Creative", // Would Result in "Qwen3.5 35B - Creative"
                        "config": { // Would keep the same model in memory as "Qwen3.5 35B" preset
                            "temperature": 1.0,
                            "top_p": 1.0
                        }
                    },
                    {
                        "name": "Vision", // Would Result in "Qwen3.5 35B - Vision"
                        "config": {
                            "mmproj-file": "/mnt/models/qwen3.5-35b/qwen3.5-35b-mmproj.gguf", // Using this preset, it would attempt to reload the model, since the mmproj file is a load-time parameter
                            "temperature": 0.5,
                            "top_p": 0.6
                        }
                    }
                ]      
            }
        ]
    }
}

[ngxson]

I don't see how the preset.ini system is fragile, it's just a way to write the arguments in an easier-to-read form. When a new arg is added to the project, there is 0 code required to make it compat with the existing ini system. I'm pretty sure you cannot do the same with a json-based system.

If you don't want to mix between ini and json, just save your json to a file, for ex:

• arg form: --alias-override-file file.json
• ini form: alias-override-file = file.json

But eventually none can solve the problem of having different overlay for each endpoint. I do believe that a refactoring is needed, and a discussion on what / where to refactor need to be brought up first.

[ihatemakinganaccount]

I have a different proposition. While (on CPU) I can have multiple presets with the weights not getting duplicated in RAM, everything else like the KV cache gets doubled, since each is its own process. This may be desirable if it's what you want, but not if you just want to switch reasoning or sampler settings quickly.

Instead, I propose to have some runtime presets handled by the frontend. These would be derived from one alias (or globally), but contain only settings which can be changed on runtime, like sampler settings and json injections.

These could be defined in the ini with the alias, but not parsed by the server, instead just passed on to the frontend, which could then include them in the models dropdown, or with an additional presets dropdown, or handle them in whatever other way.

Structured json would work nicely for this, something like

[Model alias]
m = model.gguf
c = 65536
runtime_presets = {
"Coding": {
"temperature": 0.3,
"min_p": 0.1,
"top_k": 20
},
"Funny": {
"temperature": 1.2,
"top_k": 50
},
"Nothink": {
"temperature": 0.5,
"json_inject": "{"chat_template_kwargs": {"enable_thinking": false}"
}
}

Or whatever format would make sense. (On one line probably. My backticks get garbled. You get the idea.)

Aliases would still define the settings which have to be fixed at model load time like n_ctx, but the runtime presets could be switched on the fly, while preserving context and having just one copy of the weights in memory.

This should be much easier to do, since almost nothing needs to change on the server side, and the frontend expansion is a one-time thing.

An additional benefit would be that this way you (or the server owner) could bring some runtime settings into the frontend to change easily, even if the UI doesn't expose them directly, like mirostat or reasoning. At least as long as the frontend doesn't discard unknkown parameters and is "willing" to pass them on back in the API call.

Thoughts, anyone? (I'm unable to get to discussions atm.)

[sgayda2]

I think this would be pretty awesome to add, as an example, llama-swap supports this using its set params field in the config. Allowing it to override the request instead of reloading the model.

Relevant config info:
https://github.com/mostlygeek/llama-swap/blob/26bb17e57e8eb7fd8a4cea2294439ac8cb89fc80/config.example.yaml#L230