Skip to content

Commit 4150da9

Browse files
authored
examples : add --kv-unified to batched example (#18774)
This commit adds the --kv-unified flag to the batched example. This flag is currently specified in the README.md as required, but is currently not available as a command line option for the batched example. The motivation for this is that specifying this flag as the README instructs, will lead to an error about the flag not being recognized, and without this option the example fail with the following error: ```console split_equal: sequential split is not supported when there are coupled sequences in the input batch (you may need to use the -kvu flag) decode: failed to find a memory slot for batch of size 4 main: llama_decode() failed ```
1 parent 8e2da77 commit 4150da9

3 files changed

Lines changed: 3 additions & 2 deletions

File tree

common/arg.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1295,7 +1295,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
12951295
[](common_params & params) {
12961296
params.kv_unified = true;
12971297
}
1298-
).set_env("LLAMA_ARG_KV_UNIFIED").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_PERPLEXITY}));
1298+
).set_env("LLAMA_ARG_KV_UNIFIED").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_PERPLEXITY, LLAMA_EXAMPLE_BATCHED}));
12991299
add_opt(common_arg(
13001300
{"--context-shift"},
13011301
{"--no-context-shift"},

common/common.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ int32_t cpu_get_num_math();
8080
//
8181

8282
enum llama_example {
83+
LLAMA_EXAMPLE_BATCHED,
8384
LLAMA_EXAMPLE_DEBUG,
8485
LLAMA_EXAMPLE_COMMON,
8586
LLAMA_EXAMPLE_SPECULATIVE,

examples/batched/batched.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ int main(int argc, char ** argv) {
2121
params.prompt = "Hello my name is";
2222
params.n_predict = 32;
2323

24-
if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_COMMON, print_usage)) {
24+
if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_BATCHED, print_usage)) {
2525
return 1;
2626
}
2727

0 commit comments

Comments
 (0)