Compile bug: s390x/zDNN linking errors

[owainkenwayucl]

Git commit

[uccaoke@vind2 llama.cpp]$ git rev-parse HEAD
f44f793

Operating systems

Linux

GGML backends

zDNN

Problem description & steps to reproduce

I'm trying to compile llama.cpp on an IBM LinuxONE system running AlmaLinux 9.x with zDNN installed (from DNF). I've successfully compiled + linked other software with zDNN but when I try to build Llama.cpp I get the following link errors when trying to link libggml-zdnn.so into executables:

[ 28%] Linking CXX executable ../bin/test-tokenizer-0
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_get_nnpa_max_dim_idx_size()'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_init_pre_transformed_desc(zdnn_data_layouts, zdnn_data_types, zdnn_tensor_desc*, ...)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_is_nnpa_installed()'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_matmul_transpose_op(zdnn_ztensor const*, zdnn_ztensor const*, zdnn_ztensor const*, bool, bool, zdnn_matmul_ops, zdnn_ztensor*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_generate_transformed_desc(zdnn_tensor_desc const*, zdnn_tensor_desc*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_free_ztensor_buffer(zdnn_ztensor const*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_init_ztensor_with_malloc(zdnn_tensor_desc*, zdnn_tensor_desc*, zdnn_ztensor*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_transform_ztensor(zdnn_ztensor*, ...)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_transform_origtensor(zdnn_ztensor const*, void*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_is_nnpa_parmblk_fmt_installed(int, ...)'
collect2: error: ld returned 1 exit status
gmake[2]: *** [tests/CMakeFiles/test-tokenizer-0.dir/build.make:104: bin/test-tokenizer-0] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1917: tests/CMakeFiles/test-tokenizer-0.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

The cmake configure line I'm using is:

cmake -S . -B build_zdnn -DCMAKE_BUILD_TYPE=Release -DGGML_ZDNN=ON

zDNN is in /usr/lib64/libzdnn.so (and .a etc), the -devel packages for it are installed.

[uccaoke@vind2 llama.cpp]$ rpm -qa | grep zdnn
libzdnn-1.1.1-1.el9.s390x
libzdnn-devel-1.1.1-1.el9.s390x
libzdnn-static-1.1.1-1.el9.s390x

I suspect I need to modify some of the cmake stuff so that it explicitly links the libzdnn.so into executables but I'm not a cmake expert so not sure where to start? Any pointers would be appreciated!

First Bad Commit

No response

Compile command

cmake -S . -B build_zdnn -DCMAKE_BUILD_TYPE=Release -DGGML_ZDNN=ON
cmake --build build_zdnn --config Release

Relevant log output

[ 28%] Linking CXX executable ../bin/test-tokenizer-0
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_get_nnpa_max_dim_idx_size()'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_init_pre_transformed_desc(zdnn_data_layouts, zdnn_data_types, zdnn_tensor_desc*, ...)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_is_nnpa_installed()'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_matmul_transpose_op(zdnn_ztensor const*, zdnn_ztensor const*, zdnn_ztensor const*, bool, bool, zdnn_matmul_ops, zdnn_ztensor*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_generate_transformed_desc(zdnn_tensor_desc const*, zdnn_tensor_desc*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_free_ztensor_buffer(zdnn_ztensor const*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_init_ztensor_with_malloc(zdnn_tensor_desc*, zdnn_tensor_desc*, zdnn_ztensor*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_transform_ztensor(zdnn_ztensor*, ...)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_transform_origtensor(zdnn_ztensor const*, void*)'
/usr/bin/ld: ../bin/libggml-zdnn.so: undefined reference to `zdnn_is_nnpa_parmblk_fmt_installed(int, ...)'
collect2: error: ld returned 1 exit status
gmake[2]: *** [tests/CMakeFiles/test-tokenizer-0.dir/build.make:104: bin/test-tokenizer-0] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1917: tests/CMakeFiles/test-tokenizer-0.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

[taronaeo]

Hello, thank you for filing this issue report. I am checking this and will report back once I have an update.

[taronaeo]

Can you please run these commands and paste the results here?

1. gcc --version
2. ld --version
3. What LinuxONE system are you running? LinuxONE 4? LinuxONE 5?

[taronaeo]

I am able to reproduce this on AlmaLinux 9.6, GCC 15.2.0, ld 2.35.2-63.el9, IBM z17.

The problem is that the dnf packages listed here are somehow not linking correctly.

$ rpm -qa | grep zdnn
libzdnn-1.1.1-1.el9.s390x
libzdnn-devel-1.1.1-1.el9.s390x

To fix it, you will need to build zDNN from source by following the instructions here: https://github.com/IBM/zDNN?tab=readme-ov-file#building-and-installing-zdnn.

Once that is done, you need to use the -DZDNN_ROOT macro to specify your zDNN prefix path where the libraries are located at. E.g.,

cmake -S . -B build_zdnn -DCMAKE_BUILD_TYPE=Release -DGGML_ZDNN=ON -DLLAMA_CURL=OFF -DZDNN_ROOT=/opt/zdnn

And compile as per normal. That will fix the linker issues.

[owainkenwayucl]

Thanks for this (sorry was at a conference yesterday).

I'll give that a go.

(Our LinuxOne is a LinuxOne Express 4 with a telum processor)

[taronaeo]

Unfortunately IBM z16 / LinuxONE 4 is not supported by the zDNN backend due to the missing fused-transpose facility. You will likely see it error out during inference. You will need an IBM z17 / LinuxONE 5 to use the zDNN backend.

[owainkenwayucl]

Ah! That's disappointing. I hope at least the linking instructions help anyone else on a newer machine who runs into the linking problems with the zDNN build from the Alma repos!

As a counterpoint, it does seem to be working - at least for granite 3.3 2bn:

[uccaoke@vind2 ~]$ which llama-simple-chat
~/llama.cpp/build_zdnn/bin/llama-simple-chat
[uccaoke@vind2 ~]$ ldd ~/llama.cpp/build_zdnn/bin/llama-simple-chat
        linux-vdso64.so.1 (0x000003ffeb0fd000)
        libllama.so => /home/uccaoke/llama.cpp/build_zdnn/bin/libllama.so (0x000003ff8df00000)
        libggml.so => /home/uccaoke/llama.cpp/build_zdnn/bin/libggml.so (0x000003ff8e180000)
        libggml-cpu.so => /home/uccaoke/llama.cpp/build_zdnn/bin/libggml-cpu.so (0x000003ff8dd00000)
        libggml-zdnn.so => /home/uccaoke/llama.cpp/build_zdnn/bin/libggml-zdnn.so (0x000003ff8de80000)
        libggml-base.so => /home/uccaoke/llama.cpp/build_zdnn/bin/libggml-base.so (0x000003ff8db00000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x000003ff8d800000)
        libm.so.6 => /lib64/libm.so.6 (0x000003ff8d700000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000003ff8da80000)
        libc.so.6 => /lib64/libc.so.6 (0x000003ff8d400000)
        /lib/ld64.so.1 (0x000003ff8e200000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x000003ff8d680000)
        libzdnn.so.0 => /home/uccaoke/zDNN/1.1.2/lib/libzdnn.so.0 (0x000003ff8d380000)
[uccaoke@vind2 ~]$ llama-simple-chat -m models/granite_33_2bn_be_q8_0.gguf 
..................................................................................................
> Give me a short recipe for a chocolate cake. Fit in in less than ten lines.
1. Preheat oven to 350°F (175°C). Grease and flour a 9-inch round cake pan.
2. In a large bowl, mix 1 3/4 cups all-purpose flour, 1 cup granulated sugar, 3/4 cup unsweetened cocoa powder, 1 1/2 tsp baking powder, and 1 1/2 tsp baking soda.
3. Add 2 large eggs, 1 cup milk, 1/2 cup vegetable oil, and 2 tsp vanilla extract. Mix until smooth.
4. Pour batter into prepared pan. Bake for 30-35 minutes or until a toothpick inserted into the center comes out clean.
5. Let cool in the pan for 10 minutes, then transfer to a wire rack to cool completely.
6. For the frosting, combine 1/2 cup softened butter, 2/3 cup unsweetened cocoa powder, 3 cups powdered sugar, 1/3 cup milk, and 1 tsp vanilla extract. Beat until smooth and creamy. Spread over cooled cake. Enjoy your homemade chocolate cake!
> 

[taronaeo]

I hope at least the linking instructions help anyone else on a newer machine who runs into the linking problems with the zDNN build from the Alma repos!

The instructions are already linked in the docs/build-s390x.md documentation. Everyone has to build zDNN from source unfortunately.

As a counterpoint, it does seem to be working - at least for granite 3.3 2bn:

Your model is Q8_0, it will fallback to CPU / SIMD routines. Only FP32 models will use zDNN for now, and thats where you will probably see it error out.

[owainkenwayucl]

OK - Thanks for all your help :)

[taronaeo]

Hi @owainkenwayucl, I might have figured out a way to support IBM z16 / LinuxONE 4 for the IBM zDNN backend without the need of the fused-transpose facility. However, since the IBM z16 / LinuxONE 4 does not have the remote zAIU feature, you will be limited to only 1 thread (1 zAIU per 8 IFLs) and performance cannot be compared against the IBM z17 / LinuxONE 5.

Out of curiosity, how interested are you in seeing this working on your machine? So that I can prioritise this with my team :)