332 ensure portability of openmp target backend across nividia and amd gpus

[pbartholomew08]

I'm opening this for visibility/discussion.

I think this implements the changes suggested by #332. However, with this I'm no longer able to compile the code with Cray (cce 20) or AMD (flang 22) compilers due to call c_f_pointer not being offloaded and is_device_addr not being supported, respectively. Of course, there may be a mistake in the implementation in which case I'll correct it and retest.

[CFD-Xing]

I did some investigation and it seems that for the Cray compiler, the device pointer MUST be a dummy argument. So, having something like this does the trick:

  subroutine vecadd_offload_helper(self, a, cp_x, dims_x, b, cp_y, dims_y, dims)

    class(omptgt_backend_t) :: self
    real(dp), intent(in) :: a
    type(c_ptr), intent(in):: cp_x
    integer, intent(in) :: dims_x(3)
    real(dp), intent(in) :: b
    type(c_ptr), intent(in):: cp_y
    integer, intent(in) :: dims_y(3)
    integer, intent(in) :: dims(3)

    integer :: i, j, k
    real(dp), dimension(:,:,:), pointer :: p_x, p_y

    !$omp target teams is_device_ptr(cp_x, cp_y)
    call c_f_pointer(cp_x, p_x, shape=dims_x)
    call c_f_pointer(cp_y, p_y, shape=dims_y)
    !$omp loop collapse(3)
    do k = 1, dims(3)
      do j = 1, dims(2)
        do i = 1, dims(1)
          p_y(i, j, k) = a*p_x(i, j, k) + b*p_y(i, j, k)
        end do
      end do
    end do
    !$omp end loop
    !$omp end target teams

  end subroutine

  subroutine vecadd_offload(self, a, x, b, y)

    class(omptgt_backend_t) :: self
    real(dp), intent(in) :: a
    type(omptgt_field_t), intent(in) :: x
    real(dp), intent(in) :: b
    type(omptgt_field_t), intent(inout) :: y

    type(c_ptr) :: cp_x, cp_y
    integer :: dims_x(3), dims_y(3)
    integer :: dims(3)

    dims = self%allocator%get_padded_dims(x%dir)

    cp_x = x%get_dev_ptr()
    cp_y = y%get_dev_ptr()
    dims_x = x%get_shape()
    dims_y = y%get_shape()

    call vecadd_offload_helper(self, a, cp_x, dims_x, b, cp_y, dims_y, dims)
  end subroutine

EDIT: Sorry, I am probably experiencing a different issue as I am compiling with Cray 16 on Archer2.

[CFD-Xing]

@pbartholomew08 What facilities are you using to compile with cce 20 and flang22? I am trying to compile on ARCHER2 and the compilation fails because ROCM_PATH can't be found. Rather than spending time to figure out how to compile on ARCHER2, I think I am better off to use a more recent facility.

[pbartholomew08]

Hi @CFD-Xing - I've not had a chance to try your suggested changes, I'll hopefully do tomorrow. I'm building using the latest flang I can get from AMD on Dirac, and I think if you log into a GPU node on ARCHER2 you should get CCE 20 for the Cray compilers

[CFD-Xing]

@pbartholomew08 Ok thanks for the information.

EDIT: I don't think I can access GPU node on ARCHER2.

PS: You can use my patch to give a quick try if you want.
omp.patch

[pbartholomew08]

Thanks @CFD-Xing - I've tried building with the patch on ARCHER2 using the Cray compiler and still have the issue around offloading c_f_pointer - have you been able to build this under amdflang, or something that isn't nvhpc?

[CFD-Xing]

@pbartholomew08 I finally managed to sort out how to compile on ARCHER2. The previous error that I was seeing was probably due to using CCE 16 instead of CCE 20. And after looking into it, it seems that the Cray compiler does not support calling c_f_pointer inside a OMP TARGET construct...

EDIT: Spend a large part of the day looking into this and I think that there is no way around with the Cray compiler. c_f_pointer must be outside the target directive. NVHPC seems to support both, just tested locally on my laptop with NVHPC 25.11. However the following test is failing: test_omptgt_allocator_omp_tgt_1

Here is a patch fixing a bug and moves the c_f_pointer outside the target directive.
patch.patch

[CFD-Xing]

This should be if (product(dims) <= self%n) then

[CFD-Xing]

@pbartholomew08 With some help from Claude, I think I managed to have something working with both NVHPC and Cray: patch2.patch