Backends: SDL3: Fix macOS mouse position incorrect for fullscreen apps

[MattGuerrette]

This commit fixes an issue with macOS on devices with a notch. SDL3 no longer provides a client area that encompasses the notch, so any reporting of the mouse location must factor this in for fullscreen SDL3 apps on macOS. SDL3 developers suggested using SDL_GetMouseState() to retrieve the correct location until they can find a solution for rendering into the notch area.

For more discussion about this issue see the following thread:
libsdl-org/SDL#10441

[claudiofreda]

Running into the same problem at the moment, I wonder if there's a quick fix or workaround I can use.

[what-am-i-doing]

This was a fairly maddening bug I encountered while switching to SDL3 (technically it's also present in SDL2, but I was evading it with a fullscreen mode that's no longer available in SDL3). I'm glad I found @MattGuerrette's fix for it and I have modified my copy of Dear ImGui to include it for now (hopefully it can be merged) - works perfectly. Thank you Matt :)

[claudiofreda]

Yeah, I ended up doing the same, I applied the fix to my own copy of Imgui's source code and it's been working pretty well. I hope this gets merged.

[ocornut]

Sorry for my late answer. I've posted a question at libsdl-org/SDL#10441 because I'm not entirely sure this isn't a bug in SDL3.

[ocornut]

@MattGuerrette @claudiofreda @what-am-i-doing Would you be able to confirm if this is still happening with latest SDL3 ? (aka you get the offset without applying this fix)
I've talked with Sam from SDL3, and we may want to investigate further.
If this is still happening, could you run the testwm app from SDL3 and report results here? I would then notify Sam. Thank you!

[what-am-i-doing]

Also confirming that the problem is still present. I use SDL3 supplied by vcpkg, which doesn't come with any test app or suchlike (so far as I can tell), and I couldn't find it on google either, so I can't report on testwm's results.

[MattGuerrette]

@ocornut Yes, this is still an issue as of SDL3 v3.2.22. It seems that SDL_GetMouseState when called for fullscreen macOS windows retrieves the cursor position for the active window adjusted for the safe area insets, but SDL_GetGlobalMouseState retrieves the mouse position position for the entire NSScreen which includes the titlebar and notch. So, I believe this PR is still valid and should be considered for merging.

This behavior can also be seen when running the testwm app though its difficult to catch, but if your cursor exits the client area into the titlebar or notch, the SDL_GetMouseState value reporting will stop while SDL_GetGlobalMouseState will always report an updated value

[ocornut]

Thanks for confirming!

Could you share a recording of testwm output with the mouse moving from the top left position and slightly down ?

seems that SDL_GetMouseState when called for fullscreen macOS windows retrieves the cursor position for the active window adjusted for the safe area insets, but SDL_GetGlobalMouseState retrieves the mouse position position for the entire NSScreen which includes the titlebar and notch

The problem may as well be as that SDL_GetWindowPos() doesn’t return value consistent with the other’s two values. The two values above may only be compared if you factor that third value.

So, I believe this PR is still valid and should be considered for merging.

My belief is that it is a more general issue in SDL3. See my post in libsdl-org/SDL#10441 + I don’t know if a fullscreen window is necessarily focused (SDL_GetMouseState() returns the position relative to focused window). I don’t mind adding the workaround but it’d be good to get to the end of this neatly.

[slouken]

I'm still confused what the bug is. SDL_GetMouseState() returns the mouse position relative to and clamped to the window client area, and SDL_GetGlobalMouseState() returns the mouse position in global screen coordinates. Fullscreen windows on macOS do not cover the notch area, so SDL_GetMouseState() would naturally not change when the mouse is moved over it.

If you need the global position of the mouse, you should call SDL_GetGlobalMouseState(), if you need the window relative position, you should call SDL_GetMouseState(). If you need global mouse coordinates relative to the window client area, you should call SDL_GetGlobalMouseState() and subtract SDL_GetWindowPosition().

What am I missing?

[MattGuerrette]

@slouken

If you need global mouse coordinates relative to the window client area, you should call SDL_GetGlobalMouseState() and subtract SDL_GetWindowPosition().

That is the bug I think that @ocornut is questioning. SDL_GetGlobalMouseState() - SDL_GetWindowPosition() does not seem to produce the expected result on macOS with fullscreen windows.

[slouken]

So what values are you seeing for SDL_GetWindowPosition() when fullscreen in this situation? I would expect x = 0 and y = 40, or something like that.

[MattGuerrette]

@slouken SDL_GetWindowPosition returns 0,0 for fullscreen window on macOS

[slouken]

Okay, I understand the problem now. SDL is assuming that fullscreen windows are at 0,0 which isn’t true in this case. I just got access to a Mac with a notch, so I’ll confirm and work on a fix. I’ll update further on the SDL bug.

BTW, in the long run, a combination of the original code and ozgurduygu/Forkster@1c96316 will work best. If an SDL window has mouse focus, you’ll want to use SDL_GetMouseState(), and if not, then use SDL_GetGlobalMouseState() and subtract the window position. This will give you good performance in the mouse focus case and valid mouse position when another application has focus.

[slouken]

This is fixed for the next SDL release!

[ocornut]

Thanks Sam!
The solution on dear imgui SDL2/SDL3 backend side also becomes very simple: f61a7ef
Essentially since that codepath was a workaround to provide unclamped mouse position for windows that are focused but not hovered not captured, we can simply avoid the path when SDL_GetMouseFocus() != NULL as in that situation we got SDL mouse motion events, which are both faster on X11 and correct with macOS notch with SDL <3.3.0.
Thanks everyone for reporting, and sorry it took a long time to get through this.