core: Do not abort on overflowing window positions in edge resistance

[ArekMiszcz]

Problem

In a Wayland session, cinnamon aborts while the user drags any window:

mutter:ERROR:../src/core/boxes.c:1669:split_edge: assertion failed: (meta_rectangle_horiz_overlap (&old_edge->rect, &remove->rect))

Every Wayland client and Xwayland go down with it, so all windows disappear.

Backtrace from the core dump (muffin 6.4.1, Debian 13):

split_edge                                                boxes.c:1669
meta_rectangle_remove_intersections_with_boxes_from_edges boxes.c:1797
apply_edge_resistance_to_each_side                        edge-resistance.c:1152
meta_window_edge_resistance_for_move                      edge-resistance.c:1209
update_move                                               window.c:6815
meta_window_handle_mouse_grab_op_event                    window.c:7079
event_callback                                            events.c:367

Cause

One of the obscuring windows was a native Wayland Google Chrome window (its previous session was maximized) with unconstrained_rect = (2147482260, 100, 1876x996), i.e. INT_MIN − 1388 after wrapping; its saved_rect.x was INT_MIN. When that rect is used to split resistance edges, BOX_RIGHT() overflows, rectangle_and_edge_intersection() reports an overlap that meta_rectangle_horiz_overlap() then rejects, and the assertion in split_edge() fires.

The bogus position itself most likely comes from the early-maximize handling that master has since reworked (a85a711, d4a6c96), but edge resistance should not take the whole session down, whatever position a client ends up with.

Fix

• Skip windows whose frame rect lies outside a sane coordinate range when collecting resistance edges (computed in 64 bits, so the check itself cannot overflow).
• In split_edge(), keep the edge unsplit and warn once instead of asserting when the edge and the removal rectangle do not overlap.

Testing

• Rebased on master: boxes.c and edge-resistance.c compile without warnings.
• The same change is backported to 6.4.1 and running in daily use on Debian 13 (Cinnamon Wayland, NVIDIA). The overflowing Chrome position has not reproduced since, so the new code path is backed by the core dump analysis rather than by a live reproduction.

🤖 Generated with Claude Code

[leigh123linux]

Did you test your change on master to validate it?

[ArekMiszcz]

Did you test your change on master to validate it?

Yes, now on master (03106ef) as well.

The crash path is unchanged on master, so I reproduced it there directly: a small program built against a master build tree calls meta_rectangle_remove_intersections_with_boxes_from_edges() with the values from the core dump (a META_SIDE_TOP edge at 1920,500 1920x0 and a box at G_MAXINT - 1387,100 1876x996).

• master: muffin:ERROR:../src/core/boxes.c:1669:split_edge: assertion failed: (meta_rectangle_horiz_overlap (&old_edge->rect, &remove->rect)), SIGABRT, the same assertion as the original crash.
• master + this PR: the edge is kept unsplit, one warning is logged, exit status 0.

What I could not do is run the whole compositor from master in a session, because my desktop is Cinnamon 6.4 (Debian 13). The runtime testing of the full change, including skipping such windows in edge resistance, was done on the 6.4.1 backport.