[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack protection to fiber stacks

[pseudo-rnd-thoughts]

Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518 KiB of live malloc (the per-task asyncio.Task, concurrent.futures.Future, Cython coroutine + scopes, and two msgpack Packers with 256 KiB internal buffers).
Closes #63290

Root cause

1. CPython 3.14 changed how it avoids stack overflow when freeing objects. Freeing one object can recursively free many others (a dict frees its values, which free their contents, …), and each level is a nested C call. To keep that from overflowing the C stack, CPython has long had a safety mechanism (the "trashcan"): when it decides it's too deep, it doesn't free the object right away. Instead it parks the object on a per-thread delete-later list and drains the list once there's stack headroom again. Up to 3.13, "too deep" was a simple recursion counter. In 3.14 it's decided by comparing the actual machine stack pointer against the stack bounds CPython recorded for the thread when it attached (from pthreads, on Linux).

2. Ray async actors don't run task code on the thread's normal stack. Each task executes on a small 256 KiB boost fiber stack allocated elsewhere in memory. The problem is that CPython still thinks the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?" check compares the fiber's stack pointer against the pthread stack's bounds. On Linux, fiber stacks happen to be allocated at lower addresses than the pthread stack, so CPython concludes the stack is hopelessly overflowed and parks every object freed during the task (including return-value serialization and end-of-task cleanup) on the delete-later list.

That list is only ever drained by a later free on the same thread state at a healthy stack margin, which never happens here as the Ray thread only runs on fibers and Ray creates a fresh Python thread state per task and destroys it at task end. This means that CPython destroys a thread state without draining its delete-later list and the parked objects are orphaned permanently. That's the leak.

Why the confusing symptoms:

• boost::make_fcontext in the issue's flamegraphs just marks where the leaked allocations were made (on a fiber stack); the fiber stacks themselves are freed correctly.
• macOS is unaffected only by luck: fiber stacks there land at higher addresses than the pthread stack, so the check passes.
• 3.13 and earlier are unaffected because their trashcan uses the counter, not the stack pointer.

Fix

CPython 3.14.2 added an official API for exactly this situation: PyUnstable_ThreadState_SetStackProtection (python/cpython#141661) lets an embedder tell CPython "this thread is currently executing on this stack." We call it with the fiber's stack bounds:

• at async-actor task entry in task_execution_handler, and
• whenever a fiber resumes after YieldCurrentFiber (concurrent fibers share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal behavior: objects are freed immediately, and the rare genuinely-deep free is parked and then properly drained.

Implementation notes: the symbol is looked up via dlsym, so _raylet still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a more severe, since-fixed stack-check bug anyway, python/cpython#141944). No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are derived from the current stack pointer minus a conservative allowance for stack already used, so the protection errs toward triggering slightly early rather than missing an overflow. Side benefit: fibers gain real C-stack overflow protection (RecursionError) on 3.14, which they currently lack entirely (boost::fibers::fixedsize_stack has no guard pages). Also makes FiberState::kStackSize public so the anchoring uses the real fiber stack size.

Related issue number

Closes #63290. Supersedes #63284 (same diagnosis direction, but hand-rolled _PyThreadStateImpl offsets, a deliberate gilstate_counter leak that freezes non-main threads, and a crash premise that CPython 3.14.2 already fixed upstream).

Checks

• Verified with a locally built cp314 Linux (aarch64, python:3.14.6 docker) wheel:
• refcount probe: +4.00 refs/task → 0.00/task (100 tasks)
• __del__ deferral probe: dealloc during return serialization on the fiber deferred → immediate
• live-malloc probe (mallinfo2, 300 tasks/shape): ~518 KiB/task → ~3 KiB/task across async call → dict/bytes, async generator, sync generator on async actor
• reporter-shaped streaming workload (400 tasks, 10 concurrent sessions): live-malloc delta 0.2 MB total, fiber-sized mapped regions 0 → 0
• async-actor smoke: correctness (echo, state, async generators, recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
• throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs baseline, same container image): fixed 5624–6076 tasks/s vs unpatched 4551–4825 tasks/s meaning no regression (the unpatched build is slower while leaking)
• baseline (unpatched) wheel from the same tree reproduces the bug: +4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation with prompting on what to check next and validate the solution

[gemini-code-assist]

Code Review

This pull request addresses a memory leak on Python 3.14+ where async actor tasks running on boost fiber stacks trigger CPython's stack overflow protection, causing GC-object deallocations to be permanently deferred. The fix introduces RayReanchorStackProtectionToCurrentFiberStack to re-anchor CPython's stack bounds to the current fiber's stack using PyUnstable_ThreadState_SetStackProtection. The feedback highlights two key improvements: documenting the critical safety constraint on used_upper_bound to prevent potential physical stack overflows, and adding a fallback definition for RTLD_DEFAULT to ensure portability and prevent compilation failures on certain Linux environments.

[gemini-code-assist]

There is a critical safety assumption in the stack limit calculation:

uintptr_t top = (uintptr_t)&anchor + used_upper_bound;
int rc = set_stack_protection(PyThreadState_Get(), (void *)(top - stack_size), stack_size);

For the calculated stack range [top - stack_size, top] to safely cover the physical fiber stack without allowing actual stack overflow, used_upper_bound must be a strict upper bound of the actual stack depth U (where U = H - &anchor, and H is the physical stack top).

If the actual stack depth U at the call site ever exceeds used_upper_bound (e.g., if the call stack is deeper than 96 KB when yielding), then:
L_est = &anchor + used_upper_bound - stack_size
L_phys = &anchor + U - stack_size

Since used_upper_bound < U, we get L_est < L_phys. This means the estimated stack limit passed to CPython is lower than the physical stack limit. CPython will believe it has more stack space than physically available, which can lead to a physical stack overflow (and a silent memory corruption or segfault) before CPython's RecursionError is triggered.

Please add a clear comment documenting this critical constraint so that future maintainers do not inadvertently deepen the call stack at these call sites without adjusting the bounds.

    char anchor;
    /* used_upper_bound must be a strict upper bound of the actual stack depth
     * at the call site to prevent overestimating the remaining stack space,
     * which could lead to physical stack overflow before RecursionError triggers. */
    uintptr_t top = (uintptr_t)&anchor + used_upper_bound;

[pseudo-rnd-thoughts]

Added a comment about this

[gemini-code-assist]

On some Linux environments (such as glibc-based systems), RTLD_DEFAULT is only defined in <dlfcn.h> if _GNU_SOURCE is defined. If _GNU_SOURCE is not defined globally or in the Cython compilation environment, this can lead to a compilation failure.

To ensure maximum portability and prevent compilation errors, we should provide a safe fallback definition for RTLD_DEFAULT when it is not defined. On Linux, RTLD_DEFAULT is defined as NULL (or (void *)0). On macOS, it is defined as (void *)-2, but macOS always defines it by default, so the fallback won't override it.

#if PY_VERSION_HEX >= 0x030E0000 && !defined(MS_WINDOWS)
#include <dlfcn.h>
#ifndef RTLD_DEFAULT
#define RTLD_DEFAULT ((void *)0)
#endif
#include "ray/core_worker/task_execution/fiber.h"

[pseudo-rnd-thoughts]

Added this check

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit f77e4d0. Configure here.}

[pseudo-rnd-thoughts]

@Kunchd I can't added a test to this yet. Does core have any general tests for memory leaks?

[pseudo-rnd-thoughts]

Added a comment about this

[pseudo-rnd-thoughts]

Added this check

[Kunchd]

Thanks for the investigation!

@pseudo-rnd-thoughts, I'm not aware of any memory leak tests. But there are test_memory_pressure.py that does deal with memory. Perhaps we could do something similar with your repro and check for the before and after memory footprint?

[Kunchd]

nit: Instead of embedding the logic within the cython. Could We pull this out to an actual file?

[Kunchd]

Why 32 * 1024 specifically?

Ideally we at least document how this number was determined, and how this number should be adjusted in the future if something were to go wrong.

[rueian]

Instead of using these numbers, how about we keep track of each fiber stack address and query them when we do the reanchor? I made a draft at #65119

[Kunchd]

How did we arrive at this number?

[MengjinYan]

I spent some time reproducing this independently on my devbox and can confirm both the leak and the fix. Sharing the some detailed numbers here:

• Environment: I run the experiment with the following permutations:
  • python 3.13.14 + master
  • python 3.14.6 + master
  • python 3.14.6 + master + the fix in this PR
• Here is the script for the test file async_actor_leak_probe.py
• With injecting the __del__ function and measure the USS and RSS growth during job execution, in a execution with 300 tasks to test deletion and 3000 tasks with 1KB return value to test memory growth, the number is as the following:

Build	__del__ ran	RSS / task
3.13.14 master (control)	300/300	5.5 KiB
3.14.6 master	0/300	25 KiB
3.14.6 + this PR	300/300	0.1 KiB

0 of 300 __del__ is called on the unfixed 3.14 and all 300 on both the 3.13 and the patched build. So your diagnosis lines up exactly.

---

At the same time, I'd like to help with the followup changes. I have a small follow-up prepared on top of your branch:

1. Add a guard to make sure the async actor creation task won't reset the start address after the task is done. The actor creation task execution path follow the same as the normal async actor task. At the same time, the creation task doesn't run on fiber. The current code exclude the actor creation task in the beginning of the task execution but not after the task execution.
2. Add a regression test to check whether the task return object can be freed
3. Add a one-time warning when dlsym misses (CPython 3.14.0/3.14.1), so the field symptom isn't "unbounded memory growth with nothing in the logs."
4. Comment fixes: Add rationals about choosing the numbers

For the estimates themselves I'd suggest a separate PR rather than expanding this one and this is what rueian's PR is trying to achieve.

[Kunchd]

I looked into the cython code a bit more, and I wanted to double check something. PyUnstable_ThreadState_SetStackProtection takes stack_start_addr and stack_size as arguments. Here, we passed in the current variable address as stack_start_addr and a magic number for the stack_size.

From there, PyUnstable_ThreadState_SetStackProtection invokes tstate_set_stack. This function sets soft and hard stack limits based on the following sketch (excuse my chicken scratchings):

So there's one particular thing to call out here. If the stack is growing down within the python execution of the sync function, the soft and hard limits doesn't seem to be doing anything since we're growing down from base, away from the limits.

Do we know if the stack is growing up or down here?

[Kunchd]

The _PyOS_STACK_MARGIN_BYTES could reach 192KiB when built with Tsan (see https://github.com/python/cpython/blob/a646c99ee12fbf421394a74955d42fd3fe850b7e/Include/internal/pycore_pythonrun.h#L66). Should we increase this value?

Otherwise, we might fail this assertion: https://github.com/python/cpython/pull/141661/changes#diff-c22186367cbe20233e843261998dc027ae5f1f8c0d2e778abfa454ae74cc59deR542-R546

[MengjinYan]

Thanks for the comment! My understanding is that, the 96 * 1024 here is not the stack size, but the estimation of how much stack space we've already used. The stack size we pass in will always be 256 KiB. In this sense, it will should always be larger.

[Kunchd]

Should we anchor the stack before invoking run_coroutine_threadsafe here?

[MengjinYan]

I don't think it is needed. Fibers are cooperatively scheduled on one thread, so the bounds can only be overwritten by another fiber, and a fiber can only run if the current on yields. The only fiber-suspension point from my understand is YieldCurrentFiber, which the code re-anchors immediately on resume. So when run_coroutine_threadsafe is called, there is no fiber switch so we don't need to re-anchor there.

[MengjinYan]

PyUnstable_ThreadState_SetStackProtection takes stack_start_addr and stack_size as arguments. Here, we passed in the current variable address as stack_start_addr and a magic number for the stack_size.

I have a bit different understanding the above. My understanding is the following:

• Here is the function syntax of the python stack protection functionint PyUnstable_ThreadState_SetStackProtection(PyThreadState *tstate, void *stack_start_addr, size_t stack_size)
  • And by the implementation of the function, the stack_start_addr is the low end of the stack space. And since stack grows downward to the lower memory addresses, this is essentially the top of the stack.
• Here is how we call the function:
  • int rc = set_stack_protection(PyThreadState_Get(), (void *)(top - stack_size), stack_size);
  • where uintptr_t top = (uintptr_t)&anchor + used_upper_bound; and used_upper_bound is the magic estimation of how much stack has been used currently
  • and size_t stack_size = ray::core::FiberState::kStackSize; which is the stack size of each fiber
• So essnetially this means that, top here is the estimated start of the stack from OS's perspective. And the stack_start_addr passed in is the estimated lower address size of the stack, and the stack_size we passed in is the fiber stack size.

From there, PyUnstable_ThreadState_SetStackProtection invokes tstate_set_stack. This function sets soft and hard stack limits based on the following sketch (excuse my chicken scratchings):

So there's one particular thing to call out here. If the stack is growing down within the python execution of the sync function, the soft and hard limits doesn't seem to be doing anything since we're growing down from base, away from the limits.

I might miss something from your drawing but I think the confusion here is that:

• From the cpython implementation, the base is the lower address end of the stack, the top is the higher address end of the stack. In C in linux or mac, the stack will grow from the top to the base.
• Also, based on the readme here, the stack protection is for the c stack, which means that the protection applies to the stack space between stack top and base which means that the limits should still be applicable.

(Generated by Claude with my understanding)
                             high addresses
                                   ▲
                                   │
  ┌────────────────────────────────────────────────┐ ← stack_top = start_addr + stack_size
  │ boost fiber entry trampoline                   │   highest address; where the fiber
  │ CoreWorker::ExecuteTask                        │   began executing
  │ task_execution_handler                         │
  │ execute_task                                   │ ⎫ already consumed by the frames
  │ function_executor                              │ ⎬ between fiber entry and here
  │ run_async_func_or_coro_in_event_loop           │ ⎭ ← Ray cannot measure this
  ├────────────────────────────────────────────────┤ ← &anchor ≈ current stack pointer
  │                                                │
  │        unused — recursion grows DOWN           │   the headroom Python is
  │                     ↓                          │   allowed to consume
  │                                                │
  ├────────────────────────────────────────────────┤ ← c_stack_soft_limit = base + 32 KiB
  │ CPython reserve (2 × _PyOS_STACK_MARGIN_BYTES) │   RecursionError once SP drops below
  ├────────────────────────────────────────────────┤ ← c_stack_hard_limit = base + 16 KiB
  │ CPython reserve (1 × _PyOS_STACK_MARGIN_BYTES) │
  └────────────────────────────────────────────────┘ ← stack_start_addr  (== base)
                                   │                  LOWEST address. Below it: not our
                                   ▼                  memory, and a malloc'd fiber stack
                             low addresses            has no guard page.

Let me know if anything doesn't makes sense or I missed anything.

[Kunchd]

Thanks for the explanation!