Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)

[paruparu]

Host memory grows ~6.8 KB per inference on aarch64 + CUDA (not on x86_64, not on CPU)

Summary

Running WhisperModel.transcribe() in a loop with a resident model and fixed-shape
input leaks about 6.8 KB of host RSS per call on aarch64 + CUDA. The growth is
strictly linear and is not recovered by gc.collect() or malloc_trim(0).

The same test shows no growth on x86_64 + CUDA and no growth on aarch64 + CPU,
using the same CTranslate2 version (4.8.1) and the same script.

At a realistic streaming-ASR rate (one 5-second chunk every 5 seconds) this is
~117 MB/day, which we confirmed in a 24-hour run.

This is host memory (RSS), not GPU memory — which distinguishes it from the
previously reported GPU-side growth in #1488 and SYSTRAN/faster-whisper#660.

Environment matrix

All rows: CTranslate2 4.8.1, faster-whisper 1.2.1, model Systran/faster-whisper-small,
compute_type=float16 (CPU row: float32), beam_size=1, no VAD,
input = twelve fixed 5-second 16 kHz mono WAV files cycled in order.

#	CPU arch	Device	CUDA	CT2 build	Python	OS	Result
1	x86_64 (i9-14900KF, RTX 4090)	CUDA	12.4	official PyPI wheel	3.10	Ubuntu 22.04	no growth
2	aarch64 (Jetson AGX Orin)	CPU	–	official PyPI wheel	3.12	Ubuntu 24.04	no growth
3	aarch64 (Jetson AGX Orin, sm_87)	CUDA	13.2 (JetPack 7.2)	built from source	3.12	Ubuntu 24.04	+6.8 KB/call
4	aarch64 (NVIDIA GB10, sm_121)	CUDA	13.0 (plain Ubuntu)	built from source	3.12	Ubuntu 24.04	+6.8 KB/call

Rows 3 and 4 are different machines, different GPUs, different CUDA versions, and were
built independently — yet produce the same per-call figure.

Measurements

Every 250 iterations we call gc.collect() then malloc_trim(0), and record RSS after
each. The table shows RSS after malloc_trim(0) (i.e. memory that cannot be reclaimed).

iterations	Jetson AGX Orin (row 3)	GB10 (row 4)
250	1533.3 MB	808.4 MB
500	1594.9 MB	810.2 MB (+1.8)
750	1596.6 MB (+1.7)	812.0 MB (+1.8)
1000	1598.3 MB (+1.7)	813.6 MB (+1.6)
1250	1600.0 MB (+1.7)	815.2 MB (+1.6)

+1.6–1.8 MB per 250 calls = ~6.8 KB per call, on both machines, with no sign of
levelling off. (The larger jump before 500 iterations is the allocator pool filling up;
that part does saturate. The linear part after it does not.)

x86_64 (row 1) over the same 1250 iterations: -0.8 to +0.4 MB, i.e. flat.

What we ruled out

• Python-side accumulation — gc.collect() reclaims 0.0 MB and the number of
  objects tracked by gc grows by +5 over 1250 iterations, on all four rows.

• glibc arena retention — malloc_trim(0) does reclaim a steady amount every time
  (7.4–66.6 MB depending on the machine), but it does not change the slope.

• The CUDA allocator implementation — we ran row 4 again with
  CT2_CUDA_ALLOCATOR=cub_caching. The switch demonstrably took effect (the amount
  reclaimed by malloc_trim changed from 31.6 MB to 7.3 MB per call), but the slope was
  identical:

  iterations	cuda_malloc_async (default)	cub_caching
  250	+2.3 MB	+2.0 MB
  500	+4.1 MB	+4.1 MB
  750	+5.9 MB	+5.4 MB
  1000	+7.5 MB	+7.5 MB
  1250	+9.0 MB	+9.3 MB

  So this is not CudaAsyncAllocator / cudaMallocAsync, which is what
  Faster whisper holding memory not releasing it. Killing the flask server SYSTRAN/faster-whisper#660 points at for a different (crash) symptom.

• A broken local build — rows 3 and 4 were configured and compiled separately, on
  different machines, for different compute capabilities, and agree to the same figure.

• Variable input shapes — every call uses one of the same twelve 5-second files, so
  shape-keyed caches cannot explain unbounded growth.

Reproduction

import ctypes, gc, sys, time
from faster_whisper import WhisperModel

libc = ctypes.CDLL("libc.so.6")

def rss_mb():
    for line in open("/proc/self/status"):
        if line.startswith("VmRSS:"):
            return int(line.split()[1]) / 1024

chunks = sys.argv[1:]                      # twelve 5-second 16 kHz mono WAV files
model = WhisperModel("small", device="cuda", compute_type="float16")
list(model.transcribe(chunks[0], language="ja", beam_size=1)[0])
gc.collect(); libc.malloc_trim(0)
base, base_obj = rss_mb(), len(gc.get_objects())
print(f"[ready] rss={base:.1f}MB obj={base_obj}")

for i in range(1, 1251):
    segments, _ = model.transcribe(chunks[i % len(chunks)], language="ja", beam_size=1)
    _ = "".join(s.text for s in segments)
    if i % 250 == 0:
        r0 = rss_mb(); gc.collect()
        r1 = rss_mb(); libc.malloc_trim(0)
        r2 = rss_mb()
        print(f"{i:5d}  rss={r0:7.1f}  after_gc={r1:7.1f}  after_trim={r2:7.1f}  "
              f"net={r2-base:+6.1f}MB  gc_objects={len(gc.get_objects())-base_obj:+d}")

Build configuration used for rows 3 and 4 (stock apart from the arch list):

-DWITH_CUDA=ON -DWITH_CUDNN=ON -DWITH_MKL=OFF -DWITH_OPENBLAS=OFF -DWITH_DNNL=OFF
-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DOPENMP_RUNTIME=COMP

Note for row 4: CMake 3.28's cuda_select_nvcc_arch_flags does not recognise sm_121,
so that call was replaced with an explicit
-gencode arch=compute_121,code=sm_121. Row 3 (sm_87) was built without any patch and
produces the same figure, so this workaround is not related to the leak.

Impact

For always-on speech recognition this is the difference between a process that runs
indefinitely and one that has to be restarted. On an 8 GB Jetson Orin Nano the practical
budget is about 20 days; we currently restart the recogniser process once a day as a
workaround (reloading the model takes ~6 s, so this is cheap — but it should not be
necessary).

Related

• Memory increase #1488 — GPU memory growth on x86_64 with variable input sizes. Different symptom
  (device memory, not host RSS) and different trigger (variable shapes).
• Faster whisper holding memory not releasing it. Killing the flask server SYSTRAN/faster-whisper#660 — several reports there point at
  ctranslate2::cuda::cudaasyncallocator::free(), but that is a crash signature; our
  cub_caching run above shows the steady host-side growth happens without that
  allocator.

[SichenLiang]

One more row for the matrix, on CUDA 12.x / JetPack 6 (not in the table yet):

Device	CUDA	Platform	CTranslate2	Result
Jetson Orin Nano Super (sm_87)	12.6	JetPack 6.2.3 (L4T R36.5.2), Python 3.10	4.8.2 and 4.8.1, source builds (-DWITH_CUDA=ON -DWITH_CUDNN=ON -DCUDA_ARCH_LIST=8.7, cuDNN 9.3.0)	bounded step growth in the first ~1000 calls, then flat

Your script unchanged (twelve 5 s 16 kHz mono WAVs, small, float16, faster-whisper 1.2.1). CTranslate2 4.8.2, 1250 calls:

[ready] rss=1300.0MB obj=50878
  250  rss= 1584.8  after_gc= 1584.8  after_trim= 1367.8  net= +67.8MB  gc_objects=+2
  500  rss= 1584.5  after_gc= 1584.5  after_trim= 1367.8  net= +67.8MB  gc_objects=+2
  750  rss= 1584.5  after_gc= 1584.5  after_trim= 1367.8  net= +67.8MB  gc_objects=+2
 1000  rss= 1586.2  after_gc= 1586.2  after_trim= 1369.5  net= +69.5MB  gc_objects=+2
 1250  rss= 1526.7  after_gc= 1526.7  after_trim= 1369.5  net= +69.5MB  gc_objects=+2

Same script with only the loop bound changed to 5000, 4.8.2:

[ready] rss=1270.0MB obj=50410
  250  rss= 1156.8  after_gc= 1156.8  after_trim= 1149.5  net=-120.5MB  gc_objects=+1
  500  rss= 1177.3  after_gc= 1177.3  after_trim= 1168.4  net=-101.6MB  gc_objects=+1
  750  rss= 1177.3  after_gc= 1177.3  after_trim= 1168.4  net=-101.6MB  gc_objects=+1
 1000  rss= 1177.3  after_gc= 1177.3  after_trim= 1168.4  net=-101.5MB  gc_objects=+1
 1250  rss= 1177.4  after_gc= 1177.4  after_trim= 1168.5  net=-101.5MB  gc_objects=+1
 1500  rss= 1177.4  after_gc= 1177.4  after_trim= 1168.5  net=-101.5MB  gc_objects=+1
 1750  rss= 1177.4  after_gc= 1177.4  after_trim= 1168.5  net=-101.5MB  gc_objects=+1
 2000  rss= 1177.4  after_gc= 1177.4  after_trim= 1168.5  net=-101.5MB  gc_objects=+1
 2250  rss= 1177.4  after_gc= 1177.4  after_trim= 1168.5  net=-101.5MB  gc_objects=+1
 2500  rss= 1177.3  after_gc= 1177.3  after_trim= 1168.5  net=-101.5MB  gc_objects=+1
 2750  rss= 1176.8  after_gc= 1176.8  after_trim= 1168.0  net=-102.0MB  gc_objects=+1
 3000  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1
 3250  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1
 3500  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1
 3750  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1
 4000  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1
 4250  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+2
 4500  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1
 4750  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1
 5000  rss= 1177.0  after_gc= 1177.0  after_trim= 1168.2  net=-101.8MB  gc_objects=+1

And with 4.8.1 (the version in your report), 5000 calls:

[ready] rss=1264.2MB obj=52339
  250  rss= 1305.5  after_gc= 1305.5  after_trim= 1296.4  net= +32.2MB  gc_objects=+1
  500  rss= 1341.8  after_gc= 1341.8  after_trim= 1333.7  net= +69.4MB  gc_objects=+1
  750  rss= 1344.7  after_gc= 1344.7  after_trim= 1336.6  net= +72.4MB  gc_objects=+1
 1000  rss= 1344.6  after_gc= 1344.6  after_trim= 1336.5  net= +72.3MB  gc_objects=+2
 1250  rss= 1344.6  after_gc= 1344.6  after_trim= 1336.6  net= +72.3MB  gc_objects=+1
 1500  rss= 1344.5  after_gc= 1344.5  after_trim= 1336.4  net= +72.2MB  gc_objects=+1
 1750  rss= 1344.6  after_gc= 1344.6  after_trim= 1336.6  net= +72.3MB  gc_objects=+1
 2000  rss= 1344.6  after_gc= 1344.6  after_trim= 1336.5  net= +72.2MB  gc_objects=+1
 2250  rss= 1345.6  after_gc= 1345.6  after_trim= 1336.6  net= +72.4MB  gc_objects=+1
 2500  rss= 1344.6  after_gc= 1344.6  after_trim= 1336.5  net= +72.3MB  gc_objects=+1
 2750  rss= 1344.7  after_gc= 1344.7  after_trim= 1336.6  net= +72.4MB  gc_objects=+1
 3000  rss= 1344.6  after_gc= 1344.6  after_trim= 1336.5  net= +72.3MB  gc_objects=+1
 3250  rss= 1344.7  after_gc= 1344.7  after_trim= 1336.6  net= +72.4MB  gc_objects=+1
 3500  rss= 1344.6  after_gc= 1344.6  after_trim= 1336.5  net= +72.3MB  gc_objects=+1
 3750  rss= 1344.7  after_gc= 1344.7  after_trim= 1336.6  net= +72.4MB  gc_objects=+1
 4000  rss= 1345.5  after_gc= 1345.5  after_trim= 1336.4  net= +72.2MB  gc_objects=+1
 4250  rss= 1344.7  after_gc= 1344.7  after_trim= 1336.6  net= +72.3MB  gc_objects=+1
 4500  rss= 1344.4  after_gc= 1344.4  after_trim= 1336.4  net= +72.1MB  gc_objects=+1
 4750  rss= 1344.5  after_gc= 1344.5  after_trim= 1336.4  net= +72.2MB  gc_objects=+2
 5000  rss= 1344.4  after_gc= 1344.4  after_trim= 1336.3  net= +72.1MB  gc_objects=+1

So the RSS does go up and gc.collect() / malloc_trim(0) don't give it back, but it comes in steps of roughly 25-70 MB inside the first ~1000 calls and then stops: between call 750 and call 5000 the change is -0.2 MB (4.8.2) and -0.3 MB (4.8.1). I don't see anything like a constant +6.8 KB/call on this configuration.

CUDA build check: ctranslate2.get_cuda_device_count() = 1, compute types ['bfloat16', 'float16', 'float32', 'int8', 'int8_bfloat16', 'int8_float16', 'int8_float32'].

Disclosure: an AI assistant helped set up the builds, ran the 4.8.1 series, and drafted this comment; all runs were on my own board and the numbers are pasted unedited. Happy to run anything else on this configuration while I still have the device (until about Sept 11).

[paruparu]

@SichenLiang Thank you — this is very useful, and it turned out to be the hint that
moved the investigation forward.

Your 5000-call run is already conclusive; no longer runs needed

I initially thought the difference might be hidden by the warm-up step, but comparing the
same post-warm-up interval on both sides makes it unambiguous:

750 → 5000 calls (same 4250-call span)	RSS change
JetPack 6.2.3 / CUDA 12.6 (yours, 4.8.1)	−0.3 MB
JetPack 6.2.3 / CUDA 12.6 (yours, 4.8.2)	−0.2 MB
JetPack 7.2 / CUDA 13.2 (mine, 4.8.1)	+28 MB (+6.7 KB/call)

And on my side it is linear, not stepwise — after the warm-up settles it is +1.7 MB per
250 calls, flat to three digits:

1250 calls  1458.9 MB
2500 calls  1467.3 MB
3750 calls  1475.8 MB
5000 calls  1484.2 MB      -> 6.96 KB/call

which matches the 6.9 KB/call I measured over a 20-hour, 14,737-call run.

So please don't spend more device time on longer runs — your existing data already
answers the question.

New evidence: this is very likely not in CTranslate2

Four measurements on the CUDA 13.2 side, all pointing outside the library:

1. Building with -DWITH_CUDNN=OFF does not stop it.
CTranslate2 only uses cuDNN in src/ops/conv1d_cudnn_gpu.cu, which Whisper's encoder
hits on every call. With cuDNN linked out (ldd shows no libcudnn), the steady slope
is unchanged at 250 calls / +1.7 MB. Only the size of the one-time warm-up step differs
(+83 MB settled by call 750, versus +115 MB settled by call 1500 with cuDNN).

2. glibc is not the one allocating. Calling malloc_info(0, ...) alongside the loop:

calls    RSS       anon      glibc "system" total    arenas
 150    +1.0MB    +1.0MB          +0.0MB               7
 300    +6.0MB    +7.2MB          +0.0MB               7
 450    +7.0MB    +8.3MB          +0.0MB               7
 600    +8.0MB    +9.3MB          +0.0MB               7

RSS climbs while the total glibc has obtained from the OS does not move at all, and the
arena count stays at 7.

3. heaptrack sees nothing growing. Same script at 50 and 400 iterations
(after a 30-call warm-up):

                        50 iterations     400 iterations
allocations                4,998,217        21,708,285
leaked allocations           106,379           106,379   <- identical
peak memory                  533.72M           533.72M   <- identical

The allocation count scales with the loop, but neither retained-at-exit nor peak
tracked memory grows.

4. All of the growth is anonymous, in a handful of existing mappings.
From /proc/self/smaps over 600 calls: 100 % of the RSS increase is in Anonymous, and
the VMA count only goes 1429 → 1432. The two largest contributors are pre-existing
anonymous regions that simply become more resident (+33.7 MB and +32.5 MB), not a stream
of new mappings.

Taken together: nothing reaches this memory through malloc/new, so there is no
CTranslate2 allocation to fix. The growth arrives through mmap done below the library —
i.e. inside the CUDA runtime/driver.

One caveat on "CUDA 13"

The delta between our two setups is not only CUDA:

	yours	mine
CUDA	12.6	13.2
cuDNN	9.3.0	9.20.0
L4T	R36.5.2 (JetPack 6.2.3)	R39.2 (JetPack 7.2)
Python	3.10	3.12

So "CUDA 13.2" is the leading hypothesis, not an established fact. Measurement 1 above
does rule out the cuDNN usage in CTranslate2, but not the cuDNN library itself.

What I plan to do next

Build a minimal reproducer that does not involve CTranslate2 — a small C++ loop issuing
cuBLAS calls with the shape variation Whisper's decoder produces — and see whether host
RSS grows the same way. If it does, this belongs with NVIDIA rather than here, and I will
report it there with the minimal case. I have the board until about Sept 29.

Thanks again for taking the time to run this on your own hardware; the CUDA 12.6 data
point is what made the split possible.

---

Disclosure: an AI assistant helped run these measurements and drafted this comment. All
numbers are from my own board and are pasted unedited.