Session Replay keeps encoding segments while rate-limited and leaks MediaCodec/MediaMuxer file descriptors (EMFILE)

[buenaflor]

Description

Integration: other (sentry-android-replay)

Build System: Gradle

AGP Version: N/A (reported downstream via sentry-flutter, see getsentry/sentry-dart#3528)

Proguard: Enabled

Other Error Monitoring Solution: No

Version: 8.48.3

---

Steps to Reproduce

Downstream report: getsentry/sentry-dart#3528 (Flutter app on Android, session mode replay, org under replay quota/rate limit).

1. Enable Session Replay in session mode (sessionSampleRate = 1.0).
2. Run the app against a DSN whose org is rate-limited for replay (server returns HTTP 429 on replay envelopes).
3. Leave the app in the foreground for ~10+ minutes.
4. Watch logcat: c2.android.avc.encoder allocate/encode/release cycles continue the whole time, each followed by Envelope discarded due all items rate limited. Eventually the process exhausts its FD limit (EMFILE / errno 24) and Flutter's Impeller/Vulkan rendering dies (Could not acquire next swapchain image: ErrorOutOfHostMemory).

Expected Result

While rate-limited for replay/all, no segments are created or encoded, and failed/empty encodes never leak codec, surface, muxer, or file handles.

Actual Result

Encoder cycles continue while the client-side rate limit is active — each segment is encoded, then immediately discarded by RateLimiter.filter(). Eventually the process hits its FD limit (EMFILE) and rendering dies (ErrorOutOfHostMemory, see downstream issue for full logs):

E/Sentry: ... response code 429
D/CCodec: allocate(c2.android.avc.encoder)
... full encode cycle ...
W/Sentry: Envelope discarded due all items rate limited.

Root cause

The rate-limit pause (#3854) only stops the recorder; segment creation is never gated:

1. SessionCaptureStrategy.pause() (SessionCaptureStrategy.kt:40) encodes a segment as part of pausing — which is then discarded client-side by definition.
2. RateLimiter re-notifies observers when the limit expires (RateLimiter.java:320) → replay resumes → next segment is sent → 429 → pause again. This flap loops forever with no backoff, allocating encoders each round.
3. SessionCaptureStrategy.onScreenshotRecorded() (SessionCaptureStrategy.kt:74) has no paused/rate-limited check. For hybrid SDKs (Flutter/RN), recorder.pause() is an async best-effort bridge call with exceptions swallowed — if it fails, a segment is encoded and discarded every 5s indefinitely.

Leaks per wasted cycle

1. ReplayCache.createVideoOf (ReplayCache.kt:200): the frameCount == 0 path returns without releasing the encoder → leaks MediaCodec, input Surface, and the MediaMuxer's open FD.
2. SimpleVideoEncoder.release() (SimpleVideoEncoder.kt:285): single try block — if drainCodec/stop() throws, codec/surface/muxer are never released; exception swallowed at DEBUG.
3. SimpleMp4FrameMuxer.release() (SimpleMp4FrameMuxer.kt:69): muxer.stop() throws when zero samples were written (matches E/MPEG4Writer: Stop() called but track is not started or stopped in the log) → muxer.release() skipped → leaked FD.
4. SentryEnvelopeItem.fromReplay (SentryEnvelopeItem.java:469): the mp4 is only deleted during serialization — envelopes dropped by the rate limiter never serialize, so the replay folder grows for the whole session.

Proposed fixes

1. Gate segment creation on rateLimiter.isActiveForCategory(All/Replay) in SessionCaptureStrategy before encoding. The gate must also drop/rotate incoming frames (rotate() only runs inside createVideoOf), otherwise this trades FDs for unbounded disk growth.
2. Skip the "pause" segment when pausing because of rate limiting.
3. Leak-proof teardown: release encoder on the frameCount == 0 path; per-resource try/catch in SimpleVideoEncoder.release(); try { stop() } finally { release() } in the muxer.
4. Delete replayEvent.videoFile when the envelope is dropped client-side.

[linear-code]

JAVA-556