Skip to content

Add keepAlive flag to prevent scale-to-zero during process execution - #176

Merged
drappier-charles merged 22 commits into
mainfrom
cdrappier/keep-alive
Mar 11, 2026
Merged

Add keepAlive flag to prevent scale-to-zero during process execution#176
drappier-charles merged 22 commits into
mainfrom
cdrappier/keep-alive

Conversation

@drappier-charles

@drappier-charles drappier-charles commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a keepAlive flag to process execution that disables sandbox auto-hibernation (scale-to-zero) while a process runs. Uses a counter-based system via blaxel.ScaleDisable/ScaleEnable.

Key changes:

  • New blaxel package (src/lib/blaxel/blaxel.go) for scale-to-zero file-based counter control with file locking
  • keepAlive and timeout parameters on process request/response (HTTP + MCP)
  • Timeout goroutine auto-kills keepAlive processes (default 600s; timeout=0 = infinite)
  • Crash recovery: ScaleReset() on startup clears stale counter
  • OpenAPI spec + LIFECYCLE.md documentation

Bug fixes (latest commits):

  • StopProcess (SIGTERM path) now clears KeepAlive and calls ScaleEnable, preventing the counter from leaking when a keepAlive process is gracefully stopped
  • All reads/writes of process.KeepAlive are synchronized via pm.mu (Lock/RLock) to eliminate the data race between KillProcess/StopProcess and the completion goroutine
  • restartProcess reads KeepAlive/Timeout under pm.mu.RLock() to match the locking discipline

Review & Testing Checklist for Human

  • Double ScaleEnable guard: Verify that when StopProcess or KillProcess clears KeepAlive=false under lock, the completion goroutine (which reads under RLock) correctly skips its own ScaleEnable call — no path should decrement the counter twice
  • StopProcess error rollback: If SIGTERM fails, KeepAlive is restored under lock — confirm this doesn't leave the counter in a bad state if the process already exited between the check and the kill
  • Scale file locking: writeWithLock uses LOCK_EX but GetCounter reads without a lock — verify this is acceptable (currently only used for logging)
  • isScaleAvailable thread safety: The scaleAvailableChecked/scaleAvailable bools are plain globals written once at first call — confirm no goroutine can race on the initial check
  • Test keepAlive end-to-end: Start a process with keepAlive=true, verify counter increments, then stop/kill and verify counter returns to 0. Test with timeout=0 (infinite) and with a short timeout that expires

Suggested test plan:

  1. Deploy locally, POST a keepAlive process, confirm scale file counter > 0
  2. Kill/stop the process, confirm counter returns to 0
  3. Start a keepAlive process, kill sandbox-api, restart it, confirm ScaleReset resets counter to 0
  4. Start two keepAlive processes, kill one — counter should be 1, not 0

Notes

drappier-charles and others added 20 commits January 25, 2026 21:33
Introduces new lifecycle features to the sandbox API, including the ability to force stop processes and retrieve the current sandbox status. The `/stop` endpoint allows for immediate or scheduled removal of the keepAlive flag from running processes, enabling auto-hibernation. The `/status` endpoint provides information on the current state of the sandbox and active keepAlive processes.

Additionally, integrates the lifecycle management with the MCP tools, enhancing the overall process control and monitoring capabilities. The scale-to-zero functionality is also improved with crash recovery mechanisms.

Updates include:
- New `LifecycleHandler` for managing lifecycle operations.
- API documentation updates for new endpoints.
- Integration of keepAlive functionality in process management.
- Comprehensive tests for lifecycle features and MCP integration.
This commit removes the `/stop` and `/status` endpoints from the sandbox API, along with the associated `LifecycleHandler` and related data structures. The lifecycle management features, including the ability to force stop processes and retrieve the current sandbox status, have been deprecated.

Updates include:
- Deletion of lifecycle-related API routes and handlers.
- Removal of lifecycle management types and structures from the codebase.
- Adjustments to documentation to reflect the removal of these features.

This change simplifies the API and focuses on core process management functionalities.
This commit introduces functionality to manage timeouts for processes with the keepAlive flag enabled. If a process is restarted and keepAlive is active with a specified timeout, a goroutine is initiated to monitor the timeout and kill the process if it exceeds the limit. For processes with an infinite timeout, the goroutine simply waits for the process to complete. This enhancement improves process management and ensures better resource handling.
…-zero warnings and timeout messages. Clear KeepAlive state before killing processes to prevent double ScaleEnable calls. Refactor related log messages for consistency.
…KeepAlive events. Enhance clarity by including process details in log messages for scale-to-zero operations and timeout handling. This improves consistency and debuggability of process management logs.
… (CWE-117)

Add sanitizeLogValue() helper that escapes newlines and control characters.
Replace structured logging (logrus.WithFields) with simple logrus.Infof/Warnf
calls that use sanitized values - clearer and more readable.
Addresses all CodeQL 'Log entries created from user input' warnings.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Replace strings.NewReplacer with byte-level filtering that strips all
control characters (< 0x20) including newlines. This provides more
thorough sanitization against log injection.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ion (CWE-117)

Replace custom byte-level loop with package-level strings.NewReplacer
variable (logSanitizer). CodeQL explicitly recognizes strings.Replacer.Replace
as a sanitizer for go/log-injection since github/codeql#11910.
Call logSanitizer.Replace() directly at each log site.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…117)

CodeQL's SafeFormatArgumentSanitizer explicitly recognizes %q as safe
because it escapes newline characters. This is the simplest and most
idiomatic fix - no helper functions or variables needed.

Removes the logSanitizer variable entirely. User-provided values (name,
command) are now logged with %q which produces Go-syntax quoted strings,
making any control characters visible in the output.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Reverts commits 6902a62, 16773fb, 4eeb6f7, 3a5b776 which added
log sanitization for CWE-117. Keeps all other agent recommendations
(structured logging with logrus.WithFields, race condition fix, etc).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
mendral-app[bot]

This comment was marked as outdated.

- StopProcess (SIGTERM path) now clears KeepAlive and calls ScaleEnable,
  preventing the scale-to-zero counter from leaking when a keepAlive
  process is gracefully stopped.

- All reads/writes of process.KeepAlive are now synchronized via pm.mu:
  KillProcess and StopProcess write under Lock(), completion goroutines
  read under RLock(). This eliminates the data race between the kill/stop
  path and the completion goroutine.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
mendral-app[bot]

This comment was marked as outdated.

Protect the read of oldProcess.KeepAlive and oldProcess.Timeout in
restartProcess with pm.mu.RLock(), matching the synchronization used
by KillProcess/StopProcess which write under pm.mu.Lock().

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@mendral-app mendral-app Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

The data race flagged in the previous review is fixed — restartProcess now reads oldProcess.KeepAlive and oldProcess.Timeout under pm.mu.RLock(), consistent with how KillProcess/StopProcess write under pm.mu.Lock(). All three race conditions from the review cycle are resolved.

Tag @mendral-app with feedback or questions. View session

@devin-ai-integration devin-ai-integration Bot changed the title Cdrappier/keep alive Add keepAlive flag to prevent scale-to-zero during process execution Mar 11, 2026
@drappier-charles
drappier-charles merged commit 7f8612d into main Mar 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants