[AOT] Add resilient parallel job scheduler - #1038
Open
zhiding512 wants to merge 11 commits into
Open
Conversation
Centralize fork-based AOT job orchestration and environment controls so downstream integrations no longer maintain their own process pools. Co-authored-by: Cursor <cursoragent@cursor.com>
Rename the helper for clearer call sites, keep the package version unchanged, and strengthen failure and environment-control coverage. Co-authored-by: Cursor <cursoragent@cursor.com>
Make memory limiting fail safe, back off after possible OOM kills, and preserve structured failure causes so large AOT builds remain diagnosable. Co-authored-by: Cursor <cursoragent@cursor.com>
Align automatic worker semantics, queue retries fairly behind pending jobs, and route precise scheduler diagnostics through the FlyDSL logger. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a reusable, fork-based parallel job scheduler for AOT compilation, adds environment-variable configuration for worker/memory/timeout/retry behavior, and documents the new knobs and failure semantics.
Changes:
- Added
flydsl.utils.parallel.run_parallel_jobsimplementing a forked worker pool with per-job timeouts, retry logic, adaptive concurrency backoff on possible OOM exits, and structured JSON failure results. - Added backend-agnostic unit tests covering ordering, bounded concurrency, retry FIFO behavior, timeout killing, and failure categorization.
- Added AOT env options (
FLYDSL_AOT_*), documented them intests/README.md, and addedpsutilas a dependency to support automatic memory-based worker limiting.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
python/flydsl/utils/parallel.py |
New fork-based parallel scheduler with file-based result transport, retries/timeouts, and structured failure reporting. |
python/flydsl/utils/env.py |
Adds AOT environment manager/options and introduces OptFloat via refactoring numeric env parsing. |
tests/unit/test_parallel_jobs.py |
New unit tests validating scheduler behavior across success, crash/signal/timeout, retries, and logging. |
tests/README.md |
Documents new AOT environment variables and scheduler failure/result semantics. |
requirements.txt |
Adds psutil dependency used for automatic memory-based worker limiting. |
MANIFEST.in |
Ensures requirements.txt is included in source distributions. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+277
to
+283
| workers = OptInt( | ||
| 0, | ||
| description=( | ||
| "Maximum concurrent worker processes; unset, empty, or non-positive values use the CPU and " | ||
| "available-memory based automatic limit" | ||
| ), | ||
| ) |
Avoid adding a package-wide runtime dependency while retaining explicit guidance for automatic AOT memory limiting. Co-authored-by: Cursor <cursoragent@cursor.com>
Trust atomically committed results, bound exception diagnostics, and make OOM/timeout backoff generation-aware with additive recovery. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep same-wave OOM siblings retryable, recover from all healthy completions, leave timeouts concurrency-neutral, and allow AOT to continue without psutil. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep retries concurrency-neutral, use explicit success accounting, and fall back to a conservative four-worker limit when memory information is unavailable. Co-authored-by: Cursor <cursoragent@cursor.com>
Localize malformed worker payloads, validate summaries before logging, clarify fork constraints, and add exact boundary and retry-exhaustion tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove lower-priority and overlapping cases while retaining the 19 tests that protect correctness, failure isolation, and memory safety. Co-authored-by: Cursor <cursoragent@cursor.com>
Make the typed environment option return its default for empty numeric values so scheduler semantics match the documented contract directly. Co-authored-by: Cursor <cursoragent@cursor.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
python -m pytest --confcutdir=tests/unit tests/unit/test_parallel_jobs.py -q(24 passed)Made with Cursor