Array-job mode leaks scheduler -j slots and deadlocks ("Selected jobs: 0" with many ready jobs)

[ScottMastro]

I was having an issue where when using array jobs, it would schedule a burst of jobs then dwindle down without more jobs being issued. Disclaimer: I used AI (Claude Fable) to read though the code and identify the bug. It did find a scheduling issue so I wanted to share it. Below is an AI generated report of the issue.

Summary

When submitting with --slurm-array-jobs (e.g. all), a wide, embarrassingly-parallel workflow wedges: the Snakemake scheduler reports hundreds of ready jobs but selects 0, only a handful of jobs actually run on the cluster, and the run hangs indefinitely. The cause is that run_jobs() silently drops a rule's job batch when it is smaller than the rule's ready count, but those jobs have already consumed the scheduler's -j concurrency budget (_nodes). Dropped jobs are never submitted and never re-offered, so each wide wave permanently leaks slots until _nodes reaches 0 and nothing can be scheduled.

This only affects array mode. Individual (non-array) submission is unaffected.

Version

• snakemake-executor-plugin-slurm 2.7.1
• Array feature introduced in feat: job arrays #174 (commit a51a5bf); the offending code has been present since.

Symptoms

Controller log during the stall:

Ready jobs: 307
Selected jobs: 0
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 0, '_job_count': 100}
Resources after job selection:  {..., '_nodes': 0, ...}
Waiting for running jobs to complete.

Simultaneously the plugin reports only a handful of active jobs (Checking the status of 11 active jobs) and squeue shows ~16 — far fewer than -j. The trajectory is a big first burst, then a sawtooth that decays to a hard wedge, consistent with slots leaking a chunk per wave.

_nodes here is Snakemake's -j budget (1 unit per job in cluster mode), not physical nodes.

Root cause

The scheduler spends a job's _nodes slot at selection time, before the executor is called:

• job_scheduler.py: job_selector → update_available_resources(selected) subtracts _nodes per selected job.
• The scheduler then does running.update(run) and dag.register_running(run) (which removes the jobs from the DAG ready set) before calling executor.run_jobs(run).

The only paths that return a _nodes slot are _finish_jobs (success) and _handle_error (failure) — both require a SubmittedJobInfo, i.e. a job that was actually submitted via sbatch. A job the executor silently drops therefore can never free its slot, and nothing re-offers it (the scheduler skips jobs already in running, and the only re-add path is the error-restart, unreachable for a job that was never submitted).

In the plugin, run_jobs() (array branch) defers — via continue with no accumulator and no put-back — whenever the batch it received is smaller than the rule's ready count and smaller than the chunk size:

https://github.com/snakemake/snakemake-executor-plugin-slurm/blob/v2.7.1/snakemake_executor_plugin_slurm/__init__.py#L716-L725

else:
    if (
        len(same_rule_jobs) < eligible_jobs
        and len(same_rule_jobs) < chunk_size
    ):
        self.logger.debug(
            "Array job collection incomplete for rule "
            f"{rule_name}: {len(same_rule_jobs)}/{eligible_jobs} "
            "arrived (< chunk size), waiting for more jobs."
        )
        continue   # <-- these jobs are never submitted and never re-offered

The single-job array branch has the same defect (it logs "Waiting for at least one full chunk" and submits nothing).

The comment "waiting for more jobs" assumes the scheduler will re-offer the deferred jobs on a later round. It does not — each job is handed to run_jobs() exactly once. The batch is simply lost, and its _nodes slots with it.

Why it triggers reliably

The default --max-jobs-per-timespan 100/1s caps each selection round to 100 jobs (visible as _job_count: 100). Any rule with more than 100 ready jobs is therefore guaranteed to arrive as a partial batch, hit the continue, and leak up to 100 slots per wave. With -j 120, two or three such waves exhaust _nodes.

Reproduction

1. A workflow with a single rule that fans out to more ready jobs than -j (e.g. 300+ independent jobs from one rule).
2. Run with --executor slurm --slurm-array-jobs all -j 120 (default --max-jobs-per-timespan 100/1s).
3. Observe: a first burst runs, then Selected jobs: 0 with hundreds ready and _nodes: 0; the run hangs.

Workaround (no patch)

Raise both -j above the peak per-rule ready width and --max-jobs-per-timespan so a rule's full ready set arrives in one round (e.g. -j 1000 --max-jobs-per-timespan 1000/1s). Raising -j alone is not enough — the 100/1s rate bucket keeps batches partial. (Or simply don't use array mode.)

Suggested fix

Every job handed to run_jobs() has already been selected and had its slot spent, so it must be submitted this round — the collection-wait cannot be correct as written. Since run_array_jobs() already chunks internally by max_array_size, submitting whatever arrived each round is safe (you get several medium arrays per rule instead of one maximal array, and lose only some array-size optimality). Concretely, remove both deferral branches so a partial batch of >1 job goes to run_array_jobs() and a lone job goes to run_job().

If preserving array-size optimality is desired, the deferred jobs would have to be released back to the scheduler (re-added to the ready set and their _nodes refunded) rather than silently dropped — but simply submitting each round is far simpler and removes the deadlock.