skill-creator: fix parallel worker false negatives in run_eval.py - #794
Open
rhys-childs wants to merge 1 commit into
Open
skill-creator: fix parallel worker false negatives in run_eval.py#794rhys-childs wants to merge 1 commit into
rhys-childs wants to merge 1 commit into
Conversation
This was referenced Mar 27, 2026
Open
rhys-childs
force-pushed
the
fix/run-eval-parallel-false-negatives
branch
from
March 27, 2026 21:42
333d685 to
078c2fe
Compare
Each parallel worker created its own command file with a unique UUID suffix, then checked whether Claude invoked that specific UUID. When multiple identically-described command files coexisted on disk, Claude would pick one arbitrarily — but only the worker whose UUID was chosen reported a trigger. All other concurrent workers reported false negatives, producing artificially low trigger rates that scale inversely with --num-workers (e.g. ~33% at 3 workers, ~0% at 10). Fix: create a single shared command file in run_eval() before spawning workers, so every claude -p session sees exactly one command and all workers match against the same name. File cleanup moves to a finally block in run_eval() that runs after ProcessPoolExecutor.shutdown() guarantees all workers have completed. Related to anthropics#556
rhys-childs
force-pushed
the
fix/run-eval-parallel-false-negatives
branch
from
March 27, 2026 21:46
078c2fe to
f5b867e
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run_eval.pycreates a separate command file per parallel worker, each with a unique UUID. Claude picks one arbitrarily, but each worker only checks for its own UUID — so N-1 out of N workers report false negatives. At the default--num-workers 10, this produces the 0% trigger rate reported in #556.The fix creates a single shared command file before spawning workers. All workers match against the same name. Full parallelism is preserved.
Reproduction
30-query eval set (15 should-trigger, 15 should-not),
--runs-per-query 3,--num-workers 10(default):Concurrent eval runs for different skills were also tested — no cross-contamination.
What changes
run_single_querycreates/deletesrun_evalcreates/deletesrun_single_queryparamsskill_name,skill_descriptioncommand_nameRelated issues
anthropics/claude-plugins-officialand has the same bug.Fixes #556