Skip to content

orchestrate.py uses global started_at for per-kernel time budget, causing premature termination of later kernels #12

Description

@yu200512

Description

In orchestrate.py, cmd_record() computes time_spent_minutes for each kernel using state["started_at"] -- the timestamp when Phase B began (the entire optimization run):

# orchestrate.py line 534-546 (current upstream)
started = state.get("started_at")
if started:
    start_dt = datetime.fromisoformat(started)
    ...
    delta = now_dt - start_dt
    target["time_spent_minutes"] = round(delta.total_seconds() / 60.0)

This means kernel #5's elapsed time includes ALL time spent on kernels #1-#4. With a max_minutes_per_kernel of 120, kernel #5 can inherit 100+ minutes before running its first experiment, hitting the time budget almost immediately.

Reproduce

Run optimization on any model with 5+ kernels:

uv run python auto_optimizer.py --optimize-only ... --iterations 300

Observe:

Expected

Per-kernel time budget should start counting from when the kernel begins optimization, not from Phase B start. Kernel #5 should get as much exploration time as kernel #1.

Root Cause

Single global started_at used for all kernels. No per-kernel start timestamp.

Fix

Add kernel_started_at field to each kernel entry, set on transition. Compute time_spent_minutes from kernel_started_at instead of started_at.


Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions