[16.0-stable] pillar/containerd: drop bogus scheduler from debug container exec - #6246
Draft
christoph-zededa wants to merge 1 commit into
Conversation
The process spec used for execs into the debug container sets Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects that in ToSchedAttr(), so the container init fails before it can execute the requested program: OCI runtime exec failed: exec failed: unable to start container process: invalid scheduler policy:: unknown Both users of RunInDebugContainer() are affected: the collect-info run triggered through a local operator console (collectinfo agent) and the bpftrace endpoint of pillar's http-debug interface. The spec has looked like this since the bpftrace interface was added, but runc ignored process.scheduler on the exec path until v1.3.0-rc1 (runc#4585), so it stayed unnoticed until the rootfs moved from runc 1.1.12 to 1.3.3 in aa18688 ("bump runc to v3.3.0, containerd to v2.2.0; addresses critical CVEs"). Rather than completing the scheduler spec, drop it. Scheduler.Deadline is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on how long a process may run, and it was assigned a unix timestamp in seconds, so it never did what it looks like it does - the kernel does not kill tasks for missing a deadline either. The timeout is enforced by RunInDebugContainer() itself, which kills the process once its timer fires. Without process.scheduler runc leaves the exec'ed process with the scheduling attributes it inherits from the debug container, which is what CtrExec() relies on as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com> (cherry picked from commit b7d4463)
9 tasks
This was referenced Jul 31, 2026
github-actions
Bot
requested review from
deitch,
eriknordmark,
europaul and
rouming
July 31, 2026 16:07
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 16.0-stable #6246 +/- ##
===============================================
+ Coverage 19.52% 23.55% +4.02%
===============================================
Files 19 19
Lines 3021 2509 -512
===============================================
+ Hits 590 591 +1
+ Misses 2310 1787 -523
- Partials 121 131 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6231 to
16.0-stable.The process spec used for execs into the debug container sets
Scheduler.Deadline, but leavesScheduler.Policyempty. runc rejects that inToSchedAttr(), so the container init fails before it can execute the requestedprogram:
Both users of
RunInDebugContainer()are affected on this branch:agent) — the bug was reported from the field on 17.0.0-rc5, but this branch
carries the same defect
runc ignored
process.scheduleron the exec path until v1.3.0-rc1(opencontainers/runc#4585), so the bogus spec stayed unnoticed until the rootfs
moved from runc 1.1.12 to 1.3.3. This branch ships runc 1.3.3, so it is
affected.
Rather than completing the scheduler spec, this drops it.
Scheduler.Deadlineis a
SCHED_DEADLINEbandwidth parameter in nanoseconds, not a limit on howlong a process may run, and it was assigned a unix timestamp in seconds, so it
never did what it looks like it does — the kernel does not kill tasks for
missing a deadline either. The timeout that matters is enforced by
RunInDebugContainer()itself, which kills the process once its timer fires.See #6231 for the full rationale.
Cherry-picked with
git cherry-pick -xfromb7d446377a5a69712d9c1f57427961b9bdc042aa(the squash-merge of #6231 onmaster). It applies cleanly and with no adaptation — the hunk inpkg/pillar/containerd/run.gois identical on this branch.How to test and validate this PR
Not covered by an automated test — the path needs a real containerd/runc, and
there is currently no unit or Eden coverage for execs into the debug container.
Reproduction of the bug (fails without this PR):
the local operator console.
running [/usr/bin/collect-info.sh -u http://…] failed: process start failed: … invalid scheduler policy:: unknownand nothing is uploaded.
collect-info.shruns and its tarball appears on thedatastore.
Please verify the tarball actually lands on the datastore rather than relying on
the pillar log line:
RunInDebugContainer()reports failure whenever the scriptwrites anything to stderr (likely even on a successful run) and reports success
when the 15 min timeout kills the script, and
collect-info.shexits 0 evenwhen the upload itself fails. Those reporting bugs are pre-existing and out of
scope here.
A second, quicker check that exercises the same code path (no datastore
needed) — run a bpftrace script through pillar's http-debug interface:
Without this PR the response is
Error happened: process start failed: … invalid scheduler policy:: unknown;with it the program runs and its JSON output is returned.
Note that
bpftrace-compiler run-via-sshdoes not exercise this path andworks either way: sshd runs inside the debug container itself, so the program
is started as a plain child of the ssh session, without an OCI exec at all.
Only
run-via-httpandrun-via-edgeviewPOST to/debug/bpftraceandtherefore go through
RunInDebugContainer().Changelog notes
Fixed collect info triggered from a local operator console, and pillar's
bpftrace debug endpoint, which both failed to start with an
invalid scheduler policyerror from the container runtime.PR Backports
All four current LTS branches ship runc 1.3.3, so all of them are affected
(verified in #6231 by extracting
/usr/bin/runcfrom thelinuxkit/runcimageeach branch pins). This PR is one of a set of four:
neither
containerd/run.gonor the collectinfo agent.Checklist
or config surface changes
identical change was verified on amd64 in pillar/containerd: drop bogus scheduler from debug container exec #6231
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.
🤖 Generated with Claude Code