Skip to content

[16.0-stable] pillar/containerd: drop bogus scheduler from debug container exec - #6246

Draft
christoph-zededa wants to merge 1 commit into
lf-edge:16.0-stablefrom
christoph-zededa:16.0-collectinfo_edgesync_fix_scheduler_policy
Draft

[16.0-stable] pillar/containerd: drop bogus scheduler from debug container exec#6246
christoph-zededa wants to merge 1 commit into
lf-edge:16.0-stablefrom
christoph-zededa:16.0-collectinfo_edgesync_fix_scheduler_policy

Conversation

@christoph-zededa

@christoph-zededa christoph-zededa commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Description

Backport of #6231 to 16.0-stable.

The process spec used for execs into the debug container sets
Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects that in
ToSchedAttr(), so the container init fails before it can execute the requested
program:

OCI runtime exec failed: exec failed: unable to start container process: invalid scheduler policy:: unknown

Both users of RunInDebugContainer() are affected on this branch:

  • the collect-info run triggered through a local operator console (collectinfo
    agent) — the bug was reported from the field on 17.0.0-rc5, but this branch
    carries the same defect
  • the bpftrace endpoint of pillar's http-debug interface

runc ignored process.scheduler on the exec path until v1.3.0-rc1
(opencontainers/runc#4585), so the bogus spec stayed unnoticed until the rootfs
moved from runc 1.1.12 to 1.3.3. This branch ships runc 1.3.3, so it is
affected.

Rather than completing the scheduler spec, this drops it. Scheduler.Deadline
is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on how
long a process may run, and it was assigned a unix timestamp in seconds, so it
never did what it looks like it does — the kernel does not kill tasks for
missing a deadline either. The timeout that matters is enforced by
RunInDebugContainer() itself, which kills the process once its timer fires.
See #6231 for the full rationale.

Cherry-picked with git cherry-pick -x from
b7d446377a5a69712d9c1f57427961b9bdc042aa (the squash-merge of #6231 on
master). It applies cleanly and with no adaptation — the hunk in
pkg/pillar/containerd/run.go is identical on this branch.

How to test and validate this PR

Not covered by an automated test — the path needs a real containerd/runc, and
there is currently no unit or Eden coverage for execs into the debug container.

Reproduction of the bug (fails without this PR):

  1. Configure a collect-info datastore of type HTTP/HTTPS with auth enabled on
    the local operator console.
  2. Trigger collect info for the device.
  3. Without this PR, pillar logs
    running [/usr/bin/collect-info.sh -u http://…] failed: process start failed: … invalid scheduler policy:: unknown
    and nothing is uploaded.
  4. With this PR, collect-info.sh runs and its tarball appears on the
    datastore.

Please verify the tarball actually lands on the datastore rather than relying on
the pillar log line: RunInDebugContainer() reports failure whenever the script
writes anything to stderr (likely even on a successful run) and reports success
when the 15 min timeout kills the script, and collect-info.sh exits 0 even
when the upload itself fails. Those reporting bugs are pre-existing and out of
scope here.

A second, quicker check that exercises the same code path (no datastore
needed) — run a bpftrace script through pillar's http-debug interface:

# on the device
eve http-debug
# on the dev host - the debug server only listens on localhost, so tunnel it
ssh -p 2222 -L 6543:localhost:6543 root@127.1
bpftrace-compiler run-via-http 127.1:6543 examples/opensnoop.bt

Without this PR the response is
Error happened: process start failed: … invalid scheduler policy:: unknown;
with it the program runs and its JSON output is returned.

Note that bpftrace-compiler run-via-ssh does not exercise this path and
works either way: sshd runs inside the debug container itself, so the program
is started as a plain child of the ssh session, without an OCI exec at all.
Only run-via-http and run-via-edgeview POST to /debug/bpftrace and
therefore go through RunInDebugContainer().

Changelog notes

Fixed collect info triggered from a local operator console, and pillar's
bpftrace debug endpoint, which both failed to start with an
invalid scheduler policy error from the container runtime.

PR Backports

All four current LTS branches ship runc 1.3.3, so all of them are affected
(verified in #6231 by extracting /usr/bin/runc from the linuxkit/runc image
each branch pins). This PR is one of a set of four:

Checklist

  • I've provided a proper description
  • I've added the proper documentation — not needed, no user-facing behaviour
    or config surface changes
  • I've tested my PR on amd64 device — not re-tested on this branch; the
    identical change was verified on amd64 in pillar/containerd: drop bogus scheduler from debug container exec #6231
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

The process spec used for execs into the debug container sets
Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects
that in ToSchedAttr(), so the container init fails before it can
execute the requested program:

  OCI runtime exec failed: exec failed: unable to start container
  process: invalid scheduler policy:: unknown

Both users of RunInDebugContainer() are affected: the collect-info run
triggered through a local operator console (collectinfo agent) and the
bpftrace endpoint of pillar's http-debug interface.

The spec has looked like this since the bpftrace interface was added,
but runc ignored process.scheduler on the exec path until v1.3.0-rc1
(runc#4585), so it stayed unnoticed until the rootfs moved from runc
1.1.12 to 1.3.3 in aa18688 ("bump runc to v3.3.0, containerd to
v2.2.0; addresses critical CVEs").

Rather than completing the scheduler spec, drop it. Scheduler.Deadline
is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on
how long a process may run, and it was assigned a unix timestamp in
seconds, so it never did what it looks like it does - the kernel does
not kill tasks for missing a deadline either. The timeout is enforced
by RunInDebugContainer() itself, which kills the process once its timer
fires. Without process.scheduler runc leaves the exec'ed process with
the scheduling attributes it inherits from the debug container, which
is what CtrExec() relies on as well.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit b7d4463)
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 23.55%. Comparing base (9f69150) to head (62a906b).
⚠️ Report is 138 commits behind head on 16.0-stable.

Additional details and impacted files
@@               Coverage Diff               @@
##           16.0-stable    #6246      +/-   ##
===============================================
+ Coverage        19.52%   23.55%   +4.02%     
===============================================
  Files               19       19              
  Lines             3021     2509     -512     
===============================================
+ Hits               590      591       +1     
+ Misses            2310     1787     -523     
- Partials           121      131      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant