Skip to content

libc: make setrlimit/getrlimit consistent and stop aborting#1435

Open
gburd wants to merge 2 commits into
cloudius-systems:masterfrom
gburd:pr/setrlimit
Open

libc: make setrlimit/getrlimit consistent and stop aborting#1435
gburd wants to merge 2 commits into
cloudius-systems:masterfrom
gburd:pr/setrlimit

Conversation

@gburd

@gburd gburd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What

Fixes two problems with the resource-limit functions:

  1. getrlimit() called abort() on any resource it did not special-case
    (RLIMIT_LOCKS, RLIMIT_SIGPENDING, RLIMIT_RTPRIO, RLIMIT_RTTIME). A
    program querying one of those crashed the whole unikernel, the same class of
    hazard as the old libaio stubs.

  2. setrlimit() was a no-op that returned success, so a later getrlimit()
    reported the old (usually infinite) value rather than what the application
    just set, an inconsistency some programs depend on.

How

Replace the getrlimit() switch with a process-wide stored table of
RLIMIT_NLIMITS entries, initialized once with the previous defaults (stack
size from the default pthread attr, RLIMIT_NOFILE 10240, RLIMIT_NICE 0,
everything else infinite). getrlimit() reads the table, setrlimit() writes
it, both guarded by a mutex. Out-of-range resources and rlim_cur > rlim_max
return EINVAL instead of crashing or being silently accepted.

The limits remain advisory (OSv is a single address space and does not enforce
most of them), but set and get are now consistent.

Wire setrlimit as syscall SYS_setrlimit (only getrlimit and prlimit64
were in the table) with a tracepoint.

Testing

tests/tst-rlimit.cc covers querying every resource without aborting, the
EINVAL paths, set/get round-trips, and prlimit. Passes on OSv under KVM.

(Recreated from #1415, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)

gburd added 2 commits July 15, 2026 08:36
Two problems with the resource-limit functions:

1. getrlimit() called abort() on any resource it did not special-case
   (RLIMIT_LOCKS, RLIMIT_SIGPENDING, RLIMIT_RTPRIO, RLIMIT_RTTIME). A program
   querying one of those crashed the whole unikernel, the same class of
   landmine as the libaio stubs.

2. setrlimit() was a no-op that returned success, so a later getrlimit()
   reported the old (usually infinite) value rather than what the application
   just set, an inconsistency some programs depend on.

Replace the switch with a process-wide stored table of RLIMIT_NLIMITS entries,
initialized once with the previous defaults (stack size from the default
pthread attr, RLIMIT_NOFILE 10240, RLIMIT_NICE 0, everything else infinite).
getrlimit() reads the table, setrlimit() writes it, both guarded by a mutex.
Out-of-range resources and rlim_cur > rlim_max return EINVAL instead of
crashing or being silently accepted. The limits remain advisory (OSv is a
single address space and does not enforce most of them), but set and get are
now consistent.

Wire setrlimit as syscall SYS_setrlimit (only getrlimit and prlimit64 were in
the table) with a tracepoint.

Add tests/tst-rlimit.cc covering querying every resource without aborting,
the EINVAL paths, set/get round-trips, and prlimit. Passes on OSv under KVM.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant