Skip to content

libaio: implement the Linux async I/O interface#1433

Open
gburd wants to merge 2 commits into
cloudius-systems:masterfrom
gburd:pr/libaio
Open

libaio: implement the Linux async I/O interface#1433
gburd wants to merge 2 commits into
cloudius-systems:masterfrom
gburd:pr/libaio

Conversation

@gburd

@gburd gburd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What

Implements the Linux async I/O interface (libaio), which was a latent crash
hazard. io_setup() returned success, but io_submit(), io_getevents(),
io_destroy() and io_cancel() were UNIMPL() stubs that called abort(). A
program that probed libaio, saw io_setup() succeed, and then submitted any I/O
would take down the whole unikernel.

How

OSv has no user/kernel boundary, so "async" I/O is done the same way
core/io_uring.cc does it: each submitted iocb runs on a short-lived detached
worker thread that performs the blocking read/write/fsync through the VFS
(sys_read/sys_write/sys_fsync, the same primitives io_uring uses), then
queues an io_event and wakes any thread parked in io_getevents().

  • io_setup(): allocate a real io_context capped at nr_events in-flight ops.
  • io_submit(): dispatch each iocb to a detached worker; honor the in-flight cap
    (partial count if some were accepted, otherwise EAGAIN, matching Linux).
    Supports PREAD, PWRITE, PREADV, PWRITEV, FSYNC, FDSYNC, NOOP;
    unknown opcodes complete with res == -EINVAL rather than aborting.
  • io_getevents(): block until min_nr events are available or the optional
    timeout elapses, then copy out up to nr events. A bad fd surfaces as
    res == -EBADF in the event, not a crash.
  • io_destroy(): mark the context draining and wait for all in-flight ops to
    retire (detached workers self-reap) before freeing, so no worker can touch a
    freed context.
  • io_cancel(): return EINVAL. Ops run to completion on a worker with no safe
    mid-flight cancellation point; EINVAL is a valid Linux response.

Best-effort eventfd notification is delivered when an iocb sets
IOCB_FLAG_RESFD.

include/api/libaio.h is filled in with the real Linux aio ABI (struct iocb,
struct io_event, IO_CMD_* opcodes, IOCB_FLAG_*), which was previously only
forward-declared.

Testing

tests/tst-libaio.cc covers the write/read round-trip (data and aio_data
echo), an 8-op batch, bad-fd (-EBADF, no abort), and the io_getevents
timeout path. It passes on OSv under KVM on both single- and multi-vCPU
configurations.

Note

This is a self-contained worker-thread-per-iocb design, which is simple and
correct for typical libaio concurrency. If a caller submits very large batches
and thread-creation cost shows up, the worker loop can be swapped for a bounded
pool like the one in core/io_uring.cc; that is left as a follow-up.

(Recreated from #1413, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)

gburd added 2 commits July 15, 2026 08:35
libaio was a landmine: io_setup() returned success, but io_submit(),
io_getevents(), io_destroy() and io_cancel() were UNIMPL() stubs that
called abort(). A program that probed libaio, saw io_setup() succeed, and
then submitted any I/O would crash the whole unikernel.

Implement the interface for real. OSv has no user/kernel boundary, so
"async" I/O is done the same way core/io_uring.cc does it: each submitted
iocb runs on a short-lived detached worker thread that performs the
blocking read/write/fsync through the VFS (sys_read/sys_write/sys_fsync,
the exact primitives io_uring uses), then queues an io_event and wakes any
thread parked in io_getevents().

- io_setup(): allocate a real io_context capped at nr_events in-flight ops.
- io_submit(): dispatch each iocb to a detached worker; honor the in-flight
  cap (partial count if some were accepted, otherwise EAGAIN, matching
  Linux). Supports PREAD, PWRITE, PREADV, PWRITEV, FSYNC, FDSYNC, NOOP;
  unknown opcodes complete with res == -EINVAL rather than aborting.
- io_getevents(): block until min_nr events are available or the (optional)
  timeout elapses, then copy out up to nr events. A bad fd surfaces as
  res == -EBADF in the event, not a crash.
- io_destroy(): mark the context draining and wait for all in-flight ops to
  retire (detached workers self-reap) before freeing, so no worker can
  touch a freed context.
- io_cancel(): return EINVAL. Ops run to completion on a worker with no safe
  mid-flight cancellation point; EINVAL is a valid Linux response.

Best-effort eventfd notification is delivered when an iocb sets
IOCB_FLAG_RESFD.

Fill in include/api/libaio.h with the real Linux aio ABI (struct iocb,
struct io_event, IO_CMD_* opcodes, IOCB_FLAG_*), which was previously only
forward-declared.

Add tests/tst-libaio.cc covering the write/read round-trip (data and
aio_data echo), an 8-op batch, bad-fd (-EBADF, no abort), and the
io_getevents timeout path. Passes on OSv under KVM, single- and
multi-vCPU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant