libaio: implement the Linux async I/O interface#1433
Open
gburd wants to merge 2 commits into
Open
Conversation
libaio was a landmine: io_setup() returned success, but io_submit(), io_getevents(), io_destroy() and io_cancel() were UNIMPL() stubs that called abort(). A program that probed libaio, saw io_setup() succeed, and then submitted any I/O would crash the whole unikernel. Implement the interface for real. OSv has no user/kernel boundary, so "async" I/O is done the same way core/io_uring.cc does it: each submitted iocb runs on a short-lived detached worker thread that performs the blocking read/write/fsync through the VFS (sys_read/sys_write/sys_fsync, the exact primitives io_uring uses), then queues an io_event and wakes any thread parked in io_getevents(). - io_setup(): allocate a real io_context capped at nr_events in-flight ops. - io_submit(): dispatch each iocb to a detached worker; honor the in-flight cap (partial count if some were accepted, otherwise EAGAIN, matching Linux). Supports PREAD, PWRITE, PREADV, PWRITEV, FSYNC, FDSYNC, NOOP; unknown opcodes complete with res == -EINVAL rather than aborting. - io_getevents(): block until min_nr events are available or the (optional) timeout elapses, then copy out up to nr events. A bad fd surfaces as res == -EBADF in the event, not a crash. - io_destroy(): mark the context draining and wait for all in-flight ops to retire (detached workers self-reap) before freeing, so no worker can touch a freed context. - io_cancel(): return EINVAL. Ops run to completion on a worker with no safe mid-flight cancellation point; EINVAL is a valid Linux response. Best-effort eventfd notification is delivered when an iocb sets IOCB_FLAG_RESFD. Fill in include/api/libaio.h with the real Linux aio ABI (struct iocb, struct io_event, IO_CMD_* opcodes, IOCB_FLAG_*), which was previously only forward-declared. Add tests/tst-libaio.cc covering the write/read round-trip (data and aio_data echo), an 8-op batch, bad-fd (-EBADF, no abort), and the io_getevents timeout path. Passes on OSv under KVM, single- and multi-vCPU.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements the Linux async I/O interface (libaio), which was a latent crash
hazard.
io_setup()returned success, butio_submit(),io_getevents(),io_destroy()andio_cancel()wereUNIMPL()stubs that calledabort(). Aprogram that probed libaio, saw
io_setup()succeed, and then submitted any I/Owould take down the whole unikernel.
How
OSv has no user/kernel boundary, so "async" I/O is done the same way
core/io_uring.ccdoes it: each submitted iocb runs on a short-lived detachedworker thread that performs the blocking read/write/fsync through the VFS
(
sys_read/sys_write/sys_fsync, the same primitives io_uring uses), thenqueues an
io_eventand wakes any thread parked inio_getevents().io_setup(): allocate a realio_contextcapped atnr_eventsin-flight ops.io_submit(): dispatch each iocb to a detached worker; honor the in-flight cap(partial count if some were accepted, otherwise
EAGAIN, matching Linux).Supports
PREAD,PWRITE,PREADV,PWRITEV,FSYNC,FDSYNC,NOOP;unknown opcodes complete with
res == -EINVALrather than aborting.io_getevents(): block untilmin_nrevents are available or the optionaltimeout elapses, then copy out up to
nrevents. A bad fd surfaces asres == -EBADFin the event, not a crash.io_destroy(): mark the context draining and wait for all in-flight ops toretire (detached workers self-reap) before freeing, so no worker can touch a
freed context.
io_cancel(): returnEINVAL. Ops run to completion on a worker with no safemid-flight cancellation point;
EINVALis a valid Linux response.Best-effort eventfd notification is delivered when an iocb sets
IOCB_FLAG_RESFD.include/api/libaio.his filled in with the real Linux aio ABI (struct iocb,struct io_event,IO_CMD_*opcodes,IOCB_FLAG_*), which was previously onlyforward-declared.
Testing
tests/tst-libaio.cccovers the write/read round-trip (data andaio_dataecho), an 8-op batch, bad-fd (
-EBADF, no abort), and theio_geteventstimeout path. It passes on OSv under KVM on both single- and multi-vCPU
configurations.
Note
This is a self-contained worker-thread-per-iocb design, which is simple and
correct for typical libaio concurrency. If a caller submits very large batches
and thread-creation cost shows up, the worker loop can be swapped for a bounded
pool like the one in
core/io_uring.cc; that is left as a follow-up.(Recreated from #1413, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)