AWS S3 filesystem - #2177
Open
francescolavra wants to merge 6 commits into
Open
Conversation
added 6 commits
August 2, 2026 16:11
The response to an HTTP HEAD request always has an empty body, even if the Content-Length header has a non-zero value. The HTTP parser invokes the handler closure only after receiving a response body matching the Content-Length value, and therefore stalls when receiving a response to a HEAD request. Add a `headers_only` parameter to allocate_http_parser(), and skip reception of a response body when the body is not expected.
When creating the canonical request string that is needed to sign an AWS API request, any parameters in the query string must be URI-encoded, otherwise the AWS API server rejects the request due to a signature mismatch.
Some AWS API requests (including requests to the S3 API) require an `x-amz-content-sha256` request header, whose value is the SHA256 hash of the request body (or the "UNSIGNED-PAYLOAD" special value). The content hash value that is inserted in the canonical request string must match the value of this header (when present). When creating the canonical request string, check for the presence of the `x-amz-content-sha256` header, and copy the hash value from the header (if present) instead of (re-)computing the hash from the request body.
Some attempts to rename filesystem entries (such as for example renaming an existing file to an existing directory) are defined as invalid by POSIX, and therefore should be rejected. The kernel enforces these rules in each filesystem implementation, except for remote filesystems (such as 9P) where it relies on the remote party to enforce the rules. However, a remote filesystem may not be fully POSIX-compliant; even in this case, POSIX rules should still be enforced. Move the checks for valid rename operations to generic filesystem code, so that POSIX rules can be enforced also for remote filesystems where the remote party does not enforce them. This will be needed when implementing the AWS S3 filesystem.
This function is used to deallocate a filesystem metadata tuple that may be linked to other filesystem entries. Move it to common filesystem code so that it will be re-used when implementing the AWS S3 filesystem.
This filesystem allows accessing objects in an AWS S3 bucket as regular
files. It is implemented as a klib (with a dependency on the existing tls
klib), and is configured as shown in the following Ops configuration
snippet:
```
"Klibs": ["s3fs", "tls"],
"ManifestPassthrough": {
"s3fs": {
"region": "us-west-1",
"access_key": "MY_ACCESS_KEY",
"secret": "my_secret",
"buckets": [
{
"name": "my-bucket",
"mount": "/mnt"
}
]
}
}
```
It is possible to specify multiple S3 buckets, each with its own mount
point, as long as all buckets are registered in the same AWS region and
accessible with the same credentials. All mount points must be existing
directories.
This filesystem allows reading and writing existing S3 objects, as well as
creating new objects, using the AWS S3 API over HTTPS. Directories are
emulated as zero-length objects whose name ends with '/'.
The current implementation has the following limitations:
- writing to an object at a non-zero offset is not supported
- each write replaces the entire contents of an object, i.e. a write
modifies the object size to match the length of the data being written
- it is not possible to write objects larger than 4 KB, unless direct I/O
(O_DIRECT flag) is used
- it is not possible to use the writev and pwritev syscalls when using
direct I/O
- symbolic links are not supported
- renaming objects using the RENAME_EXCHANGE flag is not supported
- it is not possible to truncate an object to a non-zero length
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change set adds support for a new filesystem type that allows accessing objects in an AWS S3 bucket as regular files.
This filesystem is implemented as a klib (with a dependency on the existing tls klib), and is configured as shown in the following Ops configuration snippet:
It is possible to specify multiple S3 buckets, each with its own mount point, as long as all buckets are registered in the same AWS region and accessible with the same credentials. All mount points must be existing directories.
This filesystem allows reading and writing existing S3 objects, as well as creating new objects, using the AWS S3 API over HTTPS. Directories are emulated as zero-length objects whose name ends with '/'.
The current implementation has the following limitations: