Reduce container image pull time by supporting partial pulls

[sanmai-NL]

Is your feature request related to a problem? Please describe.
Pulling in MegaLinter container images can take a long time, as noted in the docs.

Describe the solution you'd like
Experiment with eStargz lazy pulling.

Describe alternatives you've considered
All available alternatives have been explored and somewhat implemented, like splitting out images, using stages and caching.

Additional context
None.

[sanmai-NL]

See: https://www.redhat.com/sysadmin/faster-container-image-pulls. zstd:chunked is an alternative to eStargz.

[nvuillam]

I am still almost a newbie about advanced docker usage... I let the experts give their opinion ^^ cc @Kurt-von-Laven @bdovaz @echoix

[echoix]

I read just a little on this this weekend by following the links you provided.
I agree that our kind of image would be a good candidate to that kind of optimisation. It merits a little proof of concept. Maybe the zstd chunked might have a better success (since it compresses more).

Do you know if there is any more recent activity in this field since 2021? Most articles available for eStargz seem to be released in the same time frame.

In all cases, time to support it would be worth it only if it can be used by GitHub actions (and it works better for a runner with two cores, the most common usage of MegaLinter).

Before reading I was thinking that it was another "Docker optimizing SaaS" that was thinking to dynamically remove files, and using a shallow image to pull the rest when already started. But it isn't the case, and seems to be able to be compatible with existing clients (without any advantage) since it is more of a metadata and client workaround. Both options seems to still work as OCI containers.

Either way, if you'd like to play a little with it and look at where it could be supported I'll be glad to look back to it. Or other usages of it in the wild

[sanmai-NL]

@echoix Advised is to use zstd:chunked now, rather than eStargz. I use it on a private GitLab, and their Container Registry seems to accept these images fine. However, Podman needs a setting that's toggled off by default to take advantage of partial pulls. Docker (dockerd) unfortunately seems to have fallen behind on this feature. Nevertheless, it is supported under e.g., Kubernetes and container engine products so users of those products and MegaLinter will still be able to take advantage.

See https://www.slideshare.net/KoheiTokunaga/starting-up-containers-super-fast-with-lazy-pulling-of-images for a (somewhat dated) overview of this topic.

[echoix]

I explored and made a little research during small free times this week. I learned that docker is supposed to use pigz if available when pulling to decompress layers (ie unpigz is available). I tried to see if it could be used in GitHub actions (on my fork), but didn't manage to get a conclusive result since pulling the Docker image by the runner happens before I can execute a step to install pigz. When using composite actions, I don't seem to be able to specify the correct usage to the docker image, and running it manually seems to be missing context, so it seems I didn't feed the full command line and all the features GitHub actions seem to provide, so it maybe not be a good idea for general usage. If you want to test, on GitHub actions, updating apt and installing pigz takes 9 seconds, and the baseline for pulling the megalinter python beta flavor took 43-45 seconds, so the time to beat if pigz ever helped, was to reduce by more than 9 seconds. I think I read in issues and discussions that some people had like 18-23% improvement with multiple cores but I can't find back the source now :(. In that case of the Python flavor, it's no gain (43x0.23=9.89 sec) if it ever works. And decompression in Docker stays serial between layers, but a layer can be decompressed in parallel.
In summary, configuration of the Docker used by a GitHub hosted runner for an action doesn't seem easily feasible.

Next, I looked at the status of zstd compression. It could be advantageous to have the size of the download and the speed of Other container software implementations for OCI (other than Docker) seem to have it figured out for a while.
For Docker, support for pulling images where layers are compressed as zstd is finally available in version 23.0.0 (released on 2023-02-01 https://docs.docker.com/engine/release-notes/23.0/#new). (Version 24.0.0 was already released on 2023-05-16). But, it is not installed in the Ubuntu 22.04 of the GitHub hosted runners yet. They use Docker-Moby Client 20.10.25+azure-2 and Docker-Moby Server 20.10.25+azure-2 (https://github.com/actions/runner-images/blob/2ccef4f1005354274efa8c006927aec8601150d4/images/linux/Ubuntu2204-Readme.md#tools). A recent discussion talked about this: actions/runner-images#7770
If the path would be clear, my idea was to use zstd-compressed layers as https://aws.amazon.com/fr/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/ or https://docs.docker.com/build/exporters/#compression. That means we would be a fully OCI image that doesn't seem to be a problem on itself, we are playing with it for multiplatform image support and being able to load them locally. But we saw that at least GitHub runners wouldn't work for now, and even if they would, we would need to require recent versions of Docker or any other container runner for everyone, checking many platforms. Side note, that issue seem to indicate that on macOS, the Docker doesn't support zstd yet. I don't have a macOS computer available to check that affirmation, nor a M1/M2 based one to further check (as we know we have users using it locally on these platforms). But if there isn't any reasons that macOS is different, it should've worked with version 23.0.0, released in February before the issue (of the end of April), so unexpected.

I saw a lot of promising tricks that could be used by the usage of the containerd container store if the transition of Docker continues as it is going. But now it's beta/experimental and doesn't work by default, and I don't think we can expect it from our users yet.

So overall, I'm now aware of what exists, but we might just be a little too early for a mass usage.

[nvuillam]

Many thanks @echoix for your great analysis :)

[echoix]

I think we could close for now, and ping us back in another issue if ever the ecosystem changes and we don't get a word of it yet. But at least for now I know to keep a look at this optimisation in the future.
Your point has been made.

[sanmai-NL]

@echoix I'm mostly focused on user value in my proposal. If building and pushing a zstd chunked compressed image is possible, the rest doesn't matter so much, does it? If needed at all, it'd be easily possible to run Podman in CI without degrading Docker Engine related GitHub Actions, isn't it? See e.g., https://github.com/redhat-actions/push-to-registry

[echoix]

Building and pushing such image isn't a problem, I think it's pretty possible to do so. But it wouldn't be consumable by the main utilisation source, GitHub actions. At least for now.

[sanmai-NL]

Oh, we use MegaLinter on GitLab CI.

By the way, you can run Podman from a container. Should be possible under GitHub Actions, too, no?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[mdrocan]

A slim-image like the super-linter has could be useful. Of course it would mean making compromises with what is included to the image and what not.