image_load: 0.3.19 creates containerd image records the daemon cannot resolve (regression from #663)

[gergelyfabian]

Summary

Since v0.3.19, image_load into a Docker daemon that uses the containerd image
store creates an image record under the raw reference instead of the
canonical one. The record shows up in docker image ls, but docker image inspect and docker run cannot resolve the name, so the load looks successful
(exit 0, the tag is printed) while nothing can use the image.

v0.3.18 is the last working release; v0.3.19 is the first broken one.

Reproduction

MODULE.bazel:

module(name = "load_name_repro", version = "0.0.0")

bazel_dep(name = "rules_img", version = "0.3.19")

pull = use_repo_rule("@rules_img//img:pull.bzl", "pull")

pull(
    name = "alpine",
    digest = "sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1",
    registry = "index.docker.io",
    repository = "library/alpine",
    tag = "3.22",
)

BUILD.bazel:

load("@rules_img//img:load.bzl", "image_load")

image_load(
    name = "load",
    image = "@alpine//:image",
    tag = "my-app:latest",
)

$ bazel run :load
...
sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
my-app:latest
    blob transfers: 10 from disk, 0 from disk cache, 2 from container registry, ...
$ echo $?
0

$ docker image inspect my-app:latest
[]
Error response from daemon: No such image: my-app:latest

$ docker run --rm my-app:latest true
Unable to find image 'my-app:latest' locally
docker: Error response from daemon: pull access denied for my-app, ...

With only the version changed to 0.3.18, the same workspace works:

$ bazel run :load
...
docker.io/library/my-app:latest          # <- normalized name

$ docker image inspect my-app:latest >/dev/null && echo OK
OK
$ docker run --rm my-app:latest true; echo $?
0

Note the difference in the name the loader prints: my-app:latest (0.3.19) vs
docker.io/library/my-app:latest (0.3.18).

docker image ls is misleading here — it lists the unresolvable record (and can
show a duplicate row for the same tag, one of them with no ID/size in
docker image ls --tree), so the breakage only surfaces at inspect/run time.

Cause

img_tool/pkg/load/loader.go, loadContainerd, changed in #663
("fix(load): stop mangling image names that carry a registry port"):

 	for _, tag := range allTags {
-		normalizedTag := NormalizeDockerReference(tag)
 		img := containerd.Image{
-			Name:   normalizedTag,
+			Name:   tag,
 			Target: target,
 		}
 		_, err = imageService.Create(ctx, img)

Dropping the normalization is correct for the docker save tarball path, where
RepoTags should stay verbatim. For the containerd path it is not: the daemon
resolves image names as fully-qualified references, so a record named
my-app:latest is a different key from the docker.io/library/my-app:latest
that docker inspect/docker run/docker compose look up.

I suspect #663 was aimed at the localhost/ vs docker.io/ prefix question in
#563, but the containerd record needs a canonical name regardless of what the
tarball path does.

Suggested fix

Keep names verbatim for the tarball/docker save path, and normalize again for
the containerd image record — while preserving the actual fix from #663, i.e.
only qualify a reference that has no registry component, so
docker.example.com:1234/foo:latest keeps its port and is not mistaken for a
tag. NormalizeLoadReference in img_tool/pkg/api/deploy.go looks like the
natural place to distinguish "name as configured" from "name handed to the
containerd image service".

Environment

• rules_img 0.3.19 (broken), 0.3.18 (works)
• Bazel 9.2.0, bzlmod
• Docker 29.7.2 with the containerd image store
  (docker info → driver-type io.containerd.snapshotter.v1)
• linux/amd64

Only daemons using the containerd image store are affected; that is also why the
load reports success — the blobs and the record are written, just under a name
the daemon will not resolve.

[Harshal96]

@malt3

Would the right fix be to keep tags verbatim for the docker-save/tar path, but normalize only when writing containerd.Image.Name into the moby namespace? Probably using github.com/distribution/reference, since thats what Moby uses. This would also avoid bringing back the registry-port bug from before #663.

One question is whether this should apply to explicit daemon = "containerd" too, or only when a Docker load gets upgraded to the containerd fast path. If literal containerd names are part of the contract, we might need to keep the original daemon info around and only normalize Docker-originated loads.

I’ve traced this through the current rules_img, Moby and containerd sources, but havent reproduced it myself on a Linux Docker 29/containerd setup yet. Also havent checked nerdctl behavior, tag+digest refs, or whether storing both raw + canonical aliases causes duplicate image entries. So theres still a runtime/e2e gap here.

Does this sound like an issue worth fixing, or is requiring a fully qualified tag the intended behavior now?

[malt3]

Hi @gergelyfabian. Thanks for opening this issue and apologies for breaking your builds.
Have you considered changing the BUILD.bazel file to contain this instead?:

load("@rules_img//img:load.bzl", "image_load")

image_load(
    name = "load",
    image = "@alpine//:image",
    tag = "docker.io/library/my-app:latest",
)

I understand that changing the behaviour in rules_img is annoying, but trying to work around quirks (or bugs IMO) in different backends of docker / Moby is brittle as well.
You will have a much better time by just using a name that your container runtime can work with.
I'd argue that this is rather a bug in Moby, since an image gets loaded in a broken state and different backends of Docker appear to behave in different ways.
Also note that other container runtimes are able to use arbitrary names for loaded images, so performing this normalisation is not always wanted.