Distributed Image Cache

[smira]

Distributed Image Cache

When running a cluster of nodes, each node pulls container images on their own multiplying the requirement for the network bandwidth, putting more stress on the container registry, etc.

See #10400 for related work, and this is the groundwork for this implementation.

Pre-requisites

We need first to implement segmentation.

Overview

Each machine in a cluster might enable Distributed Image Cache using two toggles:

• advertise itself as an Image Cache source (allow other machines to pull from it)
• enable distributed lookup from peers (on image pull, check other machines if they already have this image).

There is no additional configuration required besides enabling the Image Cache.

Image Cache relies on Cluster Discovery to find peers advertising themselves as image sources, and plugs in into the image pull pipeline utilizing same hooks as Local Image Cache.

Local Image Cache and Distributed Image Cache can be enabled together:

• if the machine advertises itself as an Image Cache, it provides both contents of CRI image store and Local Image Cache contents to the peers
• if the machine enables distributed lookups, they are performed only if the Local Image Cache doesn’t have the requested image.

A machine always pulls image from peers coming from the same segment.

Security Considerations

As the Local Image Cache, Distributed Image Cache bypasses registry authentication, so it should never be used in multi-tenant environments.

• Potential Threat

  Machine A pulls a private image I using image credentials. Now image I is available via distributed image cache to any peer in the cluster as long as they know the image I reference.

Distributed Image Cache offers API for the peers to look up and download images which is protected by mutual TLS. If desired, the API access might be restricted further using Ingress Firewall.

Configuration

Sample proposed configuration block:

apiVersion: v1alpha1
kind: DistributedImageCacheConfig
advertise:
	enabled: true
	advertisedAddressFilter: # optional, filters machine addresses which are advertised
	  - 192.168.0.0/24
lookup:
  enabled: true
ca:
	crt: ... # regular Talos PEM & base-64 encoded CA cert and key
	key: ... # regular Talos PEM & base-64 encoded CA cert and key		

Thoughts

Distributed Image Cache doesn’t change the time it takes to unpack an image. So if the image is huge, unpack time is still there. Distributed Image Cache only helps to move the load off the registry, or helps if bandwidth to the registry is smaller that in-cluster bandwidth.

One could deploy the following air-gapped scenario: a single machine has a Local Image Cache, advertises itself, and other machines don’t have a Local Image Cache, but use Distributed Image Cache to pull images off a single machine.

Implementation

Discovery

💡 We don’t need to update Kubernetes discovery, as it’s deprecated. We can only update Discovery Serivce.

Add new field(s) to the discovery data published by Talos machine to include:

• is distributed image cache advertised
• filtered advertised addresses for distributed image cache

Talos controllers related to discovery should be updated to handle it both ways (pushing and pulling).

CA Management

We introduce new distributed image cache CA (not using any existing PKI) for the following reasons:

• separation of concerns, this CA is only for the distributed image cache and nothing more
• other PKI is asymmetric: control plane machines have full PKI while workers request PKI from the controlplane; Image Cache PKI is symmetric, all machines are equal; Image Cache should be functional including installation time when PKI might not be available yet (e.g. controlplane machine pulling installer image from a worker).

We need to add new CA to the secrets.yaml with backwards compatibility (only for Talos 1.10+).

We need to figure out a way to generate Distributed Image Cache configuration document using talosctl gen config (as a patch?).

We need a controller which given CA will generate and refresh client and server certificates based on the CA (for accessing other peers and for listening for incoming connections).

registryd

Right now registryd listens only on localhost, we need to add another listener (if advertising is enabled), which listens on * with the TLS certificate from the above.

Now, if the request hits localhost endpoint:

• check local on-disk image cache (like it is right now, if enabled)
• if distributed lookup is enabled, perform the following:
  • pick N random peers from the discovery which advertise image cache (from the same segment)
  • do a fan-out HEAD request for the image/manifest to all N peers, whenever a single OK response arrives, cancel all other requests
  • if there is an OK response, perform GET from that peer and proxy it back

If the request hits * endpoint (with either HEAD or GET, for HEAD just check if the image is there, GET downloads the image):

• check local on-disk image cache (if enabled)
• perform a lookup in the CRI Image Service, and serve the image if found

PoC

PoC implementation of the above is in #7120

[especially-relative]

Would it be possible to have docs/integration for the Spegel mirror project? Or is something deeper under consideration?

[smira]

Would it be possible to have docs/integration for the Spegel mirror project? Or is something deeper under consideration?

Spegel should work on Talos already, this issue is about Talos native (similar) feature.

So if you want to use Spegel, you can do this today (I believe).

[onedr0p]

Spegel works great with Talos I've been using for awhile. There's info published on Spegel docs for integrating it with Talos.

https://spegel.dev/docs/getting-started/#talos

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity.

[PeterFalken]

I would still like to see this as a native feature on Talos.