Skip to content

Commit 05b8f58

Browse files
authored
docs: adds details about leased IP gate in resource matching doc (#3260)
1 parent 87d6ee9 commit 05b8f58

1 file changed

Lines changed: 65 additions & 0 deletions

File tree

apps/provider-inventory/docs/resource-matching-acceptance-criteria.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,13 +21,24 @@ packing boxes into trucks. If everything fits, the bid is valid. If not, the pro
2121
The match is **all-or-nothing**: every replica of every service must find a home, or the whole
2222
deployment is rejected. Inventory is only updated if all placements succeed.
2323

24+
> **Scope note — leased IPs (AC18–AC21).** One part of the decision happens *outside* this
25+
> engine. **Leased IPs** (public IP addresses a tenant leases for a service) are checked by the
26+
> **inventory service** ([cluster/inventory.go](../cluster/inventory.go)) as a **cluster-level
27+
> pre-check, before** `tryAdjust`/`Adjust` is ever called. It is not part of the per-node engine,
28+
> but it *is* part of the same overall "can my cluster run this?" question and obeys the same
29+
> all-or-nothing outcome: if the provider lacks enough free leased IPs, the deployment is rejected
30+
> and the per-node engine never runs. The leased-IP criteria below (AC18–AC21) document that
31+
> pre-check.
32+
2433
---
2534

2635
## Core Concepts
2736

2837
- **Node** — one machine with its own CPU, RAM, GPUs, and local disk.
2938
- **Cluster storage** — shared persistent disk pools (e.g., `beta2`, `beta3`) usable by any node
3039
that supports that class.
40+
- **Leased IP** — a dedicated public IP a tenant leases for a service. Drawn from a single, finite
41+
**provider-wide** pool (managed by MetalLB), not tied to any node.
3142
- **Resource group** — one service definition + a replica count.
3243
- **Available capacity**`Allocatable − Allocated`, never negative. An `Allocatable` of `-1` is
3344
treated as effectively unlimited.
@@ -239,6 +250,7 @@ caller can inspect what would have been allocated. The engine's inventory does n
239250
- Recoverable example: Node 1 has no GPU → engine tries Node 2.
240251
- Fatal example: Cluster `beta2` pool is empty when any replica needs it → stop;
241252
`ErrInsufficientCapacity`.
253+
- Fatal example: Not enough free leased IPs (AC19) → stop before any node is tried.
242254

243255
### AC16 — "Unlimited" allocatable
244256
**WHAT:** When a node's `Allocatable` for a resource is `-1`, that dimension is treated as
@@ -258,6 +270,50 @@ failure: **CPU → GPU → Memory → Storage volumes (in declaration order)**.
258270
class-not-supported / RAM-volume-over-memory; cluster-level for pool exhaustion or bad
259271
attributes).
260272

273+
### AC18 — Leased IP counting
274+
**WHAT:** The number of leased IPs a deployment needs is the count of **unique** `LEASED_IP`
275+
endpoint sequence numbers across the whole group spec. The same endpoint referenced by several
276+
services is **one** IP, not many.
277+
**HOW:** The provider counts via `GetEndpointQuantityOfResourceGroup(gspec, Endpoint_LEASED_IP)`,
278+
which collects distinct endpoint sequence numbers into a set and returns the set size. This is
279+
computed once per reservation and stored as the reservation's endpoint quantity.
280+
281+
-**Counts as 1:** Three services all reference leased-IP endpoint `#1` → requested = 1.
282+
-**Counts as 2:** Group references leased-IP endpoints `#1` and `#2` → requested = 2.
283+
284+
### AC19 — Leased IP availability gate (cluster-level, pre-engine)
285+
**WHAT:** Before any node is examined, the provider checks it has enough free leased IPs for the
286+
whole deployment. This is a single cluster-wide pool — never per-node, never per-replica.
287+
**HOW:** `available = leased_ip.allocatable − leased_ip.allocated`, where
288+
`allocated = in-use + reserved`. `reserved` counts IPs held by reservations that the IP operator
289+
has **not yet confirmed**, so pending bids still consume availability. If `requested > available`,
290+
the reservation is rejected with **`insufficient number of IPs`** and the per-node engine never
291+
runs.
292+
293+
-**Match:** Pool of 10, 4 in use, 2 reserved → available 4; request 3 → passes; engine proceeds
294+
to fit CPU/memory/GPU/storage.
295+
-**Reject (cluster-level):** Same pool, request 5 > available 4 → `insufficient number of IPs`;
296+
no node is tried.
297+
-**No double-claiming:** Reserved-but-unconfirmed IPs still count as allocated, so two
298+
concurrent requests cannot both be offered the same free IPs — the second sees lower availability.
299+
300+
### AC20 — Provider without leased IP support
301+
**WHAT:** A provider that has no IP operator configured cannot serve leased IPs at all. Any
302+
deployment that needs at least one leased IP is rejected, regardless of node capacity.
303+
**HOW:** If the requested leased IP count is non-zero and the inventory service has no IP client,
304+
the reservation is rejected with **`no leased IPs available`**.
305+
306+
-**Reject:** Request 1 leased IP against a provider with no IP operator → `no leased IPs
307+
available`, even on an otherwise empty, fully-capable cluster.
308+
-**Unaffected:** Same provider, deployment requesting 0 leased IPs → IP check skipped (AC21).
309+
310+
### AC21 — Zero leased IPs requested
311+
**WHAT:** A deployment requesting no leased IPs skips the IP gate entirely and proceeds straight to
312+
the per-node engine (parallels AC8 for GPUs). The reservation's IPs are treated as already
313+
confirmed.
314+
315+
-**Match:** Requested leased IPs = 0 → no IP check; placement depends only on CPU/memory/GPU/storage.
316+
261317
---
262318

263319
## Worked Example — Mixed GPU Cluster, 3 Replicas
@@ -288,6 +344,8 @@ Variations:
288344
|-----------------------------|---------------------------------------------------------------------------------------------------|
289345
| `ErrInsufficientCapacity` | After exhausting all nodes, one or more replicas could not be placed; or a cluster-level resource (persistent pool, storage class, GPU constraint) blocks placement. |
290346
| `ErrGroupResourceMismatch` | Defensive guardrail — a subsequent replica produced different adjusted resources or scheduler params than the first replica of the same service. Rare in practice because request freezing usually converts heterogeneity into capacity failures. |
347+
| `no leased IPs available` | A deployment needs ≥1 leased IP but the provider has no IP operator configured (AC20). Wraps the inventory reservation error. |
348+
| `insufficient number of IPs`| Requested leased IPs exceed `leased_ip.allocatable − leased_ip.allocated` (AC19). Cluster-level — no node is tried. Wraps the inventory reservation error. |
291349

292350
---
293351

@@ -299,3 +357,10 @@ Variations:
299357
- Attribute parsing: [ParseGPUAttributes / ParseStorageAttributes](../cluster/types/v1beta3/clients/inventory/metrics.go)
300358
- Interface normalization: [FilterGPUInterface](../cluster/types/v1beta3/types.go#L160)
301359
- Test fixtures (4-node CPU example): [client_test.go:609-704](../cluster/kube/operators/clients/inventory/client_test.go#L609-L704)
360+
361+
Leased IP (AC18–AC21), enforced in the inventory service, not the per-node engine:
362+
363+
- Leased IP gate: [handleRequest](../cluster/inventory.go#L457)
364+
- Availability math: [availableLeasedIPs / countReservedIPs / leasedIPStatus](../cluster/inventory.go#L411-L444)
365+
- Request counting: [GetEndpointQuantityOfResourceGroup](../cluster/util/endpoint_quantity.go#L8)
366+
- Status reporting: `provider.Inventory.LeasedIP` (`leased_ip.{allocatable, allocated}` in the provider status stream)

0 commit comments

Comments
 (0)