TBS: persist the sampling-decision publisher UUID across restarts to avoid redundant resync

[endorama]

When tail-based sampling (TBS) is enabled, APM Server publishes finalized sampling decisions to the traces-apm.sampled-* data stream and subscribes to the same stream to consume decisions from peer APM Servers. To skip its own decisions during subscription, the query uses a must_not filter on agent.ephemeral_id, which is set to samplerUUID.

samplerUUID is a package-level var generated fresh on every process start:

apm-server/x-pack/apm-server/main.go

Lines 45 to 47 in e899b88

	// samplerUUID is a UUID used to identify sampled trace ID documents
	// published by this process.
	samplerUUID = uuid.Must(uuid.NewV4())

Because the UUID rotates on every restart, the self-filter for a restarted process no longer matches decisions the same server published in previous incarnations. Those decisions are then re-fetched from Elasticsearch even though they are already present in the local decision DB.

Nothing is functionally incorrect here (decisions are idempotent on the consumer side, keyed by trace ID). The cost is wasted bandwidth, CPU, and disk writes on every restart.

/cc @carsonip which helped uncover this

Behaviour

Single-instance deployments. Every document in the data stream is published by this instance, so every subscribe poll matches only self-docs and returns zero hits. maxObservedSeqno stays at -1 in searchIndexTraceIDs, the if maxSeqno > observedSeqno gate in searchTraceIDs never fires, and subscriber_position.json is never advanced past its initial state. On restart the resumed subscriber issues _seq_no > -1 AND agent.ephemeral_id != newUUID, which matches every document still retained in the data stream. The paginated loop drains it at 1000 docs per page.

With persistent storage the decision DB and subscriber_position.json both survive. This does not cause issues as the re-ingested decisions are idempotent overwrites. The visible effect is redundant network and disk activity.
The persistent-storage improvement from #4437 is effectively shadowed by the UUID rotation.

Multi-instance deployments. The position advances past peer-published decisions during normal operation, so the re-fetched window on restart is bounded to the tail of recently-written self-docs (between the last peer-observed _seq_no and the current global checkpoint). Still redundant, but with smaller overall impact.

Impact

Scales with throughput and ILM retention on traces-apm.sampled-* indices.

On restart, it leads to elevated CPU, disk, and network activity until the stream is drained.

No impact on ephemeral storage, as there the re-fetch is necessary anyway. Decisive impact on persistent storage where the re-fetch could be mostly avoided.

Open question

Was the per-process scoping of samplerUUID intentional? The comment at main.go:45-47 suggests yes but does not clarify for what purpose. Before opening a PR we need to confirm whether there is a correctness argument behind per-process identity.

[endorama]

I've seen that we have a persisted UUID from beats (b.Info.ID, loaded from meta.json at internal/beatcmd/beat.go:210-230).

Could we use that instead of generating a fresh UUID in the TBS package?

[endorama]

@axw maybe you can help answering the open question above? Hoping we can draw on your historical context here.

[axw]

Was the per-process scoping of samplerUUID intentional? The comment at main.go:45-47 suggests yes but does not clarify for what purpose. Before opening a PR we need to confirm whether there is a correctness argument behind per-process identity.

The comment does state the purpose, but I suppose it needs clarifying: the purpose is to identify sampled trace ID documents that originate from the process. Processes are ephemeral, so it's expected that each new process will have its own UUID.

We can change it to be per apm-server instance, rather than per-process. I've paged out just about all I used to know about beats IDs, but I think we could use the beats ID instead. Looking at git blame, it appears that's what we did prior to bd3f058

[carsonip]

We can change it to be per apm-server instance, rather than per-process.

++

but I think we could use the beats ID instead

If we don't want to rely on beats, an alternative would be to bind to the pebble / subscriber position identity, under the assumption that no one would share them across 2 running instance of apm-server because that is just incorrect. That means generating an id and write them to pebble / subscriber position, then in subsequent restarts, read from them.