`_purge` only evicts the channel cache on the node that served the request; other nodes keep serving purged doc IDs from `_changes`

[george-kosmidisWS]

Version

Couchbase Sync Gateway 3.3.5.1 EE, running as Capella App Services (multi-node
endpoint behind a load balancer). Server: Couchbase Sync Gateway/3.3.5.1 EE.

Summary

POST /{keyspace}/_purge evicts the purged doc IDs from the channel cache of the
node that handles the request, but not from any other node in the cluster. Those
nodes keep returning the purged IDs from _changes indefinitely. A client that
follows the feed gets a doc ID that GET /{keyspace}/{docid} answers 404 for.

Steps to reproduce

Against a multi-node deployment (2 nodes is enough):

1. Write a few docs into channel X.
2. GET /{keyspace}/_changes?filter=sync_gateway/bychannel&channels=X&active_only=true
   repeatedly until every node has warmed its cache for X.
3. POST /{keyspace}/_purge with {"<docid>":["*"]} for one of them. Response is
   200 and lists the doc as purged.
4. GET /{keyspace}/<docid> — consistently 404 / error code 1404, from every
   node. The document really is gone.
5. Repeat the _changes request from step 2 many times.

Actual

The purged ID is still returned by roughly half the _changes requests — whichever
node did not serve the purge. Ten identical requests in my case:

result-count distribution over 10 identical _changes queries: {21: 5, 7: 5}

Seven is correct. The 21-row responses contain 14 IDs that were purged (two earlier
batches), every one of which _bulk_get reports as missing:

GONE IC5BD4AD2C-…::FO::WSCARRIER::…
GONE RWRiskSummary::IC5BD4AD2C-…::…
… 14 total …
LIVE IC7F353AB0-…::FO::WSCARRIER::…
… 7 total …

The split is stable across hours and survives repeated purges — it is not a
propagation delay.

Expected

Either the purge invalidates the channel cache cluster-wide, or _changes does not
hand out IDs the same deployment answers 404 for.

Analysis

handlePurge does evict the cache (added by #3765, covered by TestPurgeWithChannelCache):

rest/admin_api.go
count := h.collection.RemoveFromChangeCache(h.ctx(), docIDs, startTime)

but that path is in-process only — channelCacheImpl.Remove ranges over
c.channelCaches, this node's map:

db/channel_cache.go
func (c *channelCacheImpl) Remove(...) (count int) {
    c.channelCaches.Range(removeCallback)

Nothing notifies the other nodes, and they cannot infer it from the feed either.
Purge removes the doc with DeleteWithXattrs(key, [_sync, _globalSync]), so the
DCP deletion the other nodes receive has no _sync xattr, and changeCache.DocChanged
returns before doing anything with it:

db/change_cache.go
// If the document has no xattrs, it can not have a _sync xattr
if event.DataType&base.MemcachedDataTypeXattr == 0 {
    return
}

So the eviction is unreachable for every node except the one that served the REST
call. db.Compact uses the same RemoveFromChangeCache call for tombstone purging
and looks to have the same limitation.

Impact

Any client driving reads off _changes has to tolerate 404 on IDs the feed just
gave it. In our case it surfaced as an intermittent, node-dependent test failure that
looked like a product bug for a while.

Workaround

Pass since=<sequence captured before the run> on _changes — stale cache entries
sit below that sequence, so they are filtered out regardless of which node answers.

[bbrks]

It's possible that we can close this gap more than it is today - at least so the other nodes in the cluster observe the purge and clear their caches - but it still wouldn't completely solve the issue being described.

There is still be a window of opportunity for the changes feed to serve an entry for a doc and still get a 404 back upon requesting the revision. Even today on a single node, if the changes feed, the purge and then the client turning around and requesting the revision happen in that order sequentially the client would see the 404 and still need to handle it appropriately.

The linked issue from ~2018 describes why we can't easily purge the revision from other nodes, since upon observing a purge mutation we don't know what channels the document used to belong in to directly find the channel cache entry (the xattr data containing channels got purged) - so we'd have to search all of the channel caches used to build the changes feed for the given doc ID.

This is why we were originally thinking of an async job to go and do this cleanup, but it's still reasonably costly to do it that way, and would have a cache contention impact we'd prefer to not introduce.

The other option we have is a short-lived _sync:purged:... marker doc, which can be used to notify other SG nodes that the doc ID and its channels have been purged by a Sync Gateway node. This would allow more efficient cleanup of caches on the other nodes but still wouldn't completely close up the timing gap mentioned above. The main benefit you'd get is cleanup of the channel cache before hitting size or age limits for the affected doc.

The other part in this, is that for the vast majority of real-world deployments purges are relatively rare vs. a tombstone delete which is replicated to clients. Also most consumers of the changes feed are Couchbase Lite or other Sync Gateway nodes, which will handle these corner cases already.

The workaround you suggested seems OK for this test setup (I'm assuming that's what "before the run" means here?) - but less so for real-world cases where you're actively purging documents underneath running client replications. The since value should really be your last consumed sequence value.