Persist gossip bookmarks

[plotnick]

Just identity bookmarks so far, but full rumors set persistence coming soon!

[plaidfinch]

This approach for piggybacking on bookmarks to determine which jobs to execute during replay has a number of issues, which admittedly are probably not adequately called out in the docs for bookmarking.

At a high level:

1. Bookmarking should be thought of as party identifier loss recovery exclusively, and not as a high-water-mark that reliably and promptly recovers the last-seen version. A bookmark cannot reclaim the stored prior party identity until it has observed through gossip all the messages the prior party originated — until then, the bookmark is inert. At no point does a bookmark itself cause the version of the peer to be updated before the peer organically learns information through gossip, since doing so would lead to causality violation and contagious false redaction; it can only cause the party of the peer to join with the prior identity after the peer learns enough that it can be assured it will not retread previously-issued causal versions if it stamps them with the persisted identity.
2. In the sush architecture, there is a distinction between message receipt and message execution: just because you know about a job intended for you to execute, does not mean you yet have executed it — or even that you could execute it even in principle, since the linear hash-chain sequencing of jobs need not be monotonic with the causal versions of those jobs (if, for example, they are submitted through multiple endpoints, or even merely out of order). Nothing in the protocol enforces that the partial ordering of causal message versions agrees with the total ordering of jobs, so a job A whose start message causally precedes another job B's start message may itself be sequenced after job B from the sush state machine's perspective. Nothing in any level of any protocol prevents this, nor need it. Because these two notions of ordering are fundamentally unrelated, it's not possible to use a message version as a high-water-mark to indicate which jobs have been executed, not even if you were to augment the implementation to close the "message receive" vs. "message execute" gap by recording the last-executed message's version: in some cases, this version would go backwards!

As implemented in this PR, I believe sled restarts can trigger both arbitrary skipping of jobs (if the job is received-but-not-executed, it will be skipped upon restart, if the bookmark is absorbed on bootstrap), as well as arbitrary replay of jobs (if the bookmark fails to be absorbed temporarily due to a network partition withholding prior-sent messages (this is necessary for causal safety!), the high-water-mark as witnessed by the in-memory peer will be too low, and replay will fail to filter jobs it already did run).

Consequently, I believe the correct high-water-mark for tracking job execution and preventing local replay of already-executed jobs during resynchronization of a restarted sled is the job ID itself. Because jobs have a total ordering defined by their hash chain, this alone gives a unique pointer indicating where to resume. This high-water-mark should not be embedded in the rumors bookmark, which is (and must be) automatically read and written during the course of gossip. Adding a second piece of application state to that file (without a filesystem locking mechanism or similar) presents an acute opportunity for an A-B-A problem where the read/write to update the job high-water-mark would inadvertently roll back a concurrent update to the rumors bookmark content itself. This would be a difficult-to-replicate data race that would lead to non-deterministic arbitrary corruption of the rumor set (since bookmark rollback of any kind if causally unsafe).

This approach does have a caveat: all actions other than job start must be idempotent from the perspective of side effects on the host — let's call this host idempotency — because if we only track the last-started job ID to prevent replay of job starts, we cannot use this to bootstrap replay protection for other message kinds. In particular, this is because other message kinds are not linearly sequenced like job starts are: for example, two job stop messages, when received by sush, may execute in arbitrary order relative to one another, and relative to pending job starts or other actions.

As a final note, we should also consider a second property of our messages; let's call this rumor idempotency. During replay after a restart, with the corrected replay-filtering mechanism, we can prevent jobs from starting, but we don't have a mechanism for a singular high-water-mark for other actions, like job stop. If these host-idempotent actions themselves emit new messages back into the rumor set, then every sled restart will generate a storm of duplicative acknowledgement messages during replay, which, in the worst case, is multiplicatively amplified if those messages themselves could possibly generate responses. As such, I'd like to propose the rule that if an action has no host effect (i.e. stopping a job that was already stopped), then it should not inject more messages back into the rumor set. Are there any actions which can't or shouldn't be made to fit this? Do you have thoughts about how we can avoid this problem in other ways I'm not thinking of?