CASSANDRA-21537: Serialize local metadata proposals so concurrent schema changes do not exhaust their retry budge

[pmcfadin]

Details, reproduction and measurements: CASSANDRA-21537

Concurrent CQL DDL against a CMS member fails with Could not perform commit after 12 attempts. Time remaining: 4231ms — the retry attempt cap is reached while the request deadline still has seconds left. Every concurrent commit reads the same highest consecutive epoch and proposes at epoch+1, so all but one loser per round burns an attempt, backs off, and refetches the log before recomputing.

Fix

• AbstractLocalProcessor serialises the read → execute → propose cycle for locally originating commits behind a fair ReentrantLock.
• Acquisition is bounded by the caller's own remaining Retry deadline, so queue time is charged to the waiter and never extends another request.
• Each loop iteration is one timed acquisition, one critical section, one finally that unlocks exactly once.
• A request that is never admitted reports that distinctly, and only claims nothing was proposed when no earlier iteration reached the CAS.

Why this shouldn't regress

• Not new serialisation. The epoch is the partition key of the log table, and Paxos.cas already takes a per-partition coordinator lock for
  the whole CAS, so local contenders were already serialising — just bounded by cas_contention_timeout instead of the caller's deadline, and
  only discoverable by burning an attempt.
• tryCommitOne can only apply one proposal per epoch, so racing that slot locally never added throughput.
• Lock released before the follower wait, the uninterruptible backoff sleep, and any log fetch — none of those block another proposer.
• Fair, so no starvation by later arrivals.
• Success, rejection and failure behaviour is otherwise unchanged; the only new outcome is "not admitted".
• Passing: CMSShutdownTest, DistributedLogTest, LostCommitReqResTest, RetryTest, AlterTopologyTest, CommitStartupByNonCMSNodeTest,
  ConcurrentSchemaCommitTest, AbstractLocalProcessorLockTest, plus ant checkstyle checkstyle-test rat-check.
• Single-node concurrent DDL: 48 workers / 864 statements go from 311 failures to 0.

Tests added

• ConcurrentSchemaCommitTest (in-JVM) — asserts every submitted statement succeeds and swallows no exception.
• AbstractLocalProcessorLockTest (unit, ~2s, no sleeps) — timeout while queued, interrupt while queued, and
  proposal-attempted-then-denied-readmission.

For reviewers

• Does not address races between separate CMS members; that remains the distributed log's job.
• Admission is FIFO with no per-Transformation.Kind priority, so a topology change can queue behind earlier schema DDL. Happy to add priority if wanted.
• No admission metrics yet.
• Untested on a real multi-CMS-member cluster.
• Based on trunk; cassandra-6.0 is equally affected and the TCM source is identical between them

patch by Patrick McFadin; reviewed by TBD for CASSANDRA-21537

Assisted-by: Claude Opus 5