As an operator, I'd like to know when a nimbus node is eligible for Leader so that I may check that status during instance replacement/health status.
When performing a rolling deployment, it's not possible to distinguish when a new node is "fully ready" and able to take on leadership. New nodes will refuse to becoem leaders until they have synced their blobstore. This can brick the cluster if instances are replaced before any new node has fully synced from the current leader.
There is a gap from when a nimbus node joins to when it's eligible to become a leader. To the best of my ability this status gap is not exposed, there are only three status values "Not a Leader", "Leader", and "Dead". Nor is there anything in the REST API (or any other metrics) that I can find that could hint at the sync status (say bytes in blob-cache per nimbus, sync "index" or mtime).
Example
1. Given a stable 3 node nimbus cluster.
2. Deploy some topologies of non-trivial size.
3. Replace nimbuses in a rolling fashion. (wait for new node to show "Not a Leader" status in /api/v1/nimbus/summary)
4. Cluster dies. New nimbuses will refuse to become a leader.
I propose a new status, "Syncing" or "Ineligible". Nimbuses in this status are not yet ready to become leaders and a healthy instance will transition to "Not a Leader" shortly. A new instance should not be considered ready until it has transitioned to the "Not a Leader" status.
Originally reported by dekimsey, imported from: Nimbus status/metrics don't expose criical blobstore sync status information
- status: Open
- priority: Major
- resolution: Unresolved
- imported: 2025-01-24
As an operator, I'd like to know when a nimbus node is eligible for Leader so that I may check that status during instance replacement/health status.
When performing a rolling deployment, it's not possible to distinguish when a new node is "fully ready" and able to take on leadership. New nodes will refuse to becoem leaders until they have synced their blobstore. This can brick the cluster if instances are replaced before any new node has fully synced from the current leader.
There is a gap from when a nimbus node joins to when it's eligible to become a leader. To the best of my ability this status gap is not exposed, there are only three status values "Not a Leader", "Leader", and "Dead". Nor is there anything in the REST API (or any other metrics) that I can find that could hint at the sync status (say bytes in blob-cache per nimbus, sync "index" or mtime).
Example
1. Given a stable 3 node nimbus cluster.
2. Deploy some topologies of non-trivial size.
3. Replace nimbuses in a rolling fashion. (wait for new node to show "Not a Leader" status in /api/v1/nimbus/summary)
4. Cluster dies. New nimbuses will refuse to become a leader.
I propose a new status, "Syncing" or "Ineligible". Nimbuses in this status are not yet ready to become leaders and a healthy instance will transition to "Not a Leader" shortly. A new instance should not be considered ready until it has transitioned to the "Not a Leader" status.
Originally reported by dekimsey, imported from: Nimbus status/metrics don't expose criical blobstore sync status information