fatal error: concurrent map iteration and map write in Olric.stats() — Members.Range iterated without holding the Members lock
Version
v0.7.3 (embedded mode), bug still present on current master. Go 1.26.
What happened
Under cluster membership churn (a peer became unreachable and memberlist emitted
leave/update events), a concurrent call to EmbeddedClient.Stats() crashed the process:
fatal error: concurrent map iteration and map write
goroutine 140 [running]:
internal/runtime/maps.(*Iter).Next(...)
internal/runtime/maps/table.go:819
github.com/olric-data/olric/internal/cluster/routingtable.(*Members).Range(...)
github.com/olric-data/olric@v0.7.3/internal/cluster/routingtable/members.go:64
github.com/olric-data/olric.(*Olric).stats(...)
github.com/olric-data/olric@v0.7.3/stats.go:134
github.com/olric-data/olric.(*EmbeddedClient).Stats(...)
github.com/olric-data/olric@v0.7.3/embedded_client.go:327
Root cause
routingtable.Members embeds a sync.RWMutex, but its methods (Add, Delete,
Range) don't lock — callers are expected to hold the lock.
The reader and the writer here synchronize on different mutexes:
Olric.stats() takes db.rt.RLock() (the routing table's own mutex) but iterates
db.rt.Members().Range(...) at stats.go:134 without taking Members().RLock().
RoutingTable.processClusterEvent (routingtable.go:262) mutates the members map
holding only Members().Lock() — not the routing table lock.
So a NodeJoin/NodeLeave/NodeUpdate event writing the map while stats() iterates it is
an unguarded race, and the Go runtime aborts the process (unrecoverable — recover()
can't catch it).
The correct pattern already exists in the codebase: routingtable/update.go:84 takes
r.Members().RLock() before calling Range.
Suggested fix
db.rt.Members().RLock()
db.rt.Members().Range(func(id uint64, member discovery.Member) bool {
s.ClusterMembers[stats.MemberID(id)] = toMember(member)
return true
})
db.rt.Members().RUnlock()
Thanks for the help!
fatal error: concurrent map iteration and map writeinOlric.stats()—Members.Rangeiterated without holding the Members lockVersion
v0.7.3 (embedded mode), bug still present on current master. Go 1.26.
What happened
Under cluster membership churn (a peer became unreachable and memberlist emitted
leave/update events), a concurrent call to
EmbeddedClient.Stats()crashed the process:Root cause
routingtable.Membersembeds async.RWMutex, but its methods (Add,Delete,Range) don't lock — callers are expected to hold the lock.The reader and the writer here synchronize on different mutexes:
Olric.stats()takesdb.rt.RLock()(the routing table's own mutex) but iteratesdb.rt.Members().Range(...)atstats.go:134without takingMembers().RLock().RoutingTable.processClusterEvent(routingtable.go:262) mutates the members mapholding only
Members().Lock()— not the routing table lock.So a NodeJoin/NodeLeave/NodeUpdate event writing the map while
stats()iterates it isan unguarded race, and the Go runtime aborts the process (unrecoverable —
recover()can't catch it).
The correct pattern already exists in the codebase:
routingtable/update.go:84takesr.Members().RLock()before callingRange.Suggested fix
Thanks for the help!