Skip to content

fatal error: concurrent map iteration and map write in Olric.stats() — Members.Range iterated without holding the Members lock #285

Description

@omriShukrun08

fatal error: concurrent map iteration and map write in Olric.stats()Members.Range iterated without holding the Members lock

Version

v0.7.3 (embedded mode), bug still present on current master. Go 1.26.

What happened

Under cluster membership churn (a peer became unreachable and memberlist emitted
leave/update events), a concurrent call to EmbeddedClient.Stats() crashed the process:

fatal error: concurrent map iteration and map write

goroutine 140 [running]:
internal/runtime/maps.(*Iter).Next(...)
	internal/runtime/maps/table.go:819
github.com/olric-data/olric/internal/cluster/routingtable.(*Members).Range(...)
	github.com/olric-data/olric@v0.7.3/internal/cluster/routingtable/members.go:64
github.com/olric-data/olric.(*Olric).stats(...)
	github.com/olric-data/olric@v0.7.3/stats.go:134
github.com/olric-data/olric.(*EmbeddedClient).Stats(...)
	github.com/olric-data/olric@v0.7.3/embedded_client.go:327

Root cause

routingtable.Members embeds a sync.RWMutex, but its methods (Add, Delete,
Range) don't lock — callers are expected to hold the lock.

The reader and the writer here synchronize on different mutexes:

  • Olric.stats() takes db.rt.RLock() (the routing table's own mutex) but iterates
    db.rt.Members().Range(...) at stats.go:134 without taking Members().RLock().
  • RoutingTable.processClusterEvent (routingtable.go:262) mutates the members map
    holding only Members().Lock() — not the routing table lock.

So a NodeJoin/NodeLeave/NodeUpdate event writing the map while stats() iterates it is
an unguarded race, and the Go runtime aborts the process (unrecoverable — recover()
can't catch it).

The correct pattern already exists in the codebase: routingtable/update.go:84 takes
r.Members().RLock() before calling Range.

Suggested fix

db.rt.Members().RLock()
db.rt.Members().Range(func(id uint64, member discovery.Member) bool {
	s.ClusterMembers[stats.MemberID(id)] = toMember(member)
	return true
})
db.rt.Members().RUnlock()

Thanks for the help!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions