test: flaky TestRunManager_Reloader data race

[aelnahas]

TestRunManager_Reloader is flaky
examples:

• https://github.com/elastic/apm-server/actions/runs/24197516113/job/70631641469?pr=20864#step:5:300
• https://github.com/elastic/apm-server/actions/runs/23303004458/job/67769124192#step:5:300

=== RUN   TestRunManager_Reloader
    logger.go:146: 2026-04-09T15:09:04.409Z	DEBUG	manager.centralmgmt.V2-manager	Listening for agent unit changes
    logger.go:146: 2026-04-09T15:09:04.413Z	INFO	manager.centralmgmt.V2-manager	BeatV2Manager.unitListen UnitChanged.ID(output-unit), UnitChanged.Type(added), UnitChanged.Trigger(0): added/nothing_triggered
    logger.go:146: 2026-04-09T15:09:04.413Z	INFO	manager.centralmgmt.V2-manager	BeatV2Manager.unitListen UnitChanged.ID(input-unit-1), UnitChanged.Type(added), UnitChanged.Trigger(0): added/nothing_triggered
    logger.go:146: 2026-04-09T15:09:04.414Z	DEBUG	manager.centralmgmt.V2-manager	Got output unit config 'default'
    logger.go:146: 2026-04-09T15:09:04.414Z	INFO	beat	loaded output config
    logger.go:146: 2026-04-09T15:09:04.414Z	DEBUG	manager.centralmgmt.V2-manager	Skipped reloading APM tracing; configuration didn't change
    logger.go:146: 2026-04-09T15:09:04.415Z	INFO	beat	loaded input config	{"revision": 1}
    logger.go:146: 2026-04-09T15:09:04.427Z	INFO	manager.centralmgmt.V2-manager	BeatV2Manager.unitListen UnitChanged.ID(input-unit-1), UnitChanged.Type(removed), UnitChanged.Trigger(0): removed/nothing_triggered
    logger.go:146: 2026-04-09T15:09:04.427Z	INFO	manager.centralmgmt.V2-manager	BeatV2Manager.unitListen UnitChanged.ID(elastic-apm), UnitChanged.Type(added), UnitChanged.Trigger(0): added/nothing_triggered
    logger.go:146: 2026-04-09T15:09:04.427Z	DEBUG	manager.centralmgmt.V2-manager	Skipped reloading output; configuration didn't change
    logger.go:146: 2026-04-09T15:09:04.427Z	DEBUG	manager.centralmgmt.V2-manager	Skipped reloading APM tracing; configuration didn't change
    logger.go:146: 2026-04-09T15:09:04.428Z	INFO	beat	loaded input config	{"revision": 2}
    logger.go:146: 2026-04-09T15:09:04.428Z	DEBUG	manager.centralmgmt.V2-manager	Stopping beat
--- PASS: TestRunManager_Reloader (0.02s)
==================
WARNING: DATA RACE
Read at 0x00c00000304b by goroutine 90:
  testing.(*common).destination()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:1058 +0x96
  testing.(*common).log()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:1036 +0xbe
  testing.(*common).Logf()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:1200 +0x8e
  go.uber.org/zap/zaptest.TestingWriter.Write()
      C:/Users/runneradmin/go/pkg/mod/go.uber.org/zap@v1.27.1/zaptest/logger.go:146 +0x119
  go.uber.org/zap/zaptest.(*TestingWriter).Write()
      <autogenerated>:1 +0x68
  go.uber.org/zap/zapcore.(*ioCore).Write()
      C:/Users/runneradmin/go/pkg/mod/go.uber.org/zap@v1.27.1/zapcore/core.go:99 +0x201
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      C:/Users/runneradmin/go/pkg/mod/go.uber.org/zap@v1.27.1/zapcore/entry.go:258 +0x215
  go.uber.org/zap.(*SugaredLogger).log()
      C:/Users/runneradmin/go/pkg/mod/go.uber.org/zap@v1.27.1/sugar.go:355 +0x12c
  go.uber.org/zap.(*SugaredLogger).Errorf()
      C:/Users/runneradmin/go/pkg/mod/go.uber.org/zap@v1.27.1/sugar.go:216 +0x239
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Errorf()
      C:/Users/runneradmin/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.33.3/logp/logger.go:251 +0x1ec
  github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).watchErrChan()
      C:/Users/runneradmin/go/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20260331120055-5c8389728bd9/x-pack/libbeat/management/managerV2.go:481 +0x16d
  github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start.gowrap1()
      C:/Users/runneradmin/go/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20260331120055-5c8389728bd9/x-pack/libbeat/management/managerV2.go:299 +0x46

Previous write at 0x00c00000304b by goroutine 82:
  testing.tRunner.func1()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2023 +0x8f5
  runtime.deferreturn()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/runtime/panic.go:668 +0x5d
  testing.(*T).Run.gowrap1()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2101 +0x38

Goroutine 90 (running) created at:
  github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start()
      C:/Users/runneradmin/go/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20260331120055-5c8389728bd9/x-pack/libbeat/management/managerV2.go:299 +0x2f3
  github.com/elastic/apm-server/internal/beatcmd.TestRunManager_Reloader()
      D:/a/apm-server/apm-server/internal/beatcmd/beat_test.go:527 +0x2114
  testing.tRunner()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2036 +0x1ca
  testing.(*T).Run.gowrap1()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2101 +0x38

Goroutine 82 (running) created at:
  testing.(*T).Run()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2101 +0xb2a
  testing.runTests.func1()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2585 +0x85
  testing.tRunner()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2036 +0x1ca
  testing.runTests()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2583 +0x9f7
  testing.(*M).Run()
      C:/hostedtoolcache/windows/go/1.26.1/x64/src/testing/testing.go:2443 +0xf4b
  main.main()
      _testmain.go:76 +0x164
==================
=== RUN   TestRunManager_Reloader_newRunnerError
    logger.go:146: 2026-04-09T15:09:04.431Z	DEBUG	manager.centralmgmt.V2-manager	Listening for agent unit changes
    logger.go:146: 2026-04-09T15:09:04.433Z	INFO	manager.centralmgmt.V2-manager	BeatV2Manager.unitListen UnitChanged.ID(output-unit), UnitChanged.Type(added), UnitChanged.Trigger(0): added/nothing_triggered
    logger.go:146: 2026-04-09T15:09:04.433Z	DEBUG	manager.centralmgmt.V2-manager	Got output unit config 'default'
    logger.go:146: 2026-04-09T15:09:04.433Z	INFO	beat	loaded output config
    logger.go:146: 2026-04-09T15:09:04.433Z	DEBUG	manager.centralmgmt.V2-manager	Skipped reloading APM tracing; configuration didn't change
    logger.go:146: 2026-04-09T15:09:04.433Z	INFO	manager.centralmgmt.V2-manager	BeatV2Manager.unitListen UnitChanged.ID(input-unit-1), UnitChanged.Type(added), UnitChanged.Trigger(0): added/nothing_triggered
    logger.go:146: 2026-04-09T15:09:04.433Z	DEBUG	manager.centralmgmt.V2-manager	Skipped reloading output; configuration didn't change
    logger.go:146: 2026-04-09T15:09:04.433Z	DEBUG	manager.centralmgmt.V2-manager	Skipped reloading APM tracing; configuration didn't change
    logger.go:146: 2026-04-09T15:09:04.435Z	DEBUG	manager.centralmgmt.V2-manager	Stopping beat
    logger.go:146: 2026-04-09T15:09:04.436Z	ERROR	manager.centralmgmt.V2-manager	elastic-agent-client error: rpc error: code = Canceled desc = context canceled
    logger.go:146: 2026-04-09T15:09:04.436Z	ERROR	manager.centralmgmt.V2-manager	elastic-agent-client error: rpc error: code = Canceled desc = context canceled

log artifact for run 24197516113

hypothesis: It seems both failures seem to involve TestRunManager_Reloader and BeatV2Manager with the background logging happening around beat_test:527

[aelnahas]

This might be related to #19715 but I wanted to avoid confusion if the root cause is different

[rsmihir389]

Hey — I've been seeing this exact pattern a lot. Flaky tests like this usually cost teams $30-40 per rerun in CI time + developer context-switching.

I built a tool that detects flaky tests and shows the dollar cost: kleore.com. Here's what a report looks like on a real repo: sample report.

60-second install as a GitHub App, free tier shows the cost. Might help here.