9.4.0 Test Plan

[aelnahas]

Manual Test Plan

• When picking up a test case, please add your name to this overview beforehand and tick the checkbox when finished.
• Testing can be started when the first build candidate (BC) is available in the CFT region.
• For each repository, update the compare version range to get the list of commits to review.

ES apm-data plugin

List of changes: elastic/elasticsearch@v9.3.3...9.4

• Refactoring UserAgentPlugin (#140712) — elastic/elasticsearch@c409f7d
• apm-data: explicit map of timestamp.us to long (#143173) — elastic/elasticsearch@9b887f1
• Deprecate aggregate metric double default metric configuration (#141877) — elastic/elasticsearch@f19d2f1

apm-aggregation

No version change detected.

go-docappender

No version change detected.

apm-data

No version change detected.

apm-server

List of changes: v9.3.3...main

other

• 227af48: feat: bump go to 1.26.1 (#20585) (by
kruskall on 2026-03-10)
• e2bf81c: [updatecli] Bump Golang version to 1.25.7
(#20311) (by
elastic-observability-automation[bot] on 2026-02-09)
• 413166c: [updatecli] Bump Golang version to 1.25.6
(#20148) (by
elastic-observability-automation[bot] on 2026-01-19)

function changes

• 63f71fc: feat: move docker labels to opencontainers
  (#20773) (by kruskall)
• c48b935: add timestamp.us system and integration tests
  (#20696) (by Edoardo Tenani)
• b696dc7: fix: Ensure shutdown logging is called within Runner.Run lifetime
  (#20667) (by Carson Ip)
• 164a94a: feat: replace archived yaml dependency
  (#20586) (by kruskall)
• 227af48: feat: bump go to 1.26.1 (#20585) (by
  kruskall)
• 60e89a9: add tbs storage limit and disk-related metrics
  (#20464) (by Eric)
• 67a5a2b: tbs: Fix potential data race
  (#19948) (by Eric)
• 79fcb9a: Fix integration server test upgrade config for 9.4 upgrade
  (#20000) (by Eric)

dependency updates

Elastic stack
• 37ecc3b: [updatecli] Bump elastic stack version to 9.4.0-88e3ed81
(#20839) (by
elastic-observability-automation[bot] on 2026-04-07)
• 949b066: [updatecli] Bump elastic stack version to 9.4.0-9a772a93
(#20809) (by
elastic-observability-automation[bot] on 2026-04-02)
• cc74eea: [updatecli] Bump elastic stack version to 9.4.0-eb8fc255
(#20794) (by
elastic-observability-automation[bot] on 2026-04-01)
• ebce39e: [updatecli] Bump elastic stack version to 9.4.0-572e1baa
(#20779) (by
elastic-observability-automation[bot] on 2026-03-27)
• 71690dc: [updatecli] Bump elastic stack version to 9.4.0-8e2edea2
(#20715) (by
elastic-observability-automation[bot] on 2026-03-16)
• e40611f: Update docker-compose and stack (#20692
(#20692)) (by elastic-observability-automation[bot] on
2026-03-13)
• b0efc3a: [updatecli] Bump elastic stack version to 9.4.0-78bfc322
(#20649) (by
elastic-observability-automation[bot] on 2026-03-12)
• 642da82: [updatecli] Bump elastic stack version to 9.4.0-9b4e1705
(#20614) (by
elastic-observability-automation[bot] on 2026-03-11)
• 59dfffb: [updatecli] Bump elastic stack version to 9.4.0-05288400
(#20579) (by
elastic-observability-automation[bot] on 2026-03-09)
• a556cc6: [updatecli] Bump elastic stack version to 9.4.0-e2040fd9
(#20457) (by
elastic-observability-automation[bot] on 2026-02-23)
• b6a7957: [updatecli] Bump elastic stack version to 9.4.0-f338c365
(#20367) (by
elastic-observability-automation[bot] on 2026-02-17)
• 9e82adf: [updatecli] Bump elastic stack version to 9.4.0-8c1b6467
(#20350) (by
elastic-observability-automation[bot] on 2026-02-11)
• b7048be: [updatecli] Bump elastic stack version to 9.4.0-2f21cc70
(#20314) (by
elastic-observability-automation[bot] on 2026-02-09)
• c0e797e: [updatecli] Bump elastic stack version to 9.4.0-40d32ed8
(#20292) (by
elastic-observability-automation[bot] on 2026-02-04)
• 648e413: [updatecli] Bump elastic stack version to 9.4.0-b8a4201f
(#20244) (by
elastic-observability-automation[bot] on 2026-02-02)
• ecd777b: [updatecli] Bump elastic stack version to 9.4.0-bed05e1e
(#20225) (by
elastic-observability-automation[bot] on 2026-01-28)
• 581d99e: [updatecli] Bump elastic stack version to 9.4.0-674da02b
(#20196) (by
elastic-observability-automation[bot] on 2026-01-28)
• 55fabe5: [updatecli] Bump elastic stack version to 9.4.0-d212f69d
(#20191) (by
elastic-observability-automation[bot] on 2026-01-23)
• 76994ac: [updatecli] Bump elastic stack version to 9.4.0-8b03330f
(#20186) (by
elastic-observability-automation[bot] on 2026-01-22)
• c7b7c2f: [updatecli] Bump elastic stack version to 9.4.0-3411db1b
(#20176) (by
elastic-observability-automation[bot] on 2026-01-22)
• c0d38e8: [updatecli] Bump elastic stack version to 9.4.0-55e54a90
(#20026) (by
elastic-observability-automation[bot] on 2025-12-30)
• d23c6ea: [updatecli] Bump elastic stack version to 9.4.0-ae56b40b
(#20009) (by
elastic-observability-automation[bot] on 2025-12-29)
• dde9bdd: [updatecli] Bump elastic stack version to 9.4.0-301b9151
(#19996) (by
elastic-observability-automation[bot] on 2025-12-29)
• cb61543: [updatecli] Bump elastic stack version to 9.4.0-04d54b57
(#19989) (by
elastic-observability-automation[bot] on 2025-12-24)
• daf100c: [updatecli] Bump elastic stack version to 9.4.0-c14f651c
(#19974) (by
elastic-observability-automation[bot] on 2025-12-23)
• 1cb0cee: [updatecli] Bump elastic stack version to 9.4.0-77888edb
(#19961) (by
elastic-observability-automation[bot] on 2025-12-22)

Elastic Beats
• 30bc688: [updatecli][main] Update to elastic/beats
(#20757) (by
elastic-observability-automation[bot] on 2026-04-01)
• f64df43: [updatecli][main] Update to elastic/beats
(#20732) (by
elastic-observability-automation[bot] on 2026-03-18)
• 33e8c78: [updatecli][main] Update to elastic/beats
(#20693) (by
elastic-observability-automation[bot] on 2026-03-16)
• b67261b: chore: update-beats with main (#20647
(#20647)) (by elastic-observability-automation[bot] on
2026-03-12)
• 01b9710: [updatecli][main] Update to elastic/beats
(#20552) (by
elastic-observability-automation[bot] on 2026-03-11)
• 91ff3e3: [updatecli][main] Update to elastic/beats
(#20516) (by
elastic-observability-automation[bot] on 2026-03-03)
• 0559292: [updatecli][main] Update to elastic/beats
(#20452) (by
elastic-observability-automation[bot] on 2026-02-26)
• 19134ec: [updatecli][main] Update to elastic/beats
(#20406) (by
elastic-observability-automation[bot] on 2026-02-18)
• 83dea74: [updatecli][main] Update to elastic/beats
(#20183) (by
elastic-observability-automation[bot] on 2026-01-26)
• 180970b: chore: update-beats with main (#20172
(#20172)) (by elastic-observability-automation[bot] on
2026-01-21)
• 801ea4f: [updatecli][main] Update to elastic/beats
(#20128) (by
elastic-observability-automation[bot] on 2026-01-20)
• 6099446: [updatecli][main] Update to elastic/beats
(#20049) (by
elastic-observability-automation[bot] on 2026-01-15)
• 8177b34: [updatecli][main] Update to elastic/beats
(#20017) (by
elastic-observability-automation[bot] on 2025-12-30)

Test cases from the GitHub board

Label the relevant 9.4.0 Issues / PRs with the test-plan label https://github.com/elastic/apm-server/issues?page=1&q=-label%3Atest-plan+label%94.0+-label%3Atest-plan-ok-->
apm-server 9.4.0 test-plan -->

Add yourself as assignee on the PR before you start testing.

[aelnahas]

I am looking at these commits potentially for testing:
67a5a2b: tbs: Fix potential data race (#19948 (#19948))
(by Eric on 2026-01-16)
- ccfeb4c: Report status on major health issues (
#20166) (by Eric on 2026-02-04)
- 60e89a9: add tbs storage limit and disk-related metrics (
(#20464) (by Eric on 2026-02-24)
- b94217b: Known issue - timestamp.us as float for APM Server when non-Elasticsearch output is used
(#20508) (by Luca Belluccini on 2026-02-27)
- 164a94a: feat: replace archived yaml dependency (
(#20586) (by kruskall on 2026-03-11)
- 63f71fc: feat: move docker labels to opencontainers (
(#20773) (by kruskall on 2026-03-31)
- b696dc7: fix: Ensure shutdown logging is called within Runner.Run lifetime
(#20667) (by Carson Ip on 2026-03-12)
- c48b935: add timestamp.us system and integration tests
(#20696) (by Edoardo Tenani on 2026-03-17)

Confirmed these were already tested in previous releases.

[aelnahas]

Tested

f8d4b08: Fix HTTP/2 SETTINGS mismatch connection failure in gmux (#20908) (by Carson Ip on 2026-04-16)

Testing notes:

Steps

  go build -o ./apm-server-9.4 ./cmd/apm-server # build 9.4 
  ./apm-server-9.4 -e --strict.perms=false -E apm-server.host=127.0.0.1:3000 -E apm-server.ssl.enabled=true -E apm-server.ssl.certificate=testdata/tls/certificate.pem -E apm-server.ssl.key=testdata/tls/key.pem # run binary

made a request

curl -vk --http2 https://127.0.0.1:3000
*   Trying 127.0.0.1:3000...
* Connected to 127.0.0.1 (127.0.0.1) port 3000
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: O=mkcert development certificate; OU=andrew@alloy
*  start date: Dec 18 15:01:22 2019 GMT
*  expire date: Dec 18 15:01:22 2029 GMT
*  issuer: O=mkcert development CA; OU=andrew@alloy; CN=mkcert andrew@alloy
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:3000/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:3000]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: 127.0.0.1:3000
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 150
< date: Fri, 17 Apr 2026 14:45:03 GMT
<
{
  "build_date": "2026-04-16T22:13:46Z",
  "build_sha": "f8d4b08f44413981be133cef4ba573b206c35766",
  "publish_ready": false,
  "version": "9.4.0"
}
* Connection #0 to host 127.0.0.1 left intact

(Status 200, Exit code 0)

I also used nghttp2 to get a better view of the frames level comms

nghttp -v --no-verify-peer https://127.0.0.1:3000/
# h2 negotiated
[  0.008] Connected
The negotiated protocol: h2

# this I think is gmux sniffing the settings
[  0.010] recv SETTINGS frame <length=36, flags=0x00, stream_id=0>
          (niv=6)
          [SETTINGS_MAX_FRAME_SIZE(0x05):1048576]
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):250]
          [SETTINGS_MAX_HEADER_LIST_SIZE(0x06):1048896]
          [SETTINGS_HEADER_TABLE_SIZE(0x01):4096]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):1048576]
          [SETTINGS_NO_RFC7540_PRIORITIES(0x09):1]
[  0.010] send SETTINGS frame <length=18, flags=0x00, stream_id=0>
          (niv=3)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
          [SETTINGS_NO_RFC7540_PRIORITIES(0x09):1]
[  0.010] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.010] send HEADERS frame <length=48, flags=0x05, stream_id=1>
          ; END_STREAM | END_HEADERS
          (padlen=0)
          ; Open new stream
          :method: GET
          :path: /
          :scheme: https
          :authority: 127.0.0.1:3000
          priority: u=3
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.68.1

[  0.011] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)

# actual apm-server's HTTP/2 settings. They are mirrored now
[  0.011] recv SETTINGS frame <length=36, flags=0x00, stream_id=0>
          (niv=6)
          [SETTINGS_MAX_FRAME_SIZE(0x05):1048576]
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):250]
          [SETTINGS_MAX_HEADER_LIST_SIZE(0x06):1048896]
          [SETTINGS_HEADER_TABLE_SIZE(0x01):4096]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):1048576]
          [SETTINGS_NO_RFC7540_PRIORITIES(0x09):1]
[  0.011] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=983041)
[  0.011] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.011] recv (stream_id=1) :status: 200
[  0.011] recv (stream_id=1) content-type: application/json
[  0.011] recv (stream_id=1) x-content-type-options: nosniff
[  0.011] recv (stream_id=1) content-length: 150
[  0.011] recv (stream_id=1) date: Fri, 17 Apr 2026 14:51:22 GMT
[  0.011] recv HEADERS frame <length=66, flags=0x04, stream_id=1>
          ; END_HEADERS
          (padlen=0)
          ; First response header
{
  "build_date": "2026-04-16T22:13:46Z",
  "build_sha": "f8d4b08f44413981be133cef4ba573b206c35766",
  "publish_ready": false,
  "version": "9.4.0"
}
[  0.011] recv DATA frame <length=150, flags=0x01, stream_id=1>
          ; END_STREAM

# this is from nghttp client 
[  0.011] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])