fix(instr-mongodb): fix instr to no longer break mongodb >=6.4.0

[dependabot]

Bumps mongodb from 6.3.0 to 6.4.0.

Release notes

Sourced from mongodb's releases.

v6.4.0

6.4.0 (2024-02-29)

The MongoDB Node.js team is pleased to announce version 6.4.0 of the mongodb package!

Release Notes

Server selection will use a different Mongos on retry

When retrying reads or writes on a sharded cluster, the driver will attempt to select a different mongos for the retry if multiple are present. This should heuristically avoid encountering the original error that caused the need to retry the operation.

Caching AWS credentials provider per client

Instead of creating a new AWS provider for each authentication, we cache the AWS credentials provider per client to prevent overwhelming the auth endpoint and ensure that cached credentials are not shared with other clients.

BSON upgraded to ^6.4.0

BSON has had a number of performance increases in the last two releases (6.3.0 and 6.4.0). Small basic latin (ASCII) only strings, small memory allocations (ObjectId and Decimal128) and numeric parsing operations (int32, doubles, and longs) have all had optimizations applied to them.

For details check out the release notes here: BSON 6.3.0 and BSON 6.4.0 🐎

ExceededTimeLimit was made a retryable reads error

Read operations will be retried after receiving an error with the ExceededTimeLimit label.

Fixed unresolved request issue in KMS requester

Internal to the field-level encryption machinery is a helper that opens a TLS socket to the KMS provider endpoint and submits a KMS request. The code neglected to add a 'close' event listener to the socket, which had the potential to improperly leave the promise pending indefinitely if no error was encountered.

The base64 padding is now preserved in the saslContinue command

The authentication was rejected by the saslContinue command from mongosh due to missing "=" padding from the client. We fixed the way we parse payload to preserve trailing "="s.

countDocuments now types the filter using the collection Schema

Previously, countDocuments had a weakly typed Document type for the filter allowing any JS object as input. The filter is now typed as Filter<Schema> to enable autocompletion, and, hopefully, catch minor bugs.

Thank you to @​pashok88895 for contributing to this improvement.

The type error with $addToSet in bulkWrite was fixed

Previously the following code sample would show a type error:

interface IndexSingatureTestDocument extends Document {
    readonly myId: number;
    readonly mySet: number[];
  }
const indexSingatureCollection = undefined as unknown as Collection<IndexSingatureTestDocument>;
indexSingatureCollection.bulkWrite([
</tr></table> 

... (truncated)

Changelog

Sourced from mongodb's changelog.

6.4.0 (2024-02-29)

Features

• NODE-3449: Add serverConnectionId to Command Monitoring Spec (735f7aa)
• NODE-3470: retry selects another mongos (#3963) (84959ee)
• NODE-3689: require hello command for connection handshake to use OP_MSG disallowing OP_QUERY (#3938) (ce7df0f)
• NODE-4686: Add log messages to CLAM (#3955) (e3bfa30)
• NODE-4687: Add logging to server selection (#3946) (7f3ce0b)
• NODE-4719: add SDAM Logging Spec (#3940) (a3c0298)
• NODE-4847: Add config error handling to logging (#3970) (8f7bb59)
• NODE-5717: make ExceededTimeLimit retryable reads error (#3947) (106ab09)
• NODE-5885: upgrade BSON to ^6.3.0 (#3983) (9401d09)
• NODE-5939: Implement 6.x: cache the AWS credentials provider in the MONGODB-AWS auth logic (#3991) (e0a37e5)
• NODE-5978: upgrade BSON to ^6.4.0 (#4007) (90f2f70)

Bug Fixes

• NODE-5127: implement reject kmsRequest on server close (#3964) (568e05f)
• NODE-5609: node driver omits base64 padding in sasl-continue command (#3975) (b7d28d3)
• NODE-5765: change type for countDocuments (#3932) (22cae0f)
• NODE-5791: type error with $addToSet in bulkWrite (#3953) (b93d405)
• NODE-5818: Add feature flagging to server selection logging (#3974) (55203ef)
• NODE-5839: support for multibyte code-points in stringifyWithMaxLen (#3979) (aed1cf0)
• NODE-5840: heartbeat duration includes socket creation (#3973) (a42039b)
• NODE-5901: propagate errors to transformed stream in cursor (#3985) (ecfc615)
• NODE-5944: make AWS session token optional (#4002) (f26de76)

Performance Improvements

• NODE-5771: improve new connection (#3948) (a4776cf)
• NODE-5854: Conditional logger instantiation and precompute willLog perf fix (#3984) (a63fbc2)
• NODE-5928: consolidate signal use and abort promise wrap (#3992) (38742c2)

Commits

• 9ac2e38 chore(main): release 6.4.0 [skip-ci] (#3935)
• 5f62f56 docs: generate docs from latest main [skip-ci] (#3977)
• 90f2f70 feat(NODE-5978): upgrade BSON to ^6.4.0 (#4007)
• 99a0059 test(NODE-5731): add serverless proxy testing (#4003)
• 1ca6269 test(NODE-5929): convert txn legacy spec tests (#3987)
• f26de76 fix(NODE-5944): make AWS session token optional (#4002)
• 09c9b0b chore(NODE-5972): specify TS 5.0 in package.json and package-lock (#4004)
• eb5e2ab chore(NODE-5829): update driver dependencies (#3994)
• 233a2e0 refactor(NODE-5964): clean up prepareHandshakeDocument (#4001)
• ff8b5f5 refactor(NODE-5912): make server.command an async function (#3986)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[trentm]

run docs-build

[trentm]

symptom

The mongodb tests are failing here with:

 # running: (cd "/home/runner/work/apm-agent-nodejs/apm-agent-nodejs/test/instrumentation/modules/mongodb" && TEST_HOST=127.0.0.1 TEST_PORT=27017 TEST_DB=elasticapm TEST_COLLECTION=test TEST_USE_CALLBACKS=false node fixtures/use-mongodb.js)
# elapsed: 0.402s
# err:
# |Error: Command failed: /opt/hostedtoolcache/node/21.6.2/x64/bin/node fixtures/use-mongodb.js
# stdout:
# |{"log.level":"info","@timestamp":"2024-03-04T01:27:21.796Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.4.1","env":{"pid":7295,"proctitle":"/opt/hostedtoolcache/node/21.6.2/x64/bin/node","os":"linux 6.5.0-1015-azure","arch":"x64","host":"fv-az1429-514","timezone":"UTC+00","runtime":"Node.js v21.6.2"},"config":{"captureExceptions":{"source":"start","value":false},"centralConfig":{"source":"environment","value":false,"sourceValue":"false"},"cloudProvider":{"source":"environment","value":"none"},"logLevel":{"source":"start","value":"info","commonName":"log_level"},"metricsInterval":{"source":"environment","value":0,"sourceValue":"0s"},"serverUrl":{"source":"environment","value":"http://localhost:39231/","commonName":"server_url"},"spanCompressionEnabled":{"source":"start","value":false},"stackTraceLimit":{"source":"start","value":4},"serviceName":{"source":"start","value":"use-mongodb","commonName":"service_name"},"serviceVersion":{"source":"default","value":"4.4.1","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.4.1"}
# |{"log.level":"info","@timestamp":"2024-03-04T01:27:21.964Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","event.module":"app","url":"mongodb://127.0.0.1:27017","database":"elasticapm","collection":"test","useCallbacks":false,"message":"connect"}
# |{"log.level":"error","@timestamp":"2024-03-04T01:27:21.968Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","error":{"type":"MongoBulkWriteError","message":"Cannot read properties of undefined (reading 'name')","stack_trace":"MongoBulkWriteError: Cannot read properties of undefined (reading 'name')\n    at resultHandler (/home/runner/work/apm-agent-nodejs/apm-agent-nodejs/node_modules/mongodb/lib/bulk/common.js:294:29)\n    at /home/runner/work/apm-agent-nodejs/apm-agent-nodejs/node_modules/mongodb/lib/bulk/common.js:344:159\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"},"message":"useMongodb rejected"}
# stderr: <empty>
not ok 1 fixtures/use-mongodb.js exited successfully: err=Error: Command failed: /opt/hostedtoolcache/node/21.6.2/x64/bin/node fixtures/use-mongodb.js 

i.e. the node fixtures/use-mongodb.js fixture is crashing.

bug

IIUC, this is a bug caused by our instrumentation of mongodb/lib/cmap/connection_pool.js.
We added that instrumentation in #3665 to fix async context tracking for queued MongoDB commands.

    class ConnectionPoolTraced extends mod.ConnectionPool {
      checkOut(callback) {
        return super.checkOut(AsyncResource.bind(callback));
      }
    }

However, mongodb@6.4.0 includes mongodb/node-mongodb-native@ff8b5f5#diff-c464ff2f674f52c3696a24e1a90ef79e3f300a4fed92884a8c3cb6a1d0d143a5L353-R360 which changes the signature of that checkOut method such that it breaks:

-  checkOut(callback: Callback<Connection>): void {
+  async checkOut(): Promise<Connection> {

fix

I'd started out with this change:

diff --git a/lib/instrumentation/modules/mongodb/lib/cmap/connection_pool.js b/lib/instrumentation/modules/mongodb/lib/cmap/connection_pool.js
index 22089015..a8a5b09a 100644
--- a/lib/instrumentation/modules/mongodb/lib/cmap/connection_pool.js
+++ b/lib/instrumentation/modules/mongodb/lib/cmap/connection_pool.js
@@ -23,7 +23,14 @@ module.exports = (mod, agent, { version, enabled }) => {
   if (mod.ConnectionPool) {
     class ConnectionPoolTraced extends mod.ConnectionPool {
       checkOut(callback) {
-        return super.checkOut(AsyncResource.bind(callback));
+        if (typeof callback === 'function') {
+          return super.checkOut(AsyncResource.bind(callback));
+        } else {
+          // mongodb@>=6.4.0
+          //   -  checkOut(callback: Callback<Connection>): void {
+          //   +  async checkOut(): Promise<Connection> {
+          return super.checkOut();
+        }
       }
     }

This fixes the crash, but I expected that we'd again then get the async context tracking issue that #3665 fixed.
However, that isn't the case. I.e. the async context tracking is fine. Running node fixtures/use-mongodb-async-context.js results in the expected trace data (each transaction has a single mongodb span).

So now I think we should have that ConnectionPool instrumentation only apply to the earlier mongodb versions.

-  if (!semver.satisfies(version, '>=3.3 <7.0')) {
+  if (!semver.satisfies(version, '>=3.3 <6.4.0')) {

Qs

@david-luna Do you recall if you/we ever opened a node-mongodb-native issue about this AsyncResource.bind usage? I couldn't find anything searching https://jira.mongodb.org/projects/NODE/issues

[trentm]

run docs-build

[trentm]

run docs-build

[trentm]

Ah here it is: https://jira.mongodb.org/browse/NODE-5639

And TAV tests are failing for otel-js-contrib as well: open-telemetry/opentelemetry-js-contrib#1978 (comment)

[trentm]

Ref: open-telemetry/opentelemetry-js-contrib#1983