K8SPSMDB-1504 Fix MongoDB Connection Leaks in PBM Operations

[hors]

CHANGE DESCRIPTION

Problem:

In various places inside the code, when we are calling pbm, err := backup.NewPBM(ctx, r.client, cr) , we never actually call pbm.Close(ctx). This leads to connections piling up leading eventually to OOMKilled.

Related issue: https://forums.percona.com/t/percona-operator-for-mongodb-endlessly-spawning-connections-until-oomkilled/39634

Cause:
Short explanation of the root cause of the issue if applicable.

Solution:
Properly closing the connections keeps everything normal

Adjusted also the e2e test init-deploy to verify that connections don't grow over a specific sanity limit.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported MongoDB version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test Name	Result	Time
arbiter	passed	00:00:00
balancer	passed	00:00:00
cross-site-sharded	passed	00:00:00
custom-replset-name	passed	00:00:00
custom-tls	passed	00:00:00
custom-users-roles	passed	00:00:00
custom-users-roles-sharded	passed	00:00:00
data-at-rest-encryption	passed	00:00:00
data-sharded	passed	00:00:00
demand-backup	passed	00:00:00
demand-backup-eks-credentials-irsa	passed	00:00:00
demand-backup-fs	passed	00:00:00
demand-backup-if-unhealthy	passed	00:00:00
demand-backup-incremental	passed	00:00:00
demand-backup-incremental-sharded	passed	00:00:00
demand-backup-physical-parallel	passed	00:00:00
demand-backup-physical-aws	passed	00:00:00
demand-backup-physical-azure	passed	00:00:00
demand-backup-physical-gcp-s3	passed	00:00:00
demand-backup-physical-gcp-native	passed	00:00:00
demand-backup-physical-minio	passed	00:00:00
demand-backup-physical-sharded-parallel	passed	00:00:00
demand-backup-physical-sharded-aws	passed	00:00:00
demand-backup-physical-sharded-azure	passed	00:00:00
demand-backup-physical-sharded-gcp-native	passed	00:00:00
demand-backup-physical-sharded-minio	passed	00:00:00
demand-backup-sharded	passed	00:00:00
expose-sharded	passed	00:00:00
finalizer	passed	00:00:00
ignore-labels-annotations	passed	00:07:10
init-deploy	passed	00:00:00
ldap	passed	00:00:00
ldap-tls	passed	00:00:00
limits	passed	00:00:00
liveness	passed	00:00:00
mongod-major-upgrade	passed	00:00:00
mongod-major-upgrade-sharded	passed	00:00:00
monitoring-2-0	passed	00:00:00
monitoring-pmm3	passed	00:00:00
multi-cluster-service	passed	00:00:00
multi-storage	passed	00:00:00
non-voting-and-hidden	passed	00:00:00
one-pod	passed	00:00:00
operator-self-healing-chaos	passed	00:00:00
pitr	passed	00:00:00
pitr-physical	passed	00:00:00
pitr-sharded	passed	00:00:00
pitr-to-new-cluster	passed	00:00:00
pitr-physical-backup-source	passed	00:00:00
preinit-updates	passed	00:00:00
pvc-resize	passed	00:00:00
recover-no-primary	passed	00:00:00
replset-overrides	passed	00:00:00
rs-shard-migration	passed	00:00:00
scaling	passed	00:00:00
scheduled-backup	passed	00:00:00
security-context	passed	00:00:00
self-healing-chaos	passed	00:00:00
service-per-pod	passed	00:00:00
serviceless-external-nodes	passed	00:00:00
smart-update	passed	00:00:00
split-horizon	passed	00:00:00
stable-resource-version	passed	00:00:00
storage	passed	00:00:00
tls-issue-cert-manager	passed	00:00:00
upgrade	passed	00:00:00
upgrade-consistency	passed	00:00:00
upgrade-consistency-sharded-tls	passed	00:00:00
upgrade-sharded	passed	00:00:00
upgrade-partial-backup	passed	00:00:00
users	passed	00:00:00
version-service	passed	00:00:00

Summary	Value
Tests Run	72/72
Job Duration	00:36:43
Total Test Time	00:07:10

commit: 8ff553e
image: perconalab/percona-server-mongodb-operator:PR-2098-8ff553e4