NoStackTraceTimeoutException during connection to Kubernetes API server after upgrade to Spring Boot 4.0.3

[bhu-talendbj]

Version

vertx: 4.5.14

Context

Hello Vertx team,

We have met a strange issue since upgrade the Spring Boot to 4.0.3 that calling fabric8 kubernetes client serverSideApply() method failed with a request timeout during connection to the cluster API service.

Caused by: io.vertx.core.impl.NoStackTraceTimeoutException: The timeout period of 10000ms has been exceeded while executing GET /apis/cluster.x-k8s.io/v1beta1 for server E6F777209F3DCDD41D556D462490F7D4.gr7.us-east-1.eks.amazonaws.com:443

Which is similar with an issue reported before.

And even the workaround mentioned there to use a okhttp client instead of vert.x works for us...

However, the difference is we don't have such issue with kubernetes 7.X until upgrade the Spring Boot version to 4.0.3 ( or the latest available version 4.1.0 ).

We have raised an issue for Spring Boot team, but they preferred the issue from vertx...

Thanks!

Steps to reproduce

No response

Do you have a reproducer?

No response

[tsegismont]

Thanks for the report. Unfortunately there's not enough here for us to investigate.

A good first step would be diffing the dependency trees between the working and broken versions. If a Vert.x or Netty artifact changed transitively through Spring Boot, that's likely your culprit.

Beyond that, it would help to have:

• Vert.x/Netty wire-level logs around the failing request
• A minimal reproducer (doesn't need a real cluster, Minikube would do)

[bhu-talendbj]

Thanks for the response!

Comparing the dependency trees, the vertx is with same version, but not netty.
Output of SB 4.0.2:

+--- io.vertx:vertx-core:4.5.14
|    |    |    +--- io.netty:netty-common:4.1.118.Final -> 4.2.9.Final
|    |    |    +--- io.netty:netty-buffer:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    +--- io.netty:netty-transport:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    +--- io.netty:netty-handler:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    +--- io.netty:netty-handler-proxy:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    +--- io.netty:netty-codec-http:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    +--- io.netty:netty-codec-http2:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    +--- io.netty:netty-resolver:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    +--- io.netty:netty-resolver-dns:4.1.118.Final -> 4.2.9.Final (*)
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.16.1 -> 2.22.0 (*)
|    |    \--- io.vertx:vertx-web-client:4.5.14
|    |         +--- io.vertx:vertx-web-common:4.5.14
|    |         |    \--- io.vertx:vertx-core:4.5.14 (*)
|    |         +--- io.vertx:vertx-auth-common:4.5.14
|    |         |    \--- io.vertx:vertx-core:4.5.14 (*)
|    |         \--- io.vertx:vertx-core:4.5.14 (*)

Output of SB 4.1.0:

+--- io.vertx:vertx-core:4.5.14
|    |    |    +--- io.netty:netty-common:4.1.118.Final -> 4.2.15.Final
|    |    |    +--- io.netty:netty-buffer:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    +--- io.netty:netty-transport:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    +--- io.netty:netty-handler:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    +--- io.netty:netty-handler-proxy:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    +--- io.netty:netty-codec-http:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    +--- io.netty:netty-codec-http2:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    +--- io.netty:netty-resolver:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    +--- io.netty:netty-resolver-dns:4.1.118.Final -> 4.2.15.Final (*)
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.16.1 -> 2.21.4 (*)
|    |    \--- io.vertx:vertx-web-client:4.5.14
|    |         +--- io.vertx:vertx-web-common:4.5.14
|    |         |    \--- io.vertx:vertx-core:4.5.14 (*)
|    |         +--- io.vertx:vertx-auth-common:4.5.14
|    |         |    \--- io.vertx:vertx-core:4.5.14 (*)
|    |         \--- io.vertx:vertx-core:4.5.14 (*)

Here is the full stack of the exception, where vertx is implicitly called during a serverSideApply():

[07:57:31.286 INFO K8sClientBuilder.build] Kubernetes connection timeout: 10000
[07:57:31.286 INFO K8sClientBuilder.build] Kubernetes request timeout: 10000
[07:57:31.526 INFO VClusterExecutor.createVCluster] vCluster 'dpeqa-cluster-az-sandbox-347' -- apply resources
[07:57:41.814 ERROR VClusterExecutor.createVCluster] Kubernetes client exception
io.fabric8.kubernetes.client.KubernetesClientException: The timeout period of 10000ms has been exceeded while executing GET /apis/cluster.x-k8s.io/v1beta1 for server E6F777209F3DCDD41D556D462490F7D4.gr7.us-east-1.eks.amazonaws.com:443
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.restCall(OperationSupport.java:693)
	at io.fabric8.kubernetes.client.impl.BaseClient.getApiResources(BaseClient.java:307)
	at io.fabric8.kubernetes.client.impl.Handlers.lambda$getResourceDefinitionContext$2(Handlers.java:103)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at io.fabric8.kubernetes.client.impl.Handlers.getResourceDefinitionContext(Handlers.java:102)
	at io.fabric8.kubernetes.client.impl.Handlers.get(Handlers.java:68)
	at io.fabric8.kubernetes.client.impl.KubernetesClientImpl.resource(KubernetesClientImpl.java:374)
	at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.getResource(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:103)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.getResources(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:95)
	at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.performOperation(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:281)
	at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.serverSideApply(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:311)
	at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.serverSideApply(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:52)
	at org.talend.tests.acceptance.dynamicengine.actions.aws.VClusterExecutor.createVCluster(VClusterExecutor.java:72)
	at org.talend.tests.acceptance.dynamicengine.actions.aws.VClusterAction.createCluster(VClusterAction.java:31)
	at org.talend.tests.acceptance.dynamicengine.lifecycle.kubernetes.vcluster.VClusterClusterLifeCycle.setupCluster(VClusterClusterLifeCycle.java:114)
	at org.talend.tests.acceptance.dynamicengine.lifecycle.dynamicengine.DynamicEngineLifeCycle.setupDynamicEngine(DynamicEngineLifeCycle.java:593)
	at org.talend.tests.acceptance.dynamicengine.actions.DynamicEngineProvisionerAction.createDynamicEngine(DynamicEngineProvisionerAction.java:149)
	at org.talend.tests.acceptance.dynamicengine.steps.processing.DynamicEngineProvisionerSteps.createDynamicEngine(DynamicEngineProvisionerSteps.java:330)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.cucumber.java.Invoker.doInvoke(Invoker.java:66)
	at io.cucumber.java.Invoker.invoke(Invoker.java:24)
	at io.cucumber.java.AbstractGlueDefinition.invokeMethod(AbstractGlueDefinition.java:47)
	at io.cucumber.java.JavaStepDefinition.execute(JavaStepDefinition.java:29)
	at io.cucumber.core.runner.CoreStepDefinition.execute(CoreStepDefinition.java:66)
	at io.cucumber.core.runner.PickleStepDefinitionMatch.runStep(PickleStepDefinitionMatch.java:62)
	at io.cucumber.core.runner.ExecutionMode$1.execute(ExecutionMode.java:10)
	at io.cucumber.core.runner.TestStep.executeStep(TestStep.java:81)
	at io.cucumber.core.runner.TestStep.run(TestStep.java:53)
	at io.cucumber.core.runner.PickleStepTestStep.run(PickleStepTestStep.java:51)
	at io.cucumber.core.runner.TestCase.run(TestCase.java:84)
	at io.cucumber.core.runner.Runner.runPickle(Runner.java:75)
	at io.cucumber.core.runtime.Runtime.lambda$executePickle$6(Runtime.java:107)
	at io.cucumber.core.runtime.CucumberExecutionContext.lambda$runTestCase$5(CucumberExecutionContext.java:136)
	at io.cucumber.core.runtime.RethrowingThrowableCollector.executeAndThrow(RethrowingThrowableCollector.java:23)
	at io.cucumber.core.runtime.CucumberExecutionContext.runTestCase(CucumberExecutionContext.java:136)
	at io.cucumber.core.runtime.Runtime.lambda$executePickle$7(Runtime.java:107)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at io.cucumber.core.runtime.Runtime$SameThreadExecutorService.execute(Runtime.java:235)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
	at io.cucumber.core.runtime.Runtime.lambda$runFeatures$3(Runtime.java:89)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.SliceOps$1$1.accept(SliceOps.java:200)
	at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
	at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
	at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at io.cucumber.core.runtime.Runtime.runFeatures(Runtime.java:90)
	at io.cucumber.core.runtime.Runtime.lambda$run$0(Runtime.java:78)
	at io.cucumber.core.runtime.CucumberExecutionContext.lambda$runFeatures$6(CucumberExecutionContext.java:147)
	at io.cucumber.core.runtime.CucumberExecutionContext.execute(CucumberExecutionContext.java:162)
	at io.cucumber.core.runtime.CucumberExecutionContext.runFeatures(CucumberExecutionContext.java:145)
	at io.cucumber.core.runtime.Runtime.run(Runtime.java:78)
	at org.talend.tests.acceptance.SerenityRunner.executeTests(SerenityRunner.java:177)
	at org.talend.tests.acceptance.SerenityRunner.main(SerenityRunner.java:260)
Caused by: io.vertx.core.impl.NoStackTraceTimeoutException: The timeout period of 10000ms has been exceeded while executing GET /apis/cluster.x-k8s.io/v1beta1 for server E6F777209F3DCDD41D556D462490F7D4.gr7.us-east-1.eks.amazonaws.com:443

Sorry we can't provide a reproducer... in our understanding, it's more related to an implicit call of vertx from the kubernetes serverSideApply() under a SB framework but we can't package the whole framework.

[tsegismont]

Vert.x 4.5 is built and tested with Netty 4.1. Perhaps can you try to force the version using the Netty BOM https://central.sonatype.com/artifact/io.netty/netty-bom/4.1.136.Final

[bhu-talendbj]

From the dependencies tree seems the root cause is spring-boot-starter-webflux -> spring-boot-starter-reactor-netty -> spring-boot-reactor-netty which override the default netty version.

+--- org.springframework.boot:spring-boot-starter-webflux:4.0.2
|    +--- org.springframework.boot:spring-boot-starter-reactor-netty:4.0.2
|    |    +--- org.springframework.boot:spring-boot-starter:4.0.2 (*)
|    |    +--- org.springframework.boot:spring-boot-reactor:4.0.2
|    |    |    +--- org.springframework.boot:spring-boot:4.0.2 (*)
|    |    |    \--- io.projectreactor:reactor-core:3.8.2
|    |    |         +--- org.reactivestreams:reactive-streams:1.0.4
|    |    |         \--- org.jspecify:jspecify:1.0.0
|    |    \--- org.springframework.boot:spring-boot-reactor-netty:4.0.2
|    |         +--- org.springframework.boot:spring-boot-web-server:4.0.2 (*)
|    |         +--- io.projectreactor.netty:reactor-netty-http:1.3.2
|    |         |    +--- io.netty:netty-codec-http:4.2.9.Final

Then as a workaround it can work with SB 4.1.0 but the dependency with issue has to be kept as a lower version e.g. 4.0.2 with netty version 4.2.9.Final.
So now in my understanding this looks like an incompatible since netty 4.2.10.Final...