bigtable: Large increase in memory when Direct Access is enabled

[adamroyjones]

Client

bigtable

Environment

Statically-linked binaries (go1.25.6 on amd64) running in GKE.

Code and Dependencies

We're using cloud.google.com/go/bigtable@v1.41.0.

If it's needed, I can see about producing something minimal to reproduce things in our GKE clusters.

Expected behavior

We expected memory use to be roughly unchanged by switching Direct Access on (CBT_ENABLE_DIRECTPATH: "true").

Actual behavior

Memory use increased substantially after switching Direct Access on (CBT_ENABLE_DIRECTPATH: "true") (with credit to @philpearl for noticing this).

The graph below shows one service's memory use. The step increase (~2026-01-07 1400 UTC) and decrease (~2026-02-02 1720 UTC) correspond to setting the environment variable and leaving it unset, respectively.

This kind of change affects all of the services that we have that use Bigtable. We've some sampling profiles that suggest that much of the issue is deep in some gRPC code around resolving clusters (e.g., google.golang.org/grpc/internal/xds/balancer/clusterresolver/clusterresolver.go); if more details are needed, we can look to provide them.

I would expect that this would be easy to reproduce.

Screenshots

Additional context

None—but we like Direct Access enough to leave it on, so thank you for introducing it!

[sushanb]

Thanks for filing the issue. I am investigating this. Will report back once i have concrete findings.

[adamroyjones]

As a small update: I've upgraded our fleets from 1.41.0 to 1.42.0 today and the problem is present with that version, as well. (The drop here is, again, from leaving the environment variable unset.)

[sushanb]

We are brainstorming some optimizations with grpc go team for trying to reduce the memory footprint decrease.

[adamroyjones]

One thing I noticed today is a periodic quality to resource usage that doesn't make much sense, at least from the client's perspective. Do you have any guidance around the below? (Again, as soon as Direct Access is turned off, these phenomena vanish.)

europe-west1

Here are charts for one Deployment in europe-west1. There's one line per replica in the Deployment. The top chart shows CPU, the bottom shows memory. The changes in CPU follow our demand patterns. The changes in memory do not. Moreover, the behaviour with memory is not uniform over the replicas in the Deployment.

The memory patterns are consistent over all of our Deployments that interact with Bigtable, which suggests that it's something server-side.

us-east4

For good measure, here's a chart from us-east4. The sudden spasmodic scrawl at the end, there, is surprising, at least to me.

[sushanb]

@adamroyjones @philpearl

We have memory optimizations that are gonna be in grpc go version 1.81.0 notably which should reduce the memory usage substantially.

thanks for filing the issue and helping us. this is great colloboration

grpc/grpc-go#8906
grpc/grpc-go#8921
grpc/grpc-go#8957
grpc/grpc-go#8964

[adamroyjones]

Thanks for the update! We look forward to trying the changes in grpc/grpc-go when they're released.

We've turned off Direct Access for now (given the patterns in #13753 (comment)), but we're keen to try it again.

I'll test the new versions of things once they've been released and update this issue with my findings.

[sushanb]

Interesting, do you use any custom go garbage collection flag GOGC?

[adamroyjones]

No—we leave it (and GOMEMLIMIT) unset for the services that interact with Bigtable. The behaviour's very odd!