Health Check Causes Memory Leak in .NET 8

[debracey]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

In a dot net core 8 application, health checks executed on a regular interval eventually leak all available memory, causing the container to crash.

Expected Behavior

Health check should not leak memory. Memory usage should remain relatively consistent or at least be reclaimed by garbage collection.

Steps To Reproduce

Reproduced with the following system configuration:

• Windows 11 build 22631.3155
• SDK: 8.0.102
• Runtime: 8.0.2

1. Setup new dot net core web API application using .NET 8 as the target framework
2. Modify the auto generated Program.cs to include the following block of code

        builder.Services.AddHealthChecks();

        var app = builder.Build();
        app.MapHealthChecks("/hc", new HealthCheckOptions
        {
            AllowCachingResponses = false,
        });

From there, run the program with a memory profiler (I used DotMemory version 2023.3.3). Notice that each time /hc is called, the health check leaks ~100KB of RAM.

Now, modify the *.csproj file to change the target framework to dot net 6;

<TargetFramework>net6.0</TargetFramework>

Note:

• You will need to remove the OpenAPI reference in the *.csproj
• You will also need to remove the .WithOpenApi(); extension in Program.cs

Now repeat the profiling exercise. Notice that, although the memory is not 100% constant, it doesn't really exhibit "leaking" behavior.

Please advise as to how to proceed. If you setup a docker container to poll the service at ~10 second intervals, you'll end up with hundreds of MBs of RAM leaked over a few hour timeframe.

Severity: SEVERE; risk of production crash. Framework is effectively not usable.

Exceptions (if any)

Memory should not leak during basic health check.

.NET Version

8.0.102

Anything else?

Similar to issue reported in the runtime itself, dotnet/runtime#99304

[debracey]

Additional notes;

Did some more tests here, again using the sample API with a GET endpoint of /weatherforecast

• If I poll /weatherforecast at ~10 second intervals:
  • No memory leak is observed
  • Memory usage is more or less consistent, regardless of if call is made to builder.Services.AddHealthChecks();
• If I poll the /hc endpoint (see sample code above)
  • Memory leak is observed
  • The average leak per call to /hc is ~100KB (some health checks leak more than this, some less, some none at all)

The problem here is not the ~100KB that is leaked, the problem is the compounding nature of the requests. If the HC is polled by container infrastructure where the container orchestration specifies a max memory allocation, we'll eventually run out of RAM and crash.

[Kiechlus]

I'm trying to reproduce this, as we are also experiencing memory problems with .Net 8 which most of them seemed to have disappeared with 8.0.2. In Startup.cs, we do:

var healthChecksBuilder = services.AddHealthChecks();
healthChecksBuilder.AddCheck(...)
...
app.UseHealthChecks("/health", new HealthCheckOptions() {
  AllowCachingResponses = false,
});

I then run this locally in a alpine:8.0 container and call the health endpoint in a loop. When I do docker stats, I don't see the memory rising. Is there anything I am missing?

[debracey]

@Kiechlus I realized after I posted this that the SDK we're using is actually 8.0.1 - I'm not sure why that would make a difference, but I will try to upgrade the SDK to 8.0.2 and see if that fixes the issue.

Also, my container base image isn't alpine, it's photon. Again, not sure if that makes a difference.

I am out of the office this week but should be able to test this out next week

[GimmeDaKitty]

@debracey If you could keep us posted after you are back on whether upgrading to 8.0.2 fixed the issue taht would be appreciated.
We are seeing unexplained memory 'leaks' after upgrading all our (kubernetes) services to .NET 8. Any data point will be helpful for us.

[debracey]

@GimmeDaKitty, @Kiechlus I Went back and looked at my notes on this now that I am back in office. In the test described above I wasn't using a container at all, I was just running the service directly on windows out of my IDE (Rider 2023.3.3).

I upgraded my SDK to 8.0.203 and the issue seems to be resolved. I no longer see memory increasing (in the unmanaged memory section) with each call to HC.

I'll reach out to my infrastructure team to see what SDK our build pipelines use. I'll report back again once I can retest with an actual productionalized service.

[danmoseley]

@debracey is this still resolved for you?

@GimmeDaKitty @Kiechlus are you still seeing this on latest versions (8.0.4) ?

Wondering whether this can be closed. Part of the reason I'm here is it seems several issues have contributed to dotnet/runtime#95922 and to make progress on the whole we need to look at them all.

[mangod9]

Adding @andreiZi, who might still have a repro for this.

[danmoseley]

Linking that comment : dotnet/runtime#95922 (comment)

From what I observed, my assumption is that the microservice receives health check requests every 15 seconds, and it seems that something is not being cleaned up properly with each request.

@andreiZi is there a way for you to repeat the experiment but with health checks disabled?

[andreiZi]

Hello @danmoseley,

I've set up two scenarios as follows:

Scenario 1: Build Services with ASP.NET 8.0.4 and SDK 8.0.203 (Healthchecks enabled)

Here is the Dockerfile used:

FROM mcr.microsoft.com/dotnet/aspnet:8.0.4 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

ENV ASPNETCORE_URLS=http://*:80

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0.203  AS build
WORKDIR /src
ARG TARGETARCH

COPY ["BuildingBlocks/BuildingBlocks.Common/BuildingBlocks.Common.csproj", "BuildingBlocks/BuildingBlocks.Common/"]
COPY ["BuildingBlocks/BuildingBlocks.DataAccess/BuildingBlocks.DataAccess.csproj", "BuildingBlocks/BuildingBlocks.DataAccess/"]

COPY ["Services/User/User.Api/User.Api.csproj", "Services/User/User.Api/"]
COPY ["Services/User/User.Application/User.Application.csproj", "Services/User/User.Application/"]
COPY ["Services/User/User.Core/User.Core.csproj", "Services/User/User.Core/"]
COPY ["Services/User/User.DataAccess/User.DataAccess.csproj", "Services/User/User.DataAccess/"]

RUN dotnet restore -a ${TARGETARCH} "Services/User/User.Api/User.Api.csproj"

COPY . ./
WORKDIR "Services/User/User.Api/"
RUN dotnet build "User.Api.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "User.Api.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .

ENTRYPOINT ["dotnet", "User.Api.dll"]

Scenario 2: Deploy Services Completely Without Health Checks

My implementation of the health checks is as follows:

  services.AddHealthChecks();
  app.UseEndpoints(endpoints =>
      {
          endpoints.MapControllers().AddAsyncValidationFilter();
          endpoints.MapHealthChecks("/health");
      });

I will now observe and monitor the behavior. Probably in a few hours, I will report my observations.

[andreiZi]

Unfortunately, this approach did not solve the problem.

This morning, both services reached nearly 500MiB - 700MiB of RAM consumption. I managed to extract dumps from both services. Here is the SharePoint link where you can access them: SharePoint Link.

Below are the outputs from dumpheap -stat showing the largest memory consumers for each system:

System with Health Checks Completely Disabled (Running in Kubernetes with ASP.NET 8.0.4 and SDK 8.0.203):

7fc5f52a00c8  21,366     683,712 Serilog.Events.StructureValue
7fc5f311bf28  21,369     854,760 Serilog.Sinks.Grafana.Loki.Models.LokiLogEvent
7fc5f3ae8948  21,573     864,880 Serilog.Events.LogEventProperty[]
7fc5f5299c70  22,369   1,789,520 Serilog.Events.LogEvent
7fc5f52ab110  22,369   1,789,520 System.Collections.Generic.Dictionary<System.String, Serilog.Events.LogEventPropertyValue>
7fc5f5a2db20   5,051   2,207,624 System.Collections.Generic.KeyValuePair<System.String, Serilog.Events.LogEventPropertyValue>[]
7fc5f07b89e0  27,007   2,640,180 System.Int32[]
7fc5f4e2e680 181,134   4,347,216 Serilog.Events.ScalarValue
7fc5f2e6b438 137,286   4,393,152 Serilog.Events.LogEventProperty
7fc5f08863b8   6,140   8,878,194 System.Char[]
7fc5f52abf90  26,471  11,840,640 System.Collections.Generic.Dictionary<System.String, Serilog.Events.LogEventPropertyValue>+Entry[]
7fc5f07bd7c8 184,071  43,804,328 System.String
7fc5f1349800   4,028  50,552,143 System.Byte[]
55ffb238c7a0  35,601 180,890,880 Free
Total 800,018 objects, 320,912,524 bytes

System with Health Checks Enabled (Running in Kubernetes with ASP.NET 8.0.4 and SDK 8.0.203):

Example 1

7f4d61560980    54,053   1,297,272 System.Int32
7f4d66854398     5,001   2,104,024 System.Collections.Generic.KeyValuePair<System.String, Serilog.Events.LogEventPropertyValue>[]
7f4d63f3ce10        18   2,408,488 Serilog.Sinks.Grafana.Loki.Models.LokiLogEvent[]
7f4d660c4840   137,406   4,396,992 Serilog.Events.StructureValue
7f4d64909150   145,247   5,665,856 Serilog.Events.LogEventProperty[]
7f4d63c8b438   263,604   8,435,328 Serilog.Events.LogEventProperty
7f4d63f3bf28   251,316  10,052,640 Serilog.Sinks.Grafana.Loki.Models.LokiLogEvent
7f4d616a63b8    15,677  11,471,428 System.Char[]
7f4d615d89e0   263,406  16,636,700 System.Int32[]
7f4d660bd788   252,316  20,185,280 Serilog.Events.LogEvent
7f4d660cf888   252,316  20,185,280 System.Collections.Generic.Dictionary<System.String, Serilog.Events.LogEventPropertyValue>
7f4d62169800     4,579  30,148,160 System.Byte[]
7f4d65c625b8 1,616,214  38,789,136 Serilog.Events.ScalarValue
7f4d660f0810   261,573  67,245,936 System.Collections.Generic.Dictionary<System.String, Serilog.Events.LogEventPropertyValue>+Entry[]
7f4d615dd7c8   311,686  74,194,268 System.String
55716dec4620   280,214 116,343,616 Free
Total 4,349,002 objects, 441,525,143 bytes

Example 2:

Microsoft.AspNetCore.Server.Kestrel.Core.Internal.ConnectionLogScope+<GetEnumerator>d__7
7f49cb441260    14,776     354,624 Microsoft.Extensions.Diagnostics.HealthChecks.HealthStatus
7f49cb730638     8,460     473,760 System.IO.StringWriter
7f49c7e63c38    13,205     633,840 System.Text.StringBuilder
7f49c6c140f0    37,077     889,848 System.Double
7f49c6ba0980    96,629   2,319,096 System.Int32
7f49c957ce10         7   4,751,336 Serilog.Sinks.Grafana.Loki.Models.LokiLogEvent[]
7f49cb703148   247,891   7,932,512 Serilog.Events.StructureValue
7f49c92cb438   282,483   9,039,456 Serilog.Events.LogEventProperty
7f49c9f49150   253,965   9,734,272 Serilog.Events.LogEventProperty[]
7f49c6ce63b8    14,298  12,677,154 System.Char[]
7f49c957bf28   494,089  19,763,560 Serilog.Sinks.Grafana.Loki.Models.LokiLogEvent
7f49c6c189e0   499,985  29,983,368 System.Int32[]
7f49c77a9800     4,419  34,278,207 System.Byte[]
7f49cb6fcc78   494,139  39,531,120 Serilog.Events.LogEvent
7f49cb70e190   494,139  39,531,120 System.Collections.Generic.Dictionary<System.String, Serilog.Events.LogEventPropertyValue>
7f49cb2a25b8 3,117,157  74,811,768 Serilog.Events.ScalarValue
7f49cb70f010   498,426 118,985,088 System.Collections.Generic.Dictionary<System.String, Serilog.Events.LogEventPropertyValue>+Entry[]
7f49c6c1d7c8   492,305 126,014,400 System.String