[bug] ArgumentException thrown in AWSLambdaWrapper.OnFunctionStart

[martincostello]

Component

OpenTelemetry.Instrumentation.AWSLambda

Package Version

Package Name	Version
OpenTelemetry	1.11.2
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.11.2
OpenTelemetry.Extensions.Hosting	1.11.2
OpenTelemetry.Instrumentation.AWS	1.11.1
OpenTelemetry.Instrumentation.AWSLambda	1.11.1
OpenTelemetry.Instrumentation.Http	1.11.1
OpenTelemetry.Resources.OperatingSystem	1.11.0-beta.2
OpenTelemetry.Resources.ProcessRuntime	1.11.0-beta.2

Runtime Version

.NET 9.0.2

Description

While running the tests for an AWS Lambda function of mine, one of the tests failed with this error:

fail: MartinCostello.Testing.AwsLambdaTestServer.RuntimeHandler[0]
      Error invoking Lambda function with ARN arn:aws:lambda:eu-west-1:123456789012:function:test-function for request Id 80d04f94-3177-461e-9f61-e0e4b4ce78f6: {
        "errorType": "ArgumentException",
        "errorMessage": "Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')",
        "stackTrace": [
          "at System.Array.CopyImpl(Array, Int32, Array, Int32, Int32, Boolean) + 0x3a4",
          "at System.Collections.Generic.List`1.set_Capacity(Int32) + 0x90",
          "at System.Collections.Generic.List`1.AddWithResize(T) + 0x68",
          "at OpenTelemetry.AWS.AWSSemanticConventions.Add(AWSSemanticConventions.AttributeBuilderImpl, Func`2, Object, Boolean) + 0xb0",
          "at OpenTelemetry.Instrumentation.AWSLambda.Implementation.AWSLambdaUtils.GetFunctionTags[TInput](TInput, ILambdaContext, Boolean) + 0x6c",
          "at OpenTelemetry.Instrumentation.AWSLambda.AWSLambdaWrapper.OnFunctionStart[TInput](TInput, ILambdaContext, ActivityContext) + 0x14c",
          "at OpenTelemetry.Instrumentation.AWSLambda.AWSLambdaWrapper.<TraceInternalAsync>d__20`2.MoveNext() + 0x70",
          "--- End of stack trace from previous location ---",
          "at MartinCostello.LondonTravel.Skill.AlexaFunction.<HandlerAsync>d__5.MoveNext() + 0x6c",
          "--- End of stack trace from previous location ---",
          "at Amazon.Lambda.RuntimeSupport.HandlerWrapper.<>c__DisplayClass26_0`2.<<GetHandlerWrapper>b__0>d.MoveNext() + 0x68",
          "--- End of stack trace from previous location ---",
          "at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.<InvokeOnceAsync>d__22.MoveNext() + 0x1f8"
        ]
      }

Looks like some sort of concurrency issue when the internal array backing a List<T> is being resized as part of an Add() operation.

This comment is of interest on the call stack as it suggests there shouldn't ever be any concurrency here:

opentelemetry-dotnet-contrib/src/OpenTelemetry.Instrumentation.AWSLambda/AWSLambdaWrapper.cs

Lines 168 to 169 in f4dff45

	// No parallel invocation of the same lambda handler expected.
	var functionTags = new AWSLambdaUtils(AWSSemanticConventions).GetFunctionTags(input, context, isColdStart);

There is no explicit concurrency on my part, other than any concurrency that the test framework may be performing.

The specific test suite running when the failure was observed is using the MSTest SDK running with native AoT on macOS.

Steps to Reproduce

This is the first time I've seen this issue, and the stack trace suggests it's related to concurrency in some way, so there isn't a way to definitively reproduce it.

The tests were run from this commit: martincostello/alexa-london-travel@c6fced4.

Expected Result

No exception thrown.

Actual Result

fail: MartinCostello.Testing.AwsLambdaTestServer.RuntimeHandler[0]
      Error invoking Lambda function with ARN arn:aws:lambda:eu-west-1:123456789012:function:test-function for request Id 80d04f94-3177-461e-9f61-e0e4b4ce78f6: {
        "errorType": "ArgumentException",
        "errorMessage": "Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')",
        "stackTrace": [
          "at System.Array.CopyImpl(Array, Int32, Array, Int32, Int32, Boolean) + 0x3a4",
          "at System.Collections.Generic.List`1.set_Capacity(Int32) + 0x90",
          "at System.Collections.Generic.List`1.AddWithResize(T) + 0x68",
          "at OpenTelemetry.AWS.AWSSemanticConventions.Add(AWSSemanticConventions.AttributeBuilderImpl, Func`2, Object, Boolean) + 0xb0",
          "at OpenTelemetry.Instrumentation.AWSLambda.Implementation.AWSLambdaUtils.GetFunctionTags[TInput](TInput, ILambdaContext, Boolean) + 0x6c",
          "at OpenTelemetry.Instrumentation.AWSLambda.AWSLambdaWrapper.OnFunctionStart[TInput](TInput, ILambdaContext, ActivityContext) + 0x14c",
          "at OpenTelemetry.Instrumentation.AWSLambda.AWSLambdaWrapper.<TraceInternalAsync>d__20`2.MoveNext() + 0x70",
          "--- End of stack trace from previous location ---",
          "at MartinCostello.LondonTravel.Skill.AlexaFunction.<HandlerAsync>d__5.MoveNext() + 0x6c",
          "--- End of stack trace from previous location ---",
          "at Amazon.Lambda.RuntimeSupport.HandlerWrapper.<>c__DisplayClass26_0`2.<<GetHandlerWrapper>b__0>d.MoveNext() + 0x68",
          "--- End of stack trace from previous location ---",
          "at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.<InvokeOnceAsync>d__22.MoveNext() + 0x1f8"
        ]
      }

Additional Context

No response

[github-actions]

Tagging component owner(s).

@srprash @ppittle @muhammad-othman @rypdal @Oberon00

[ppittle]

I have a candidate fix - set initial capacity of list so it shouldn't need to expand: #2653

@martincostello - could you help test: #2653 (comment)

[ppittle]

#2653 has merged.

Preparing release in #2661

[Kielek]

Release should be available on nuget.org within couple of minutes. Closing.
GH release is available https://github.com/open-telemetry/opentelemetry-dotnet-contrib/releases/tag/Instrumentation.AWS-1.11.2