Oversampling mitigation

[atshaw43]

Issue #, if available:

Description of changes:
Oversampling Mitigation and SQS Helper

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 66.07143% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.54%. Comparing base (9487f19) to head (fcba4a0).
⚠️ Report is 56 commits behind head on master.

Files with missing lines	Patch %	Lines
...a/com/amazonaws/xray/entities/DummySubsegment.java	0.00%	5 Missing ⚠️
.../amazonaws/xray/contexts/LambdaSegmentContext.java	63.63%	1 Missing and 3 partials ⚠️
...onaws/xray/contexts/ThreadLocalSegmentContext.java	57.14%	1 Missing and 2 partials ⚠️
.../main/java/com/amazonaws/xray/AWSXRayRecorder.java	50.00%	1 Missing and 1 partial ⚠️
...va/com/amazonaws/xray/lambda/SQSMessageHelper.java	33.33%	1 Missing and 1 partial ⚠️
...core/src/main/java/com/amazonaws/xray/AWSXRay.java	0.00%	1 Missing ⚠️
...n/java/com/amazonaws/xray/entities/Subsegment.java	75.00%	1 Missing ⚠️
...va/com/amazonaws/xray/entities/SubsegmentImpl.java	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #354      +/-   ##
============================================
+ Coverage     58.30%   58.54%   +0.23%     
- Complexity     1209     1234      +25     
============================================
  Files           131      133       +2     
  Lines          4948     4986      +38     
  Branches        590      595       +5     
============================================
+ Hits           2885     2919      +34     
- Misses         1788     1793       +5     
+ Partials        275      274       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[atshaw43]

Sample code in Lambda

package example;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.SQSEvent;
import com.amazonaws.services.lambda.runtime.events.SQSEvent.SQSMessage;

import com.amazonaws.xray.AWSXRayRecorder;
import com.amazonaws.xray.sqs.SQSMessageHelper;
import com.amazonaws.xray.AWSXRay;

import java.lang.StringBuilder;
import java.util.Map;
import java.util.List;
import java.util.concurrent.CompletableFuture;

import io.opentelemetry.api.trace.Span;

public class Handler implements RequestHandler<SQSEvent, String> {
  public Handler(){
  }

  @Override
  public String handleRequest(SQSEvent event, Context context) {

    // This variable will override the SamplingStrategy to whatever we need.
    boolean toSample = false;

    for (SQSMessage message: event.getRecords()) {
      // This helper function makes it easier to tell if a SQS message is sampled.
      // There are some magic strings involved that we are hiding from the customer.
      if (SQSMessageHelper.isSampled(message)) {
        toSample = true;
      }
    }
    
    // This is the new function that allows the override of the SamplingStrategy.
    AWSXRay.getGlobalRecorder().beginSubsegmentWithSamplingOverride("Batch Trace 1", toSample);
    AWSXRay.getGlobalRecorder().endSubsegment();

    // Testing making two in a single event for good measure.
    AWSXRay.getGlobalRecorder().beginSubsegmentWithSamplingOverride("Batch Trace 2", toSample);
    AWSXRay.getGlobalRecorder().endSubsegment();
  }
}

Test 1 - This is the JSON sent to lambda

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "Hello from SQS!",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "AWSTraceHeader": "Root=1-632BB806-bd862e3fe1be46a994272793;Sampled=1"
      },
      "messageAttributes": {},
      "md5OfBody": "{{{md5_of_body}}}",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "awsRegion": "us-east-1"
    }
  ]
}

Test 2 - Note the different Sampled value

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "Hello from SQS!",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "AWSTraceHeader": "Root=1-632BB806-bd862e3fe1be46a994272793;Sampled=0"
      },
      "messageAttributes": {},
      "md5OfBody": "{{{md5_of_body}}}",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "awsRegion": "us-east-1"
    }
  ]
}

[wangzlei]

XRayLambdaSQSMessageHelper? it might be a horrible name.

[atshaw43]

I think it makes sense to change the package/module name from SQS to lambda. But let's keep the class name concise. This is what they did in the Lambda library where SQSMessage lives.

[wangzlei]

Module name could be lambda helper since it is help Lambda user, not for sqs instrumentation

[atshaw43]

Done

[wangzlei]

Not quite get why have to use DISABLED.
Can we use existing API, then update the sampled flag in SegmentImpl?

[atshaw43]

If we are changing the parent's flag (the Lambda Frontend Segment), then will that not cause an issue in multithreading?

[wangzlei]

When parentSegment.isRecording() == false and samplingStrategyOverride == SamplingStrategyOverride.TRUE, should we create a new subsegment?

[atshaw43]

Yes. The override to true will ignore any other indicators.
I guess this creates a problem of an orphaned subsegment?