Datadog v2 API fails when using formulas in query

[tukak]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

Using formulas in AnalysisTemplate with Datadog v2 API fails with API error

received non 2xx response code: 400 {"errors":["Functions are not supported in metrics data source field. Use the formulas payload."]}

To Reproduce

Use a query with formula in Datadog provider for AnalysisTemplate and set apiRevision to v2. The query works with v1.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: rollout-test-success-rate
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: default(result, 0) < 10
      provider:
        datadog:
          apiVersion: v2
          interval: 1m
          query: |
            sum:trace.rack.request.hits{env:staging AND service:service_name AND http.status_code:5*}.as_count() /
            sum:trace.rack.request.hits{env:staging AND service:service_name AND http.status_code:2*}.as_count() * 100

Expected behavior

It should work with v2 as well, or the note in https://argo-rollouts.readthedocs.io/en/stable/analysis/datadog/ should warn about using formulas (now it says that "If you switch to v2, you will not need to change any other field aside from apiVersion.")

Version

1.5.1

Logs

time="2023-05-26T10:02:02Z" level=info msg="Taking 1 Measurement(s)..." analysisrun=rollout-test-6b986bcd5b-7.1 namespace=default
time="2023-05-26T10:02:02Z" level=info msg="Measurement Completed. Result: Error" analysisrun=rollout-test-6b986bcd5b-7.1 metric=success-rate namespace=default
time="2023-05-26T10:02:02Z" level=warning msg="Measurement had error: received non 2xx response code: 400 {\"errors\":[\"Functions are not supported in metrics data source field. Use the formulas payload.\"]}" analysisrun=rollout-test-6b986bcd5b-7.1 metric=success-rate namespace=default

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[alexef]

Formulas also need separate named queries, so in your case it could look like this:

   provider:
        datadog:
          apiVersion: v2
          interval: 1m
          queries:
            a: sum:trace.rack.request.hits{env:staging AND service:service_name AND http.status_code:5*}.as_count()
            b: sum:trace.rack.request.hits{env:staging AND service:service_name AND http.status_code:2*}.as_count() * 100
          formula: a/b

this is NOT supported yet by argo-rollouts, I will update the docs accordingly.

[alexef]

created: #2819

[meeech]

I'm interested in grabbing this one if no one has yet

[joey100]

Formulas also need separate named queries, so in your case it could look like this:

   provider:
        datadog:
          apiVersion: v2
          interval: 1m
          queries:
            a: sum:trace.rack.request.hits{env:staging AND service:service_name AND http.status_code:5*}.as_count()
            b: sum:trace.rack.request.hits{env:staging AND service:service_name AND http.status_code:2*}.as_count() * 100
          formula: a/b

this is NOT supported yet by argo-rollouts, I will update the docs accordingly.

May I know any plan to support this, when will we be able to use this? This is important if we switch to api version v2, there are lots of functions and formulas will be used in many datadog metrics queries. Otherwise we won't be able to use api version v2.

[meeech]

@joey100 just picked up the ticket the other day, so I can't give you a timeframe. that being said, one reason I picked up the ticket is we rely on formulas as well and would like to move to v2.
Will add details once i have some to share

[Sineaggi]

Related, would it also make sense to use the datadog-api-client-go library to make our calls to Datadog?

[meeech]

@Sineaggi I'll look into that as well and see if it makes sense

[meeech]

Adding more context: I will follow up on this with datadog.

The reason we think v1 is deprecated is based on @alexef expeirence
(from slack)
"I don’t have a link, we were getting 429 throttled by the API, and when asking for a rate limit increase, we were told to switch to v2 as they can increase the rate there, but not on v1"
"in communication to datadog support they referred to v1 as the “legacy one”, that’s all I have"

I will follow up with DD, since when working with their go lib, I get warning "2023/08/29 13:21:49 WARNING: Using unstable operation 'v2.QueryTimeseriesData'" so I want to get all this info sorted so we have clear guidance.