Skip to content

Fix percentile metric_aggregation rule error for rules with compound query keys - #1701

Merged
jertel merged 1 commit into
jertel:masterfrom
jhatcher1:fix-percentile-aggregation-with-compound-key
Sep 15, 2025
Merged

Fix percentile metric_aggregation rule error for rules with compound query keys#1701
jertel merged 1 commit into
jertel:masterfrom
jhatcher1:fix-percentile-aggregation-with-compound-key

Conversation

@jhatcher1

Copy link
Copy Markdown
Contributor

Description

This PR fixes metric_aggregation rules of type percentile, when used with a compound query key.

Prior to this fix, using such a rule would raise the following KeyError:

ERROR:elastalert:Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/elastalert/elastalert.py", line 1277, in handle_rule_execution
    num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
  File "/usr/local/lib/python3.13/site-packages/elastalert/elastalert.py", line 900, in run_rule
    if not self.run_query(rule, tmp_endtime, endtime):
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/elastalert/elastalert.py", line 654, in run_query
    rule_inst.add_aggregation_data(data)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 1041, in add_aggregation_data
    self.unwrap_term_buckets(timestamp, payload_data['bucket_aggs']['buckets'])
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 1055, in unwrap_term_buckets
    self.check_matches(timestamp, term_data['key'], term_data)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 1103, in check_matches
    self.check_matches_recursive(timestamp, query_key, aggregation_data, self.rules['compound_query_key'], dict())
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 1131, in check_matches_recursive
    self.check_matches_recursive(timestamp,
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
                                 query_key,
                                 ^^^^^^^^^^
                                 result,
                                 ^^^^^^^
                                 compound_keys[1:],
                                 ^^^^^^^^^^^^^^^^^^
                                 match_data)
                                 ^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/elastalert/ruletypes.py", line 1140, in check_matches_recursive
    metric_val_arr = [aggregation_data[self.metric_key]['value']]
                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'value'

I was able to reproduce this issue with the following minimal example:

  1. Insert some documents to a local elasticsearch cluster:
  curl -X POST "http://localhost:9200/logstash-test/_bulk" \
    -H 'Content-Type: application/json' \
    -d '
  { "index": { "_index": "logstash-test" } }
  { "environment": "production", "kubernetes": { "pod_name": "example-pod" }, "query_state": "complete", "region": "europe", "message": "Query complete: took 225ms", "query_elapsed_ms": 225, "@timestamp": "2025-09-12T10:21:36.831Z" }
  { "index": { "_index": "logstash-test" } }
  { "environment": "staging", "kubernetes": { "pod_name": "example-pod" }, "query_state": "complete", "region": "europe", "message": "Query complete: took 225ms", "query_elapsed_ms": 225, "@timestamp": "2025-09-12T10:21:36.831Z" }
  { "index": { "_index": "logstash-test" } }
  { "environment": "production", "kubernetes": { "pod_name": "example-pod" }, "query_state": "complete", "region": "US", "message": "Query complete: took 225ms", "query_elapsed_ms": 225, "@timestamp": "2025-09-12T10:21:36.831Z" }
  '
  1. Create a file rule.yaml
name: High query time rule
type: metric_aggregation
index: logstash-*

metric_agg_key: query_elapsed_ms
metric_agg_type: percentiles
percentile_range: 90

timeframe:
  minutes: 5

filter:
  - query:
      query_string:
        query: |
          (kubernetes.pod_name:"example-pod")
          AND (query_state:"complete")

alert:
  - "debug"

max_threshold: 1

query_key:
  - environment.keyword
  - region.keyword
  1. Test the rule with elastalert-test-rule rule.yaml --days 10
  2. The KeyError is observed without the fix, and alerts are fired as expected with the fix applied.

Checklist

  • I have reviewed the contributing guidelines.
  • I have included unit tests for my changes or additions.
  • I have successfully run make test-docker with my changes.
  • I have manually tested all relevant modes of the change in this PR.
  • I have updated the documentation.
  • I have updated the changelog.

Questions or Comments

@jertel
jertel merged commit f582d94 into jertel:master Sep 15, 2025
1 check passed
@jertel

jertel commented Sep 16, 2025

Copy link
Copy Markdown
Owner

Thanks for fixing this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants