Skip to content

Disable exception alerting when service is disabled#1383

Merged
MatMoore merged 1 commit intomainfrom
disable-5xx-alerts-when-environment-is-off
Apr 30, 2026
Merged

Disable exception alerting when service is disabled#1383
MatMoore merged 1 commit intomainfrom
disable-5xx-alerts-when-environment-is-off

Conversation

@MatMoore
Copy link
Copy Markdown
Contributor

@MatMoore MatMoore commented Apr 28, 2026

Description

If SERVICE_ENABLED is false we put up a maintenence page which is a 503 response. This is considered an exception so it sets off our app insights exception alert.

Instead I was thinking we can just not enable this alert in this situation.

The rest of the alerts should still be enabled as the environment is still running regardless of whether the app is serving normal requests.

Jira link

Review notes

I moved locals.env_from_yaml to the top level in order to make the variables accessible in the infra module. Not sure if this is the best way to do it.

I wasn't particularly confident in my terraform-fu so I ran some terraform plans to check, and it will have the intended effect of destroying the appi-prod-uks-manbrs-exceptions-alert on prod.

The plan also includes some update-in-place stuff I wasn't expecting. Not sure where the drift has come from as it seems unrelated to my change - is it safe to ignore?

E.g. this one is being updated to match the devops templates, which has enabled_metric = ["AllMetrics"] `

  # module.container-apps[0].module.storage.module.diagnostic-settings["queueServices"].azurerm_monitor_diagnostic_setting.this will be updated in-place
  ~ resource "azurerm_monitor_diagnostic_setting" "this" {
        id                             = "/subscriptions/b3182f91-c8a7-4be0-be3e-0b34a95a48d7/resourceGroups/rg-manbrs-dev-container-app-uks/providers/Microsoft.Storage/storageAccounts/stmanbrsdevuks/queueServices/default|stmanbrsdevuks-diagnotic-setting-storage"
        name                           = "stmanbrsdevuks-diagnotic-setting-storage"
        # (5 unchanged attributes hidden)

      - enabled_metric {
          - category = "Capacity" -> null
        }
      - enabled_metric {
          - category = "Transaction" -> null
        }
      + enabled_metric {
          + category = "AllMetrics"
        }

        # (5 unchanged blocks hidden)
    }

  # module.container-apps[0].module.storage.module.diagnostic-settings["tableServices"].azurerm_monitor_diagnostic_setting.this will be updated in-place
  ~ resource "azurerm_monitor_diagnostic_setting" "this" {
        id                             = "/subscriptions/b3182f91-c8a7-4be0-be3e-0b34a95a48d7/resourceGroups/rg-manbrs-dev-container-app-uks/providers/Microsoft.Storage/storageAccounts/stmanbrsdevuks/tableServices/default|stmanbrsdevuks-diagnotic-setting-storage"
        name                           = "stmanbrsdevuks-diagnotic-setting-storage"
        # (5 unchanged attributes hidden)

      - enabled_metric {
          - category = "Capacity" -> null
        }
      - enabled_metric {
          - category = "Transaction" -> null
        }
      + enabled_metric {
          + category = "AllMetrics"
        }

        # (5 unchanged blocks hidden)
    }

Review checklist

  • Check database queries are correctly scoped to current_provider
  • If this changes the gateway API (/api/v1/), confirm whether it is a breaking change — if so, a new major version (/api/v2/) is required (see ADR-006)

@MatMoore MatMoore force-pushed the disable-5xx-alerts-when-environment-is-off branch from dee171a to 9e150d9 Compare April 28, 2026 14:05
If SERVICE_ENABLED is false we put up a maintenence page which is a 503
response. This is considered an exception so it sets off our app
insights exception alert.

Instead we can just not enable this alert in this situation, however the
rest of the alerts are still enabled as the environment is still
running regardless of whether the app is serving normal requests.
@MatMoore MatMoore force-pushed the disable-5xx-alerts-when-environment-is-off branch from 9e150d9 to 6eb1f10 Compare April 28, 2026 14:15
@sonarqubecloud
Copy link
Copy Markdown

@MatMoore MatMoore marked this pull request as ready for review April 29, 2026 15:29
@MatMoore MatMoore merged commit 4d937ce into main Apr 30, 2026
12 checks passed
@MatMoore MatMoore deleted the disable-5xx-alerts-when-environment-is-off branch April 30, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants