CosmosDB Metrics Retrieval Failed

[shivtej-opsverse]

• I am trying to pull metrics from a Azure Cosmos DB for NoSQL account into prometheus using the following scrape config

scrape_configs:
  - job_name: azure-metrics-example
    scrape_interval: 1m
    metrics_path: /probe/metrics/list
    params:
      name:
        - azure_metric
      template:
        - '{name}_{metric}'
      help:
        - Azure metric {metric} for {aggregation}
      subscription:
        - <subscription_id>
      resourceType:
        - Microsoft.DocumentDB/DatabaseAccounts
      metric:
        - DeleteAccount
        - CreateAccount
        - UpdateDiagnosticsSettings
        - UpdateAccountKeys
        - UpdateAccountNetworkSettings
        - UpdateAccountReplicationSettings
        - AutoscaleMaxThroughput
        - AutoscaledRU
        - DataUsage
        - DedicatedGatewayAverageCPUUsage
        - DedicatedGatewayAverageMemoryUsage
        - TotalRequests
        - DedicatedGatewayMaximumCPUUsage
        - DocumentCount
        - DocumentQuota
        - IndexUsage
        - IntegratedCacheEvictedEntriesSize
        - IntegratedCacheItemExpirationCount
        - IntegratedCacheItemHitRate
        - IntegratedCacheQueryExpirationCount
        - IntegratedCacheQueryHitRate
        - MaterializedViewCatchupGapInMinutes
        - MaterializedViewsBuilderAverageCPUUsage
        - MaterializedViewsBuilderAverageMemoryUsage
        - MaterializedViewsBuilderMaximumCPUUsage
        - MetadataRequests
        - NormalizedRUConsumption
        - PhysicalPartitionCount
        - ReplicationLatency
        - PhysicalPartitionSizeInfo
        - PhysicalPartitionThroughputInfo
        - ProvisionedThroughput
        - AddRegion
        - RegionFailover
        - OfflineRegion
        - OnlineRegion
        - RemoveRegion
        - ServerSideLatency
        - ServerSideLatencyDirect
        - ServerSideLatencyGateway
        - ServiceAvailability
        - SqlContainerCreate
        - SqlContainerDelete
        - SqlContainerThroughputUpdate
        - SqlContainerUpdate
        - SqlDatabaseCreate
        - SqlDatabaseDelete
        - SqlDatabaseThroughputUpdate
        - SqlDatabaseUpdate
        - TableTableCreate
        - TotalRequestsPreview
        - TotalRequestUnits
        - TotalRequestUnitsPreview
      validateDimensions:
        - 'true'
    static_configs:
      - targets:
          - url-to-your-azure-metrics-exporter-instance

• The exporter only responnd's with the following metrics

# HELP azure_metric_documentquota Azure metric DocumentQuota for total
# TYPE azure_metric_documentquota gauge
azure_metric_documentquota{<labels>} 0
# HELP azure_metric_totalrequests Azure metric TotalRequests for count
# TYPE azure_metric_totalrequests gauge
azure_metric_totalrequests{<labels>} 0

• There is warning log which says

{"level":"warn","caller":"metrics/prober.go:345","msg":"GET https://management.azure.com/subscriptions/<resource-uri>/providers/Microsoft.Insights/metrics\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: BadRequest\n--------------------------------------------------------------------------------\n{\n  \"code\": \"BadRequest\",\n  \"message\": \"Failed to find a common time grain between request metrics: IntegratedCacheQueryHitRate,MaterializedViewCatchupGapInMinutes,MaterializedViewsBuilderAverageCPUUsage,MaterializedViewsBuilderAverageMemoryUsage,MaterializedViewsBuilderMaximumCPUUsage,MetadataRequests,NormalizedRUConsumption,PhysicalPartitionCount,ReplicationLatency,PhysicalPartitionSizeInfo,PhysicalPartitionThroughputInfo,ProvisionedThroughput,AddRegion,RegionFailover,OfflineRegion,OnlineRegion,RemoveRegion,ServerSideLatency,ServerSideLatencyDirect,ServerSideLatencyGateway, TraceId: {84806b23-d5b1-40fa-ba82-54c04d2ebbc9}\"\n}\n--------------------------------------------------------------------------------\n","requestPath":"/probe/metrics/list","paramValidateDimensions":["true"],"paramName":["azure_metric"],"paramTemplate":["{name}_{metric}"],"paramHelp":["Azure metric {metric} for {aggregation}"],"paramSubscription":["<subscriptionId>"],"paramResourceType":["Microsoft.DocumentDB/DatabaseAccounts"],"paramMetric":["DeleteAccount,CreateAccount,UpdateDiagnosticsSettings,UpdateAccountKeys,UpdateAccountNetworkSettings,UpdateAccountReplicationSettings,AutoscaleMaxThroughput,AutoscaledRU,DataUsage,DedicatedGatewayAverageCPUUsage,DedicatedGatewayAverageMemoryUsage,TotalRequests,DedicatedGatewayMaximumCPUUsage,DocumentCount,DocumentQuota,IndexUsage,IntegratedCacheEvictedEntriesSize,IntegratedCacheItemExpirationCount,IntegratedCacheItemHitRate,IntegratedCacheQueryExpirationCount,IntegratedCacheQueryHitRate,MaterializedViewCatchupGapInMinutes,MaterializedViewsBuilderAverageCPUUsage,MaterializedViewsBuilderAverageMemoryUsage,MaterializedViewsBuilderMaximumCPUUsage,MetadataRequests,NormalizedRUConsumption,PhysicalPartitionCount,ReplicationLatency,PhysicalPartitionSizeInfo,PhysicalPartitionThroughputInfo,ProvisionedThroughput,AddRegion,RegionFailover,OfflineRegion,OnlineRegion,RemoveRegion,ServerSideLatency,ServerSideLatencyDirect,ServerSideLatencyGateway,ServiceAvailability,SqlContainerCreate,SqlContainerDelete,SqlContainerThroughputUpdate,SqlContainerUpdate,SqlDatabaseCreate,SqlDatabaseDelete,SqlDatabaseThroughputUpdate,SqlDatabaseUpdate,TableTableCreate,TotalRequestsPreview,TotalRequestUnits,TotalRequestUnitsPreview"],"resourceID":"<resource-uri>"}

Is something wrong with my config.
Any help would be appreciated.

[WhitneyLampkin]

Hi @shivtej-opsverse, I know this is over a year too late but hopefully this response will help others. I took a detailed look at the log message you posted and the message points to the issue:

"Failed to find a common time grain between request metrics"

I've ran into this error as well. Azure Monitor doesn't support requests for metrics with different time grains. This same error message shows in Azure Metrics when trying to view the same metrics in the Azure Portal. In my example below, I configured all of the metrics with the PT1H time grain, but TotalRequests only has one time grain available, which is PT1M. This leads to the same error.