Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
When you enable Prometheus metrics collection in Azure Monitor from your Kubernetes cluster, it uses a default configuration for targets, dashboards, and recording rules. This article describes the default configuration and the scenarios where you may choose to customize it for your specific requirements.
Minimal ingestion profile
Minimal ingestion profile is a setting that is enabled by default when Prometheus metrics are enabled in Azure Monitor for a cluster. This setting reduces the volume of metrics ingested by limiting them to only metrics used by default dashboards, default recording rules, and default alerts. These targets and metrics are listed in this article. If this setting is disabled, then all available metrics for the default targets are collected which can significantly increase ingestion volume.
You can change the minimal ingestion profile setting by modifying the metrics setting ConfigMap as described in Customize scraping of Prometheus metrics in Azure Monitor using ConfigMap.
Customization scenarios
You may choose to use the default configuration or customize collection for your particular requirements. The following table lists the four potential collection scenarios and the recommended method to achieve each.
| Scenario | Method |
|---|---|
| Ingest only minimal metrics for each default target. | No change needed. Use the default behavior without modification. Only metrics listed in this article are ingested for each of the default targets. |
| Ingest a few other metrics for one or more default targets in addition to minimal metrics. | Keep minimal ingestion enabled and specify the appropriate keep lists specific to the target. See Customize metrics collected by default targets. |
| Ingest only a specific set of metrics for a default target. | Disable minimal ingestion and specify the appropriate keep list specific to the target in a custom scrape job. See Create custom Prometheus scrape job from your Kubernetes cluster using ConfigMap. |
| Ingest all metrics scraped for the default target. | Disable minimal ingestion and don't specify any keep list for that target. See Customize metrics collected by default targets |
Targets scraped by default
Following are the targets that the Azure Monitor metrics add-on can scrape by default and the conditions under which they are enabled. See Enable and disable default targets to enable/disable default targets.
The following targets are enabled by default.
cadvisornodeexporterkubeletkube-state-metricsnetworkobservabilityRetina
Th following targets are enabled when control plane metrics (preview) is enabled.
controlplane-apiservercontrolplane-etcd
The following targets are enable when Container Network Observability is enabled.
networkobservabilityHubblenetworkobservabilityCilium
The following targets are enabled when Azure Container Storage is enabled.
acstor-capacity-provisioneracstor-metrics-exporter
The following targets are disabled by default.
corednskubeproxyapiserver
The following targets are disabled by default and require Windows metrics collection (preview) to be enabled.
windows-exporterkube-proxy-windows
Metrics collected from default targets
The following metrics are collected by default from each default target. All other metrics are dropped through relabeling rules. The target must be enabled for the metrics to be collected.
kubelet
kubelet_volume_stats_used_byteskubelet_node_namekubelet_running_podskubelet_running_pod_countkubelet_running_containerskubelet_running_container_countvolume_manager_total_volumeskubelet_node_config_errorkubelet_runtime_operations_totalkubelet_runtime_operations_errors_totalkubelet_runtime_operations_duration_secondskubelet_runtime_operations_duration_seconds_bucketkubelet_runtime_operations_duration_seconds_sumkubelet_runtime_operations_duration_seconds_countkubelet_pod_start_duration_secondskubelet_pod_start_duration_seconds_bucketkubelet_pod_start_duration_seconds_sumkubelet_pod_start_duration_seconds_countkubelet_pod_worker_duration_secondskubelet_pod_worker_duration_seconds_bucketkubelet_pod_worker_duration_seconds_sumkubelet_pod_worker_duration_seconds_countstorage_operation_duration_secondsstorage_operation_duration_seconds_bucketstorage_operation_duration_seconds_sumstorage_operation_duration_seconds_countstorage_operation_errors_totalkubelet_cgroup_manager_duration_secondskubelet_cgroup_manager_duration_seconds_bucketkubelet_cgroup_manager_duration_seconds_sumkubelet_cgroup_manager_duration_seconds_countkubelet_pleg_relist_duration_secondskubelet_pleg_relist_duration_seconds_bucketkubelet_pleg_relist_duration_sumkubelet_pleg_relist_duration_seconds_countkubelet_pleg_relist_interval_secondskubelet_pleg_relist_interval_seconds_bucketkubelet_pleg_relist_interval_seconds_sumkubelet_pleg_relist_interval_seconds_countrest_client_requests_totalrest_client_request_duration_secondsrest_client_request_duration_seconds_bucketrest_client_request_duration_seconds_sumrest_client_request_duration_seconds_countprocess_resident_memory_bytesprocess_cpu_seconds_totalgo_goroutineskubelet_volume_stats_capacity_byteskubelet_volume_stats_available_byteskubelet_volume_stats_inodes_usedkubelet_volume_stats_inodeskubernetes_build_info"
coredns
coredns_build_infocoredns_panics_totalcoredns_dns_responses_totalcoredns_forward_responses_totalcoredns_dns_request_duration_secondscoredns_dns_request_duration_seconds_bucketcoredns_dns_request_duration_seconds_sumcoredns_dns_request_duration_seconds_countcoredns_forward_request_duration_secondscoredns_forward_request_duration_seconds_bucketcoredns_forward_request_duration_seconds_sumcoredns_forward_request_duration_seconds_countcoredns_dns_requests_totalcoredns_forward_requests_totalcoredns_cache_hits_totalcoredns_cache_misses_totalcoredns_cache_entriescoredns_plugin_enabledcoredns_dns_request_size_bytescoredns_dns_request_size_bytes_bucketcoredns_dns_request_size_bytes_sumcoredns_dns_request_size_bytes_countcoredns_dns_response_size_bytescoredns_dns_response_size_bytes_bucketcoredns_dns_response_size_bytes_sumcoredns_dns_response_size_bytes_countcoredns_dns_response_size_bytescoredns_dns_response_size_bytes_bucketcoredns_dns_response_size_bytes_sumcoredns_dns_response_size_bytes_countprocess_resident_memory_bytesprocess_cpu_seconds_totalgo_goroutineskubernetes_build_info"
cadvisor
container_spec_cpu_periodcontainer_spec_cpu_quotacontainer_cpu_usage_seconds_totalcontainer_memory_rsscontainer_network_receive_bytes_totalcontainer_network_transmit_bytes_totalcontainer_network_receive_packets_totalcontainer_network_transmit_packets_totalcontainer_network_receive_packets_dropped_totalcontainer_network_transmit_packets_dropped_totalcontainer_fs_reads_totalcontainer_fs_writes_totalcontainer_fs_reads_bytes_totalcontainer_fs_writes_bytes_totalcontainer_memory_working_set_bytescontainer_memory_cachecontainer_memory_swapcontainer_cpu_cfs_throttled_periods_totalcontainer_cpu_cfs_periods_totalkubernetes_build_info"
kubeproxy
kubeproxy_sync_proxy_rules_duration_secondskubeproxy_sync_proxy_rules_duration_seconds_bucketkubeproxy_sync_proxy_rules_duration_seconds_sumkubeproxy_sync_proxy_rules_duration_seconds_countkubeproxy_network_programming_duration_secondskubeproxy_network_programming_duration_secondskubeproxy_network_programming_duration_seconds_bucketkubeproxy_network_programming_duration_seconds_sumkubeproxy_network_programming_duration_seconds_countrest_client_requests_totalrest_client_request_duration_secondsrest_client_request_duration_seconds_bucketrest_client_request_duration_seconds_sumrest_client_request_duration_seconds_countprocess_resident_memory_bytesprocess_cpu_seconds_totalgo_goroutineskubernetes_build_info"
apiserver
apiserver_request_duration_secondsapiserver_request_duration_seconds_bucketapiserver_request_duration_seconds_sumapiserver_request_duration_seconds_countapiserver_request_totalworkqueue_adds_total``workqueue_depthworkqueue_queue_duration_secondsworkqueue_queue_duration_seconds_bucketworkqueue_queue_duration_seconds_sumworkqueue_queue_duration_seconds_countprocess_resident_memory_bytesprocess_cpu_seconds_totalgo_goroutineskubernetes_build_info"
kube-state
kube_job_status_succeededkube_job_spec_completionskube_daemonset_status_desired_number_scheduledkube_daemonset_status_number_readykube_deployment_status_replicas_readykube_pod_container_status_last_terminated_reasonkube_pod_container_status_waiting_reasonkube_pod_container_status_restarts_totalkube_node_status_allocatablekube_pod_ownerkube_pod_container_resource_requestskube_pod_status_phasekube_pod_container_resource_limitskube_replicaset_ownerkube_resourcequotakube_namespace_status_phasekube_node_status_capacitykube_node_infokube_pod_infokube_deployment_spec_replicaskube_deployment_status_replicas_availablekube_deployment_status_replicas_updatedkube_statefulset_status_replicas_readykube_statefulset_status_replicaskube_statefulset_status_replicas_updatedkube_job_status_start_timekube_job_status_activekube_job_failedkube_horizontalpodautoscaler_status_desired_replicaskube_horizontalpodautoscaler_status_current_replicaskube_horizontalpodautoscaler_spec_min_replicaskube_horizontalpodautoscaler_spec_max_replicaskubernetes_build_infokube_node_status_conditionkube_node_spec_taintkube_pod_container_infokube_resource_labels(ex - kube_pod_labels, kube_deployment_labels)kube_resource_annotations(ex - kube_pod_annotations, kube_deployment_annotations)
nodeexporter
node_cpu_seconds_totalnode_memory_MemAvailable_bytesnode_memory_Buffers_bytesnode_memory_Cached_bytesnode_memory_MemFree_bytesnode_memory_Slab_bytesnode_memory_MemTotal_bytesnode_netstat_Tcp_RetransSegsnode_netstat_Tcp_OutSegsnode_netstat_TcpExt_TCPSynRetransnode_load1``node_load5node_load15node_disk_read_bytes_totalnode_disk_written_bytes_totalnode_disk_io_time_seconds_totalnode_filesystem_size_bytesnode_filesystem_avail_bytesnode_filesystem_readonlynode_network_receive_bytes_totalnode_network_transmit_bytes_totalnode_vmstat_pgmajfaultnode_network_receive_drop_totalnode_network_transmit_drop_totalnode_disk_io_time_weighted_seconds_totalnode_exporter_build_infonode_time_secondsnode_uname_info"
windowsexporter
windows_system_system_up_timewindows_cpu_time_totalwindows_memory_available_byteswindows_os_visible_memory_byteswindows_memory_cache_byteswindows_memory_modified_page_list_byteswindows_memory_standby_cache_core_byteswindows_memory_standby_cache_normal_priority_byteswindows_memory_standby_cache_reserve_byteswindows_memory_swap_page_operations_totalwindows_logical_disk_read_seconds_totalwindows_logical_disk_write_seconds_totalwindows_logical_disk_size_byteswindows_logical_disk_free_byteswindows_net_bytes_totalwindows_net_packets_received_discarded_totalwindows_net_packets_outbound_discarded_totalwindows_container_availablewindows_container_cpu_usage_seconds_totalwindows_container_memory_usage_commit_byteswindows_container_memory_usage_private_working_set_byteswindows_container_network_receive_bytes_totalwindows_container_network_transmit_bytes_total
windowskubeproxy
kubeproxy_sync_proxy_rules_duration_secondskubeproxy_sync_proxy_rules_duration_seconds_bucketkubeproxy_sync_proxy_rules_duration_seconds_sumkubeproxy_sync_proxy_rules_duration_seconds_countrest_client_requests_totalrest_client_request_duration_secondsrest_client_request_duration_seconds_bucketrest_client_request_duration_seconds_sumrest_client_request_duration_seconds_countprocess_resident_memory_bytesprocess_cpu_seconds_totalgo_goroutines
networkobservabilityHubble
networkobservabilityCilium
controlplane-apiserver
apiserver_request_totalapiserver_cache_list_fetched_objects_totalapiserver_cache_list_returned_objects_totalapiserver_flowcontrol_demand_seats_averageapiserver_flowcontrol_current_limit_seatsapiserver_request_sli_duration_seconds_bucket{le=+inf}apiserver_request_sli_duration_seconds_countapiserver_request_sli_duration_seconds_sumprocess_start_time_secondsapiserver_request_duration_seconds_bucket{le=+inf}apiserver_request_duration_seconds_countapiserver_request_duration_seconds_sumapiserver_storage_list_fetched_objects_totalapiserver_storage_list_returned_objects_totalapiserver_current_inflight_requests
Note
apiserver_request_duration_seconds and apiserver_request_sli_duration_seconds are histogram metrics which have high cardinality and all series are not collected by default. Only the sum and count are used for gathering the average latencies.
controlplane-cluster-autoscaler
rest_client_requests_totalcluster_autoscaler_last_activitycluster_autoscaler_cluster_safe_to_autoscalecluster_autoscaler_scale_down_in_cooldowncluster_autoscaler_scaled_up_nodes_totalcluster_autoscaler_unneeded_nodes_countcluster_autoscaler_unschedulable_pods_countcluster_autoscaler_nodes_countcloudprovider_azure_api_request_errorscloudprovider_azure_api_request_duration_seconds_bucketcloudprovider_azure_api_request_duration_seconds_count
controlplane-node-auto-provisioning
karpenter_pods_statekarpenter_nodes_created_totalkarpenter_nodes_terminated_totalkarpenter_nodeclaims_disrupted_totalkarpenter_voluntary_disruption_eligible_nodeskarpenter_voluntary_disruption_decisions_total
controlplane-kube-scheduler
scheduler_pending_podsscheduler_unschedulable_podsscheduler_pod_scheduling_attemptsscheduler_queue_incoming_pods_totalscheduler_preemption_attempts_totalscheduler_preemption_victimsscheduler_scheduling_attempt_duration_secondsscheduler_schedule_attempts_totalscheduler_pod_scheduling_duration_seconds
controlplane-kube-controller-manager
rest_client_request_duration_secondsrest_client_requests_totalworkqueue_depth
controlplane-etcd
etcd_server_has_leaderrest_client_requests_totaletcd_mvcc_db_total_size_in_bytesetcd_mvcc_db_total_size_in_use_in_bytesetcd_server_slow_read_indexes_totaletcd_server_slow_apply_totaletcd_network_client_grpc_sent_bytes_totaletcd_server_heartbeat_send_failures_total
acstor-capacity-provisioner (job=acstor-capacity-provisioner)
acstor-metrics-exporter (job=acstor-metrics-exporter)
Dashboards
The following default dashboards are automatically provisioned and configured by Azure Monitor managed service for Prometheus when you link your Azure Monitor workspace to an Azure Managed Grafana instance. They are provisioned in the specified Azure Grafana instance under the Managed Prometheus folder. These are the standard open source community dashboards for monitoring Kubernetes clusters with Prometheus and Grafana.
Kubernetes / Compute Resources / ClusterKubernetes / Compute Resources / Namespace (Pods)Kubernetes / Compute Resources / Node (Pods)Kubernetes / Compute Resources / PodKubernetes / Compute Resources / Namespace (Workloads)Kubernetes / Compute Resources / WorkloadKubernetes / KubeletNode Exporter / USE Method / NodeNode Exporter / NodesKubernetes / Compute Resources / Cluster (Windows)Kubernetes / Compute Resources / Namespace (Windows)Kubernetes / Compute Resources / Pod (Windows)Kubernetes / USE Method / Cluster (Windows)Kubernetes / USE Method / Node (Windows)
Recording rules
The following default recording rules are automatically configured by Azure Monitor managed service for Prometheus when you configure Prometheus metrics to be scraped from an Azure Kubernetes Service (AKS) cluster. Source code for these recording rules can be found in this GitHub repository. These are the standard open source recording rules used in the dashboards above.
cluster:node_cpu:ratio_rate5mnamespace_cpu:kube_pod_container_resource_requests:sumnamespace_cpu:kube_pod_container_resource_limits:sum:node_memory_MemAvailable_bytes:sumnamespace_memory:kube_pod_container_resource_requests:sumnamespace_memory:kube_pod_container_resource_limits:sumnamespace_workload_pod:kube_pod_owner:relabelnode_namespace_pod_container:container_cpu_usage_seconds_total:sum_iratecluster:namespace:pod_cpu:active:kube_pod_container_resource_requestscluster:namespace:pod_cpu:active:kube_pod_container_resource_limitscluster:namespace:pod_memory:active:kube_pod_container_resource_requestscluster:namespace:pod_memory:active:kube_pod_container_resource_limitsnode_namespace_pod_container:container_memory_working_set_bytesnode_namespace_pod_container:container_memory_rssnode_namespace_pod_container:container_memory_cachenode_namespace_pod_container:container_memory_swapinstance:node_cpu_utilisation:rate5minstance:node_load1_per_cpu:ratioinstance:node_memory_utilisation:ratioinstance:node_vmstat_pgmajfault:rate5minstance:node_network_receive_bytes_excluding_lo:rate5minstance:node_network_transmit_bytes_excluding_lo:rate5minstance:node_network_receive_drop_excluding_lo:rate5minstance:node_network_transmit_drop_excluding_lo:rate5minstance_device:node_disk_io_time_seconds:rate5minstance_device:node_disk_io_time_weighted_seconds:rate5minstance:node_num_cpu:sumnode:windows_node:sumnode:windows_node_num_cpu:sum:windows_node_cpu_utilisation:avg5mnode:windows_node_cpu_utilisation:avg5m:windows_node_memory_utilisation::windows_node_memory_MemFreeCached_bytes:sumnode:windows_node_memory_totalCached_bytes:sum:windows_node_memory_MemTotal_bytes:sumnode:windows_node_memory_bytes_available:sumnode:windows_node_memory_bytes_total:sumnode:windows_node_memory_utilisation:rationode:windows_node_memory_utilisation:node:windows_node_memory_swap_io_pages:irate:windows_node_disk_utilisation:avg_iratenode:windows_node_disk_utilisation:avg_iratenode:windows_node_filesystem_usage:node:windows_node_filesystem_avail::windows_node_net_utilisation:sum_iratenode:windows_node_net_utilisation:sum_irate:windows_node_net_saturation:sum_iratenode:windows_node_net_saturation:sum_iratewindows_pod_container_availablewindows_container_total_runtimewindows_container_memory_usagewindows_container_private_working_set_usagewindows_container_network_received_bytes_totalwindows_container_network_transmitted_bytes_totalkube_pod_windows_container_resource_memory_requestkube_pod_windows_container_resource_memory_limitkube_pod_windows_container_resource_cpu_cores_requestkube_pod_windows_container_resource_cpu_cores_limitnamespace_pod_container:windows_container_cpu_usage_seconds_total:sum_rate
Prometheus visualization recording rules
The following recording rules are automatically deployed to support Prometheus visualizations.
ux:cluster_pod_phase_count:sumux:node_cpu_usage:sum_irateux:node_memory_usage:sumux:controller_pod_phase_count:sumux:controller_container_count:sumux:controller_workingset_memory:sumux:controller_cpu_usage:sum_irateux:controller_rss_memory:sumux:controller_resource_limit:sumux:controller_container_restarts:maxux:pod_container_count:sumux:pod_cpu_usage:sum_irateux:pod_workingset_memory:sumux:pod_rss_memory:sumux:pod_resource_limit:sumux:pod_container_restarts:maxux:node_network_receive_drop_total:sum_irateux:node_network_transmit_drop_total:sum_irate
The following recording rules are required for Windows support. They're deployed automitcally but aren't enabled by default. See enabling and disabling rule groups to enable them.
ux:node_cpu_usage_windows:sum_irateux:node_memory_usage_windows:sumux:controller_cpu_usage_windows:sum_irateux:controller_workingset_memory_windows:sumux:pod_cpu_usage_windows:sum_irateux:pod_workingset_memory_windows:sum