k8s/AutoScaler

k8s - builtin GA autoscaler - https://github.com/kubernetes/autoscaler/tree/master

View/Debug

History and status

 kubectl get configmaps -n kube-system cluster-autoscaler-status -o yaml

apiVersion: v1
data:
  status: |+
    Cluster-autoscaler status at 2024-03-20 01:10:58.844440912 +0000 UTC:
    Cluster-wide:
      Health:      Healthy (ready=5 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=5 longUnregistered=0)
                   LastProbeTime:      2024-03-20 01:10:58.66229529 +0000 UTC m=+412459.398307389
                   LastTransitionTime: 2024-03-15 06:36:53.20437346 +0000 UTC m=+13.940385459
      ScaleUp:     NoActivity (ready=5 registered=5)
                   LastProbeTime:      2024-03-20 01:10:58.66229529 +0000 UTC m=+412459.398307389
                   LastTransitionTime: 2024-03-15 06:36:53.20437346 +0000 UTC m=+13.940385459
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2024-03-20 01:10:58.66229529 +0000 UTC m=+412459.398307389
                   LastTransitionTime: 2024-03-15 06:36:53.20437346 +0000 UTC m=+13.940385459

Config / Priority Scaling

The k8s cluster must be deployed/configured with expander=priority. e.g. terraform snippet

resource "azurerm_kubernetes_cluster" "k8s" {
  name       = ...
  ...
  ...
 auto_scaler_profile {
    # https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile
    balance_similar_node_groups   = false # Pick Zone's 2022 only on scale up
    expander                      = "priority"
    max_graceful_termination_sec  = 60    # default 600 seconds
    max_node_provisioning_time    = "10m" # default 15m
    skip_nodes_with_local_storage = false
    skip_nodes_with_system_pods   = false
    #lots more settings see https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#private_cluster_public_fqdn_enabled
  }
  ...
}

Deploy configmap into cluster

#Config map used with AKS priority scaling - autoscaler - expander - priority.
# regex match .* = anyting NodePool name. from high to low.
# https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/expander/priority
# Status:  kubectl get configmaps -n kube-system cluster-autoscaler-status -o yaml
# See: Terrafrom - tm-infra/terraform/modules/aks-k8s-cluster/main-k8s.tf for priority.
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  # Regex to match VMSS scaleset
  priorities: |-
        70:
          - "aks-spote4sv3-.*"
        60:
          - "aks-spotd4sv3-.*"
        40:
          - "aks-spotd8sv3-.*"
          - "aks-spote8sv3-.*"
        30:
          - ".*spot.*"
        10:
          - ".*"