• k8s
• StudyNotes
• ClusterMaintenance&Upgrades

k8s/StudyNotes/ClusterMaintenance&Upgrades

• k8s version v1(Major).11(minor).2(patch) , v1.0-2015, up to 1.22.2-2021 , some components external versions e.g. etcd
• Node down 5min, pods terminated, --pod-eviction-timeout=5m0 on kube-controller-manager
  • If node comes back it will be blank.
  • Risky to upgrade node, and hope it reboots in under 5min

    • Rather drain pods to other nodes

               kubectl drain node-1
               # kubectl codon node-1   << does not drain
               ## reboot / upg after drain
               kubectl uncordon node-1

• Upgrades
  • Components can be at different versions, kube-apiserver should be highest x, rest can be x-1, kubelet/kube-proxy x-2
    • kubectl (x+1), x, (x-1) to facilitate upgrades.
    • 3 latest versions supported e.g. 1.22, 1.21, 1.20
    • If master nodes goes down, nodes still run, but no admin or restarts.
  • Tools
    • Cloud provider ?

    •  kubeadm upgrade plan / apply 

    • "The hard way" manual
  • Upg Steps

    1. Start v1.10, get kubeadm for next version e.g.v1.11, using  apt-get upgrade -y kubeadm=1.11.0-00 

    2. Upg Master v1.10 -> v1.11 (No mgmt, but nodes and pods keep going during upg.)

       1. kubeadm upgrade apply v1.11.0
       2. kubectl get nodes, still show nodes at older v1.10, including master

          • So we have to now upgrade kubelet on master node  apt-get upgrade -y kubelet=1.12.0-00 

            • restart  systemctl restart kubelet 

       3a. Move pods off single node, and upgrade v1.10 -> v1.11, then re-join.

       •  kubectl drain node-1 

       •               #ssh to node-1
                       apt-get upgrade -y kubeadm=1.11.0-00 kubelet=1.11.0-00
                       kubeadm upgrade node config --kubelet-version v1.11.0
                       systemctl restart kubelet

       •  kubectl uncordon node-1 

       3b. In cloud create new node, add to cluster, and move pods off old nodes. Remove old nodes.
• etc - backups

  • set api_v3 with

    export ETCDCTL_API=3
    etcdctl --version
    etcdctl snapshot ...

  • full etcdctl snapshot

     ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    snapshot save /opt/snapshot-pre-boot.db