= k8s/StudyNotes/ClusterMaintenance&Upgrades =

 * k8s version  v1(Major).11(minor).2(patch)  , v1.0-2015, up to 1.22.2-2021 , some components external versions e.g. etcd

 * Node down 5min, pods terminated, --pod-eviction-timeout=5m0 on kube-controller-manager
   * If node comes back it will be blank.
   * Risky to upgrade node, and hope it reboots in under 5min
     * Rather drain pods to other nodes {{{
         kubectl drain node-1
         # kubectl codon node-1   << does not drain
         ## reboot / upg after drain
         kubectl uncordon node-1
         }}}

 * Upgrades
   * Components can be at different versions, kube-apiserver should be highest x, rest can be x-1, kubelet/kube-proxy x-2
     * kubectl (x+1), x, (x-1) to facilitate upgrades.
     * 3 latest versions supported e.g. 1.22, 1.21, 1.20
     * If master nodes goes down, nodes still run, but no admin or restarts.
   * Tools 
     * Cloud provider ?
     * {{{ kubeadm upgrade plan / apply }}}
     * "The hard way" manual
   * Upg Steps
     1. Start v1.10, get kubeadm for next version e.g.v1.11, using {{{ apt-get upgrade -y kubeadm=1.11.0-00 }}}
     2. Upg Master v1.10 -> v1.11 (No mgmt, but nodes and pods keep going during upg.)
         1. kubeadm upgrade apply v1.11.0
         2. kubectl get nodes, still show nodes at older v1.10, including master
            * So we have to now upgrade kubelet on master node {{{ apt-get upgrade -y kubelet=1.12.0-00 }}}
              * restart {{{ systemctl restart kubelet }}}
     3a. Move pods off single node, and upgrade v1.10 -> v1.11, then re-join.
         * {{{ kubectl drain node-1 }}}
         * {{{ 
              #ssh to node-1
              apt-get upgrade -y kubeadm=1.11.0-00 kubelet=1.11.0-00
              kubeadm upgrade node config --kubelet-version v1.11.0
              systemctl restart kubelet
           }}}
         * {{{ kubectl uncordon node-1 }}}
     3b. In cloud create new node, add to cluster, and move pods off old nodes.  Remove old nodes.


 * etc - backups
   * set api_v3 with {{{ 
export ETCDCTL_API=3
etcdctl --version
etcdctl snapshot ...
}}}
   * full etcdctl snapshot {{{
 ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/snapshot-pre-boot.db
}}}