k8s MultiAzStorage Problem

As of 2023-09 there seem to be a fundamental problem with k8s and statefull workloads.

Problem 1 - POD and PV(Storage) not in same AZ

For deployments this can be avoided by only creating the PV from the PVC once the pod is scheduled with Storage Class config
```
volumeBindingMode: WaitForFirstConsumer
```
- Storage wait for POD to start and then get's created in same AZ
The real issue is when there is a server failure, or other re-schedule of pods, and they land on a different AZ than there PV(Permanent volume/Disk)
- Example error in k8s event log
```
Warning   FailedScheduling   pod/infra-elasticsearch-master-0   0/4 nodes are available: 1 Insufficient cpu, 3 node(s) had volume node affinity conflict. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling..
```
- If you can loose the pvc data, then just scale to 0 and delete the pvc, scale backup up and get new volume in correct AZ
- If you can't loose data you have to move the pod to the correct az (Impossible for Statefullset) or move the data, backup, create new PV and restore.