Moving Kubernetes Persistent Volumes within the same cloud account/cluster
During restructuring of my Digital Ocean Kubernetes cluster I needed to move some Persistent Volumes across namespace and some of them across the clusters.
One approach was to manually copy all the data using dvsync
script but fortunately both clusters existed under the same Digital Ocean account which meant it should be possible to reuse already created DO volume.
Note: this approach should work as well with Amazon AWS, Google Cloud GKS and Azure AKS.
Here are the steps I took:
First make sure removing PV won't remove the underlying volume
By default, the Persistent Volumes have persistentVolumeReclaimPolicy
set to Delete
. This would automatically delete the underlying EBS/DO volume. We can change it to Retain
by patching the PV:
$ kubectl patch pv ${PV_NAME} -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
Then export both Persistent Volume and its Claim
$ kubectl get pv/${PV_NAME} --export -o yaml > ${PV_NAME}.yaml
$ kubectl get pvc/${PVC_NAME} --export -o yaml > ${PVC_NAME}.yaml
Create all necessary objects in the new cluster
If you're using Helm to manage deployments and create PVC's, you need to run it first and allow it to create the PVC on its own.
After this, delete the created PVC (you'll replace it with the exported one in the next step).
$ kubectl delete pvc/${PVC_NAME}
Note: this will also delete the new PV due to default persistentVolumeReclaimPolicy
mentioned above.
Insert the exported PV and PVC
Kubernetes uses UID
to determine connection between PVC and PV therefore you'll need to patch the PV with correct one:
$ kubectl apply -f ${PVC_NAME}.yaml
$ PVC_UID=$(kubectl get pvc/${PVC_NAME} -o jsonpath='{.metadata.uid}')
$ # If you're operating in the same cluster, you don't need to do the next step
$ kubectl apply -f ${PV_NAME}.yaml
$ kubectl patch pv ${PV_NAME} -p "{\"spec\":{\"claimRef\":{\"uid\":\"${PVC_UID}\"}}}"
Stop/remove the old deployment
You won't be able to use this volume in the new cluster as long it is attached to the old node, so you need to remove the old PV/PVC.
You're done!
Now the new deployment should mount the volume. Usually it takes a bit to reattach the EBS/DO volume, so don't get discouraged if you need to wait a minute or two.