Wojtek @suda Siudzinski

Python/Node/Golang/Rust developer, DIY hacker, rookie designer, 3D print junkie. CEO @ Gaia Charge

Moving Kubernetes Persistent Volumes within the same cloud account/cluster

During restructuring of my Digital Ocean Kubernetes cluster I needed to move some Persistent Volumes across namespace and some of them across the clusters.

One approach was to manually copy all the data using dvsync script but fortunately both clusters existed under the same Digital Ocean account which meant it should be possible to reuse already created DO volume.

Note: this approach should work as well with Amazon AWS, Google Cloud GKS and Azure AKS.

Here are the steps I took:

First make sure removing PV won't remove the underlying volume

By default, the Persistent Volumes have persistentVolumeReclaimPolicy set to Delete. This would automatically delete the underlying EBS/DO volume. We can change it to Retain by patching the PV:

$ kubectl patch pv ${PV_NAME} -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Then export both Persistent Volume and its Claim

$ kubectl get pv/${PV_NAME} --export -o yaml > ${PV_NAME}.yaml
$ kubectl get pvc/${PVC_NAME} --export -o yaml > ${PVC_NAME}.yaml

Create all necessary objects in the new cluster

If you're using Helm to manage deployments and create PVC's, you need to run it first and allow it to create the PVC on its own.
After this, delete the created PVC (you'll replace it with the exported one in the next step).

$ kubectl delete pvc/${PVC_NAME}

Note: this will also delete the new PV due to default persistentVolumeReclaimPolicy mentioned above.

Insert the exported PV and PVC

Kubernetes uses UID to determine connection between PVC and PV therefore you'll need to patch the PV with correct one:

$ kubectl apply -f ${PVC_NAME}.yaml
$ PVC_UID=$(kubectl get pvc/${PVC_NAME} -o jsonpath='{.metadata.uid}')
$ # If you're operating in the same cluster, you don't need to do the next step
$ kubectl apply -f ${PV_NAME}.yaml
$ kubectl patch pv ${PV_NAME} -p "{\"spec\":{\"claimRef\":{\"uid\":\"${PVC_UID}\"}}}"

Stop/remove the old deployment

You won't be able to use this volume in the new cluster as long it is attached to the old node, so you need to remove the old PV/PVC.

You're done!

Now the new deployment should mount the volume. Usually it takes a bit to reattach the EBS/DO volume, so don't get discouraged if you need to wait a minute or two.

comments powered by Disqus