Skip to main content

Resizing Persistent Volumes

Stephanie R.
Updated

Learn how to resize Persistent Volumes (PVs) on your OVHcloud Managed Kubernetes Service.

The Kubernetes PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, Kubernetes provides two API resources: PersistentVolume (PVs) and PersistentVolumeClaim (PVCs).

Since Kubernetes 1.11, support for expanding PersistentVolumeClaims (PVCs) is enabled by default, and in this tutorial, you will learn how to do it.

NOTE: Kubernetes PVCs resizing only allows expanding volumes, not decreasing them.

Volume resizing works with all storage classes, including LUKS-encrypted volumes (csi-cinder-high-speed-luks, csi-cinder-classic-luks, csi-cinder-high-speed-gen2-luks). The encryption is transparently maintained during the resize operation.

 

Before you begin

This tutorial presupposes that you already have a working OVHcloud Managed Kubernetes cluster and some basic knowledge of how to operate it. If you want to know more about those topics, please look at the deploying a Hello World application documentation.

You also need to know how PVs are handled on OVHcloud Managed Kubernetes service. Please refer to the Persistent Volumes on OVHcloud Managed Kubernetes guide.

NOTE: When a Persistent Volumes resource is created inside a Managed Kubernetes cluster, an associated Public Cloud Block Storage volume is automatically created with it. This volume is hourly charged and will appear in your Public Cloud project. For more information, please refer to our Volume Block Storage pricing page.

 

Instructions

Let’s make a Persistent Volume Claim

To test the PV resizing, we will need a PV associated with the cluster, i.e., we need to deploy a service making a PVC. To keep things simple, we choose to deploy a single instance of MySQL.

Let’s begin by creating a mysql-pvc.yaml to define an initial PVC with 2 GB of allocated space:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

And apply it to the cluster:

# kubectl apply -f mysql/mysql-pvc.yaml

We can then verify that the PVC is correctly created and bound to a PV:

# kubectl describe pvc mysql-pv-claim

Now we create a mysql-deployment.yaml file to deploy MySQL using that PVC:

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

And we deploy and verify it:

# kubectl apply -f mysql-deployment.yaml
# kubectl describe deployment mysql

In my example cluster, the preceding commands obtain:

# kubectl apply -f mysql/mysql-pvc.yaml
persistentvolumeclaim/mysql-pv-claim created

# kubectl describe pvc mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     default
StorageClass:  csi-cinder-high-speed
Status:        Bound
Volume:        ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    mysql-c85f7f79c-wz4w7
Events:
  Type    Reason                 Age                From                                                                                         Message
  ----    ------                 ----               ----                                                                                         -------
  Normal  ExternalProvisioning   72s (x2 over 72s)  persistentvolume-controller                                                                  waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
  Normal  Provisioning           72s                cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b  External provisioner is provisioning volume for claim "default/mysql-pv-claim"
  Normal  ProvisioningSucceeded  70s                cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b  Successfully provisioned volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d

# kubectl apply -f mysql/mysql-deployment.yaml
service/mysql created
deployment.apps/mysql created

# kubectl describe deployment mysql
Name:               mysql
Namespace:          default
CreationTimestamp:  Mon, 06 Jan 2020 11:45:03 +0100
Labels:             <none>
Annotations:        deployment.kubernetes.io/revision: 1
                    kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"mysql","namespace":"default"},"spec":{"replicas":1,"selec...
Selector:           app=mysql
Replicas:           1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:  app=mysql
  Containers:
   mysql:
    Image:      mysql:5.6
    Port:       3306/TCP
    Host Port:  0/TCP
    Environment:
      MYSQL_ROOT_PASSWORD:  password
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
  Volumes:
   mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   mysql-c85f7f79c (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  27s   deployment-controller  Scaled up replica set mysql-c85f7f79c to 1

 

Accessing the MySQL instance and initializing a database

The preceding YAML file creates a service that allows other Pods in the cluster to access the database. The Service option clusterIP: None lets the Service DNS name resolve directly to the Pod’s IP address. This is optimal when you have only one Pod behind a Service, and you don’t intend to increase the number of Pods.

Now we are going to use kubectl to create a mysql client on the fly to connect to the database:

# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword

And then simply create a database with a table:

CREATE DATABASE testingResize;
USE testingResize;
CREATE TABLE anEmptyTable (k VARCHAR(256), v TEXT);
SHOW TABLES;

On my example cluster:

# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
If you don't see a command prompt, try pressing enter.

mysql> CREATE DATABASE testingResize;
Query OK, 1 row affected (0.01 sec)

mysql> USE testingResize;
Database changed

mysql> CREATE TABLE anEmptyTable (k VARCHAR(256), v TEXT);
Query OK, 0 rows affected (0.04 sec)

mysql> SHOW TABLES;
+-------------------------+
| Tables_in_testingResize |
+-------------------------+
| anEmptyTable            |
+-------------------------+
1 row in set (0.00 sec)

 

Expand the PVs

To expand the persistent volume, the first step is to unbind the PVC from the deployments using it. To do that, we set the deployment’s replicas to 0:

NOTE: Do not forget to downscale your deployment before resizing your volume

# kubectl patch deployment mysql -p '{ "spec": { "replicas": 0 }}'

Then we path the PVC definition to expand the volume to 6 GB:

# kubectl patch pvc mysql-pv-claim -p '{ "spec": { "resources": { "requests": { "storage": "6Gi" }}}}'

NOTE: Kubernetes PVCs resizing only allows to expand volumes, not to decrease them. If you try to decrease the storage size, you will get a message like

The PersistentVolumeClaim "mysql-pv-claim" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous

We verify that the volume has been expanded:

# kubectl describe pvc mysql-pv-claim

In the “conditions” field shown by the output of the previous command line, we can see that the PVC is waiting for the user to start a pod to finish the file system resize of the volume. Let’s put replicas back to 1 on mysql-deployment.yaml, and deploy it again to start a pod:

# kubectl patch deployment mysql -p '{ "spec": { "replicas": 1 }}'

After the pod starts, we can use again kubectl describe pvc mysql-pv-claim and we see that the PV size is 6 GB.

On my example cluster:

# kubectl patch deployment mysql -p '{ "spec": { "replicas": 0 }}'
deployment.extensions/mysql patched

# kubectl get deployment mysql
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
mysql   0/0     0            0           4h0m

# kubectl patch pvc mysql-pv-claim -p '{ "spec": { "resources": { "requests": { "storage": "6Gi" }}}}'
persistentvolumeclaim/mysql-pv-claim patched

# kubectl describe pvc mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     default
StorageClass:  csi-cinder-high-speed
Status:        Bound
Volume:        ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    <none>
Conditions:
  Type                      Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----                      ------  -----------------                 ------------------                ------  -------
  FileSystemResizePending   True    Mon, 01 Jan 0001 00:00:00 +0000   Mon, 06 Jan 2020 11:47:22 +0100           Waiting for user to (re-)start a pod to finish file system resize of volume on node.
Events:
  Type     Reason                 Age                    From                                                                                         Message
  ----     ------                 ----                   ----                                                                                         -------
  Normal   ExternalProvisioning   2m27s (x2 over 2m27s)  persistentvolume-controller                                                                  waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
  Normal   Provisioning           2m27s                  cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b  External provisioner is provisioning volume for claim "default/mysql-pv-claim"
  Normal   ProvisioningSucceeded  2m25s                  cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b  Successfully provisioned volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
  Normal   Resizing                  9s (x9 over 13s)  external-resizer cinder.csi.openstack.org  External resizer is resizing volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
  Normal   FileSystemResizeRequired  8s                external-resizer cinder.csi.openstack.org  Require file system resize of volume on node

# kubectl patch deployment mysql -p '{ "spec": { "replicas": 1 }}'
deployment.extensions/mysql patched

# kubectl get deployment mysql
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
mysql   0/1     1            0           4h2m

# kubectl get deployment mysql
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
mysql   1/1     1            1           4h4m

# kubectl describe pvc mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     default
StorageClass:  csi-cinder-high-speed
Status:        Bound
Volume:        ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      6Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    mysql-c85f7f79c-9nn8q
Events:
  Type     Reason                 Age                    From                                                                                         Message
  ----     ------                 ----                   ----                                                                                         -------
  Normal   ExternalProvisioning   8m8s (x2 over 8m8s)    persistentvolume-controller                                                                  waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
  Normal   Provisioning           8m8s                   cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b  External provisioner is provisioning volume for claim "default/mysql-pv-claim"
  Normal   ProvisioningSucceeded  8m6s                   cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b  Successfully provisioned volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
  Normal   Resizing                  5m50s (x9 over 5m54s)  external-resizer cinder.csi.openstack.org  External resizer is resizing volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
  Normal   FileSystemResizeRequired  5m49s                  external-resizer cinder.csi.openstack.org  Require file system resize of volume on node

 

Verifying data integrity

So we launch again a MySQL client to verify that we can still read our database:

# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword

A SHOW DATABASES; should allow us to see our testingResize database, we can select it and find our anEmptyTable table.

On my example cluster:

# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
If you don't see a command prompt, try pressing enter.

mysql> SHOW DATABASES;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| #mysql50#lost+found |
| mysql               |
| performance_schema  |
| testingResize       |
+---------------------+
5 rows in set (0.01 sec)

mysql> USE testingResize;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SHOW TABLES;
+-------------------------+
| Tables_in_testingResize |
+-------------------------+
| anEmptyTable            |
+-------------------------+
1 row in set (0.00 sec)

 

Go further

For more information and tutorials, please see our other Managed Kubernetes or Platform as a Service guides. You can also explore the guides for other OVHcloud products and services.

Related articles