Backup and restore OVHcloud Managed Kubernetes cluster, namespace, and applications using TrilioVault for Kubernetes

Please follow the steps below to install TrilioVault via Helm:

First, clone the OVHcloud Docs Git repository and change the directory to your local copy:

git clone https://github.com/ovh/docs.git
cd docs/pages/platform/kubernetes-k8s/backup-and-restore-cluster-namespace-and-applications-with-trilio/

Next, add the TrilioVault Helm repository, and list the available charts:

helm repo add triliovault-operator http://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator
helm repo update
helm search repo triliovault-operator

The output should look similar to the following:

NAME                                            CHART VERSION   APP VERSION     DESCRIPTION
triliovault-operator/k8s-triliovault-operator   2.9.3           2.9.3           K8s-TrilioVault-Operator is an operator designe...

The chart of interest is triliovault-operator/k8s-triliovault-operator, which will install TrilioVault for Kubernetes Operator on the cluster. You can run the helm install command to install the Operator, which will also install the TrilioVault Manager CRD. Install the TrilioVault for Kubernetes Operator using Helm:

TVK allows the user to alter the values to be used by the TVK Operator installation using the --set option. Check the detailed instructions in the One-click Installation page.

helm install triliovault-operator triliovault-operator/k8s-triliovault-operator --namespace tvk --create-namespace

Now, please check your TVK deployment:

helm ls -n tvk

The output should look similar to the following (STATUS column should display “deployed”):

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
triliovault-manager-tvk tvk             1               2022-06-21 07:15:03.681891176 +0000 UTC deployed        k8s-triliovault-2.9.3           2.9.3
triliovault-operator    tvk             1               2022-06-21 07:13:18.731129339 +0000 UTC deployed        k8s-triliovault-operator-2.9.3  2.9.3

Next, verify that the TrilioVault-Operator and TrilioVault-Manager applications are up and running:

kubectl get deployments -n tvk

The output should look similar to the following (deployment pods must be in the Ready state):

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
k8s-triliovault-admission-webhook               1/1     1            1           45d
k8s-triliovault-control-plane                   1/1     1            1           45d
k8s-triliovault-exporter                        1/1     1            1           45d
k8s-triliovault-ingress-nginx-controller        1/1     1            1           13d
k8s-triliovault-web                             1/1     1            1           45d
k8s-triliovault-web-backend                     1/1     1            1           45d
triliovault-operator-k8s-triliovault-operator   1/1     1            1           45d

Now, please check your triliovaultmanagers CRDs and the tvm CR as well:

kubectl get crd | grep trilio

The output should look similar to the following:

backupplans.triliovault.trilio.io                     2022-06-21T07:39:38Z
backups.triliovault.trilio.io                         2022-06-21T07:39:38Z
clusterbackupplans.triliovault.trilio.io              2022-06-21T07:39:39Z
clusterbackups.triliovault.trilio.io                  2022-06-21T07:39:39Z
clusterrestores.triliovault.trilio.io                 2022-06-21T07:39:39Z
hooks.triliovault.trilio.io                           2022-06-21T07:39:39Z
licenses.triliovault.trilio.io                        2022-06-21T07:39:39Z
policies.triliovault.trilio.io                        2022-06-21T07:39:40Z
restores.triliovault.trilio.io                        2022-06-21T07:39:40Z
targets.triliovault.trilio.io                         2022-06-21T07:39:40Z
triliovaultmanagers.triliovault.trilio.io             2022-06-21T07:38:30Z

You can also check if the TVM Custom Resource is created.

kubectl get triliovaultmanagers -n tvk

The output looks similar to the following:

NAME                  TRILIOVAULT-VERSION   SCOPE     STATUS     RESTORE-NAMESPACES
triliovault-manager   2.9.3                 Cluster   Deployed

If the output looks like the one above, you have installed TVK successfully. Next, you will learn how to check the license type and validity, as well as how to renew a license.

By default, when installing TVK via Helm, there is no Free Trial license generated. This tutorial will help you install the ‘Cluster’ scoped license, which is of the type ‘Basic’ for a cluster capacity of 500 CPUs and an expiration time of five years.

You can always go to the Trilio website and generate a new license for your cluster that suits your needs.

Please run the command below to see what license is available for your cluster (it is managed via the License CRD):

curl -LO https://raw.githubusercontent.com/ovh/docs/develop/pages/platform/kubernetes-k8s/backup-and-restore-cluster-namespace-and-applications-with-trilio/manifests/tvk_install_license.yaml
kubectl apply -f tvk_install_license.yaml -n tvk

Run the below command to verify if the license is successfully created for OVHcloud users:

kubectl get license -n tvk

The output looks similar to the following (notice the STATUS, which should be “Active”, as well as the license type in the EDITION column and EXPIRATION TIME):

NAMESPACE   NAME             STATUS   MESSAGE                                   CURRENT NODE COUNT   GRACE PERIOD END TIME   EDITION   CAPACITY   EXPIRATION TIME        MAX NODES
tvk         trilio-license   Active   Cluster License Activated successfully.   3                                            Basic     500        2027-06-21T00:00:00Z   3

The license is managed via a special CRD, namely the License object. You can inspect it by running the following command:

kubectl describe license test-license-1 -n tvk

The output looks similar to the following (notice the Message and Capacity fields, as well as the Edition):

Name:         test-license-1
Namespace:    tvk
Labels:       <none>
Annotations:  generation: 1
              triliovault.trilio.io/creator: kubernetes-admin
              triliovault.trilio.io/instance-id: 46188ee1-8ce1-4c45-96fa-c262f2214ced
              triliovault.trilio.io/updater:
                [{"username":"system:serviceaccount:tvk:k8s-triliovault","lastUpdatedTimestamp":"2022-06-21T10:06:59.796280418Z"}]
API Version:  triliovault.trilio.io/v1
Kind:         License
Metadata:
  Creation Timestamp:  2022-06-21T10:56:14Z
...
  Current Node Count:  3
  Max Nodes:           3
  Message:             Cluster License Activated successfully.
  Properties:
    Active:                        true
    Capacity:                      500
    Company:                       OVHCloud License For Users
    Creation Timestamp:            2022-06-21T00:00:00Z
    Edition:                       Basic
    Expiration Timestamp:          2027-06-21T00:00:00Z
    Kube UID:                      46188ee1-8ce1-4c45-96fa-c262f2214ced
    License ID:                    TVAULT-4ddf3f72-d2ab-11ec-9a22-4b4849af53ee
    Maintenance Expiry Timestamp:  2027-06-21T00:00:00Z
    Number Of Users:               -1
    Purchase Timestamp:            2022-06-21T00:00:00Z
    Scope:                         Cluster
...

The above output will also tell you when the license is going to expire in the Expiration Timestamp field, and the Scope (Cluster based in this case). You can opt for a cluster-wide license type or for a namespace based license. More details can be found on the Trilio Licensing documentation page.

To renew the license, you will have to request a new one from the Trilio website by navigating to the licensing page. After completing the form, you should receive the License YAML manifest, which can be applied to your cluster using kubectl. The following commands assume that TVK is installed in the default tvk namespace (please replace the <> placeholders accordingly, where required):

kubectl apply -f <YOUR_LICENSE_FILE_NAME>.yaml -n tvk

You can check the new license status as you already learned via:

# List available TVK licenses first from the `tvk` namespace
kubectl get license -n tvk

# Get information about a specific license from the `tvk` namespace
kubectl describe license <YOUR_LICENSE_NAME_HERE> -n tvk

In the next step, you will learn how to define the storage backend for TrilioVault to store backups, called a target.

To access the console and explore its features, you can use either the Load Balancer or the Node Port, or you can port forward the ingress-nginx-controller service for TVK.

First, you need to identify the ingress-nginx-controller service from the tvk namespace:

kubectl get svc -n tvk

The output looks similar to the following (search for the k8s-triliovault-ingress-nginx-controller line, and notice that it listens on port 80 in the PORT(S) column):

NAME                                                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
k8s-triliovault-admission-webhook                               ClusterIP   10.3.241.124   <none>        443/TCP                      45d
k8s-triliovault-ingress-nginx-controller                        NodePort    10.3.183.125   <none>        80:31879/TCP,443:31921/TCP   13d
k8s-triliovault-ingress-nginx-controller-admission              ClusterIP   10.3.20.89     <none>        443/TCP                      13d
k8s-triliovault-web                                             ClusterIP   10.3.56.86     <none>        80/TCP                       45d
k8s-triliovault-web-backend                                     ClusterIP   10.3.236.30    <none>        80/TCP                       45d
triliovault-operator-k8s-triliovault-operator-webhook-service   ClusterIP   10.3.8.249     <none>        443/TCP                      45d

If you are using the Load Balancer for the ingress-nginx-controller, then the output would look like:

NAME                                                            TYPE            CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
k8s-triliovault-admission-webhook                               ClusterIP       10.3.241.124   <none>           443/TCP                      45d
k8s-triliovault-ingress-nginx-controller                        LoadBalancer    10.3.183.125   51.222.45.171    80:31879/TCP,443:31921/TCP   13d
k8s-triliovault-ingress-nginx-controller-admission              ClusterIP       10.3.20.89     <none>           443/TCP                      13d
k8s-triliovault-web                                             ClusterIP       10.3.56.86     <none>           80/TCP                       45d
k8s-triliovault-web-backend                                     ClusterIP       10.3.236.30    <none>           80/TCP                       45d
triliovault-operator-k8s-triliovault-operator-webhook-service   ClusterIP       10.3.8.249     <none>           443/TCP                      45d

TVK is using an Nginx Ingress Controller to route traffic to the management web console services. Routing is host based, and the host name is ovh-k8s-tvk.demo.trilio.io as defined in the Helm values file from the ovh/docs:

# The host name to use when accessing the web console via the TVK ingress controller service
ingressConfig:
  host: "ovh-k8s-tvk.demo.trilio.io"

Having the above information on hand, please go ahead and edit the/etc/hosts file, and add this entry:

127.0.0.1 ovh-k8s-tvk.demo.trilio.io

Next, create the port forward for the TVK ingress controller service:

kubectl port-forward svc/k8s-triliovault-ingress-nginx-controller 8080:80 -n tvk

Finally, download the kubeconfig file for your OVHcloud Managed Kubernetes Cluster present under the Service tab as Kubeconfig file. This step is required so that the web console can authenticate you using the kubeconfig file:

After following the above steps, you can access the console in your web browser by navigating to: http://ovh-k8s-tvk.demo.trilio.io. When asked for the kubeconfig file, please select the one that you created in the last command from above.

The home page looks similar to:

Go ahead and explore each section from the left:

• Cluster Management: This shows the list, including the primary cluster and other clusters having TVK instances, added to the primary OVHcloud cluster using the Multi-Cluster Management feature.
• Backup & Recovery: This is the main dashboard, which gives you a general overview of the whole cluster (e.g., Discovered namespaces, Applications, Backup plans list, Targets, Hooks, Policies, etc.)
  • Namespaces:
  • Applications:
  • backup plans:
  • Targets:
  • Scheduling Policy:
  • Retention Policy:
• Monitoring: This has two options: TrilioVault Monitoring and Velero Monitoring, if the user has Velero configured on their OVHcloud cluster.
  • TrilioVault Monitoring: It shows the backup and restore summary of the Kubernetes cluster.
  • Velero Monitoring:
• Disaster Recovery: Allows you to manage and perform disaster recovery operations.

You can also see the Object Storage Target created earlier, by navigating to Backup & Recovery > Targets > Select the TVK Namespace from the dropdown on the top (in case of ovh/docs the TVK Namespace is tvk):

Going further, you can browse the target and list the available backups by clicking the Actions button on the right. Then, select the Launch Browser option from the pop-up menu (for this to work, the target must have the enableBrowsing flag set to true):

For more information and available features, please consult the TVK Web Management Console User Interface official documentation.

Next, you will learn how to perform backup and restore operations for specific use cases, like:

• Specific namespace(s) backup and restore.
• Whole cluster backup and restore.

helm repo add stable https://charts.helm.sh/stable
helm repo update
helm install mysql-qa --set mysqlRootPassword=triliopass stable/mysql -n demo-backup-ns

To verify if the Helm release is deployed correctly, run the following command:

helm ls -n demo-backup-ns

The output should look similar to this:

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
mysql-qa        demo-backup-ns  1               2022-06-21 08:23:01.849247691 +0000 UTC deployed        mysql-1.6.9     5.7.30

Next, verify that the mysql-qa deployment is up and running:

kubectl get deployments -n demo-backup-ns

The output should look similar to this:

NAME       READY   UP-TO-DATE   AVAILABLE   AGE
mysql-qa   1/1     1            1           2m5s

This shows that the mysql-qa Helm release is ready to be backed up.

To perform backups for a single application at the namespace level (or Helm release), a Backup Plan followed by a Backup CRD is required. A Backup Plan allows you to:

• Specify a target where backups should be stored.
• Define a set of resources to back up (e.g., namespace or Helm releases).
• Set up encryption if you want to encrypt your backups on the target (this is a very nice feature for securing your backups' data).
• Define schedules for full or incremental type backups.
• Define retention policies for your backups.

In other words, a Backup Plan is a definition of the “what,” “where,” “to," and “how” of the backup process, but it doesn’t perform the actual backup. The Backup CRD is responsible for triggering the actual backup process, as dictated by the Backup Plan spec.

Typical Backup Plan CRD looks like the example below:

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: mysql-qa-helm-release-backup-plan
  namespace: demo-backup-ns
spec:
  backupConfig:
    target:
      name: trilio-ovh-s3-target
      namespace: tvk
  backupPlanComponents:
    helmReleases:
      - mysql-qa

Explanation for the above configuration:

• spec.backupConfig.target.name: Tells TVK what target name to use for storing backups.
• spec.backupConfig.target.namespace: Tells TVK in which namespace the target was created.
• spec.backupComponents: Defines a list of resources to back up (can be namespaces or Helm releases).

Typical Backup CRD looks like the example below:

apiVersion: triliovault.trilio.io/v1
kind: Backup
metadata:
  name: mysql-qa-helm-release-full-backup
  namespace: demo-backup-ns
spec:
  type: Full
  backupPlan:
    name: mysql-qa-helm-release-backup-plan
    namespace: demo-backup-ns

Explanation for the above configuration:

• spec.type: Specifies the backup type (e.g., Full or Incremental).
• spec.backupPlan: Specifies the BackupPlan which this Backup should use.

Steps to initiate the mysql-qa Helm release one-time backup:

1. First, make sure that the mysql-qa is deployed in your cluster by following these steps.

2. Next, change the directory where the docs Git repository was cloned on your local machine:

   cd docs/pages/platform/kubernetes-k8s/backup-and-restore-cluster-namespace-and-applications-with-trilio/

   Copy

3. Then, open and inspect the mysql-qa Helm release Backup Plan and Backup manifest files provided in the pages/platform/kubernetes-k8s/backup-and-restore-cluster-namespace-and-applications-with-trilio/guide.en-us.md repository, using an editor of your choice (preferably with YAML lint support). You can use VS Code, for example:

   cat manifests/mysql-qa-helm-release-backup-plan.yaml
   cat manifests/mysql-qa-helm-release-backup.yaml

   Copy

4. Create the Backup Plan resource using kubectl:

   kubectl apply -f manifests/mysql-qa-helm-release-backup-plan.yaml -n demo-backup-ns

   Copy

   Now, inspect the Backup Plan status (targeting the mysql-qa Helm release) using kubectl:

   kubectl get backupplan mysql-qa-helm-release-backup-plan -n demo-backup-ns

   Copy

   The output should look similar to the following (notice the STATUS column value, which should be set to “Available”):

   NAME                                  TARGET            ...   STATUS
   mysql-qa-helm-release-backup-plan   trilio-ovh-s3-target    ...   Available

   Copy

5. Finally, create a Backup resource using kubectl:

   kubectl apply -f manifests/mysql-qa-helm-release-backup.yaml -n demo-backup-ns

   Copy

   Now, inspect the Backup status (targeting the mysql-qa Helm release) using kubectl:

   kubect get backup mysql-qa-helm-release-full-backup -n demo-backup-ns

   Copy

   Next, check the Backup object status using kubectl:

   kubectl get backup mysql-qa-helm-release-full-backup -n demo-backup-ns

   Copy

   The output should look similar to the following (notice the STATUS column value, which should be set to “InProgress”, as well as the BACKUP TYPE set to “Full”):

   NAME                                BACKUPPLAN                          BACKUP TYPE   STATUS       ...
   mysql-qa-helm-release-full-backup   mysql-qa-helm-release-backup-plan   Full          InProgress   ...                                  

   Copy

After all the mysql-qa Helm release components finish uploading to the Object Storage target, you should get the following results:

kubectl get backup mysql-qa-helm-release-full-backup -n demo-backup-ns

The output should look similar to the following (notice that the STATUS changed to “Available”, and PERCENTAGE is “100”)

NAME                                BACKUPPLAN                          BACKUP TYPE   STATUS      ...   PERCENTAGE
mysql-qa-helm-release-full-backup   mysql-qa-helm-release-backup-plan   Full          Available   ...   100

If the output looks like this, you have successfully backed up the mysql-qa Helm release. You can go ahead and see how TrilioVault stores Kubernetes metadata by listing the TrilioVault Object Storage Bucket contents.

Finally, you can check that the backup is available in the web console as well by navigating to Backup & Recovery -> Backup Plans and selecting the demo-ns-backup Namespace from the top drop-down menu (notice that it’s in the “Available” state, and the mysql-qa Helm release was backed up in the “Component Details” sub-view).

Now, go ahead and simulate a disaster by intentionally deleting the mysql-qa Helm release:

helm delete mysql-qa -n demo-backup-ns

Next, check that the namespace resources were deleted (listing should be empty):

kubectl get all -n demo-backup-ns

Important notes:

• If restoring into the same namespace, ensure that the original application components have been removed. Especially ensure that the PVC of the application is deleted.
• If restoring to another cluster (migration scenario), ensure that TrilioVault for Kubernetes is running in the remote namespace/cluster as well. To restore into a new cluster (where the Backup CR does not exist), source.type must be set to location. Please refer to the Custom Resource Definition Restore Section to view a restore by location example.
• When you delete the demo-backup-ns namespace, the Load Balancer resource associated with the mysql-qa service will be deleted as well. So, when you restore the mysq-qa service, the Load Balancer will be recreated by OVHcloud. The issue is that you will get a New IP address for your Load Balancer, so you will need to adjust the A records to route traffic into the domains hosted on the cluster.

To restore a specific Backup, you need to create a Restore CRD. A typical Restore CRD looks like this:

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: mysql-qa-helm-release-restore
  namespace: demo-restore-ns
spec:
  source:
    type: Backup
    backup:
      name: mysql-qa-helm-release-full-backup
      namespace: demo-backup-ns
  skipIfAlreadyExists: true

Explanation for the above configuration:

• spec.source.type: Specifies what backup type to restore from.
• spec.source.backup: Contains a reference to the backup object to restore from.
• spec.skipIfAlreadyExists: Specifies whether to skip the restore of a resource if it already exists in the namespace being restored.

Restore allows you to restore the last successful Backup for an application. It is used to restore a single namespace or Helm release protected by the Backup CRD. The Backup CRD is identified by its name: mysql-qa-helm-release-full-backup.

First, inspect the Restore CRD example from the ovh/docs Git repository:

cat manifests/mysql-qa-helm-release-restore.yaml

Then, create the Restore resource using kubectl:

kubectl apply -f manifests/mysql-qa-helm-release-restore.yaml

Finally, inspect the Restore object status:

kubectl get restore mysql-qa-helm-release-restore -n demo-restore-ns

The output should look similar to the following (notice the STATUS column set toCompleted, as well as the PERCENTAGE COMPLETED set to100):

NAME                            STATUS      DATA SIZE   START TIME             END TIME               PERCENTAGE COMPLETED   DURATION
mysql-qa-helm-release-restore   Completed   0           2022-06-21T15:06:52Z   2022-06-21T15:07:35Z   100                    43.524191306s

If the output looks like the one above, then the mysql-qa Helm release restoration process has completed successfully.

Check that all the demo-restore-ns namespace resources are in place and running:

kubectl get all -n demo-restore-ns

The output looks similar to:

NAME                                                           READY   STATUS      RESTARTS   AGE
pod/mysql-qa-665f6fb548-m8tnd                                  1/1     Running     0          91m
pod/mysql-qa-helm-release-full-backup-metamover-9w7s0y-x8867   0/1     Completed   0          9m2s

NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/mysql-qa   ClusterIP   10.3.227.118   <none>        3306/TCP   91m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql-qa   1/1     1            1           91m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-qa-665f6fb548   1         1         1       91m

NAME                                                           COMPLETIONS   DURATION   AGE
job.batch/mysql-qa-helm-release-full-backup-metamover-9w7s0y   1/1           28s        9m2s

The next step deals with whole cluster backup and restore, thus covering a disaster recovery scenario.

The main idea here is to perform an OVHcloud Managed Kubernetes Cluster backup by including all important namespaces that hold your essential applications and configurations. Basically, we cannot name it a full cluster backup and restore, but rather a multi-namespace backup and restore operation. In practice, this is all that’s needed because everything is “namespaced” in Kubernetes. You will also learn how to perform a cluster restore operation via location from the target. The same flow applies when you need to perform cluster migration.

A typical Cluster Backup Plan manifest targeting multiple namespaces looks like this:

apiVersion: triliovault.trilio.io/v1
kind: ClusterBackupPlan
metadata:
  name: ovh-multi-ns-backup-plan
  namespace: default
spec:
  backupConfig:
    target:
      name: trilio-ovh-s3-target
      namespace: default
  backupComponents:
    - namespace: default
    - namespace: demo-backup-ns
    - namespace: backend
    - namespace: monitoring

Notice that kube-system (or other OVHcloud Managed Kubernetes Cluster related namespaces) is not included in the list. Usually, this is not required unless there is a special case requiring some settings to be persisted at that level.

A typical Cluster Backup manifest targeting multiple namespaces looks like this:

apiVersion: triliovault.trilio.io/v1
kind: ClusterBackup
metadata:
  name: multi-ns-backup
  namespace: default
spec:
  type: Full
  clusterBackupPlan:
    name: ovh-multi-ns-backup-plan
    namespace: default

Steps to initiate a backup for all of the important namespaces in your OVHcloud Managed Kubernetes Cluster:

1. First, go to the directory where the ovh/docs Git repository was cloned on your local machine:

   cd docs

   Copy

2. Then, open and inspect the Cluster Backup Plan and Cluster Backup manifest files provided in the docs repository.

   cat manifests/multi-ns-backup-plan.yaml
   cat manifests/multi-ns-backup.yaml

   Copy

3. Create the Cluster Backup Plan resource using kubectl:

   kubectl apply -f manifests/multi-ns-backup-plan.yaml

   Copy

   Now, inspect the Cluster Backup Plan status using kubectl:

   kubectl get clusterbackupplan multi-ns-backup-plan -n default

   Copy

   The output should look similar to the following (notice the STATUS column value, which should be set to “Available”):

   NAME                            TARGET                 ...      STATUS
   ovh-multi-ns-backup-plan        trilio-ovh-s3-target   ...      Available

   Copy

4. Finally, create the Cluster Backup resource using kubectl:

   kubectl apply -f manifests/multi-ns-cluster-backup.yaml

   Copy

   Next, check the Cluster Backup status using kubectl:

   kubectl get clusterbackup multi-ns-cluster-backup -n default

   Copy

   The output should look similar to the following (notice the STATUS column value, which should be set to “Available”, as well as the PERCENTAGE COMPLETE set to “100”):

   NAME                BACKUPPLAN                  BACKUP TYPE   STATUS        ...     COMPLETE
   multi-ns-backup     ovh-multi-ns-backup-plan    Full          Available      ...     100                               

   Copy

If the output looks like the one above, then all of your important application namespaces were backed up successfully.

You can also open the web console main dashboard and inspect the multi-namespace backup (notice how all the important namespaces that were backed up are highlighted in green, in a honeycomb structure).

An important aspect to keep in mind is that whenever you destroy an OVHcloud Managed Kubernetes Cluster and then restore it, a new Load Balancer with a new external IP is created as well when TVK restores your ingress controller. So please make sure to update your A records accordingly.

Now, delete the whole OVHcloud Managed Kubernetes Cluster using the OVHcloud Control Panel.

Next, re-create the cluster as described in Creating an OVHcloud Managed Kubernetes Cluster.

To perform the restore operation, you need to install the TVK application as described in Step 1 - Installing TrilioVault for Kubernetes. Please make sure to use the same Helm Chart version - this is important!

After the installation finishes successfully, configure the TVK target as described in Step 2 - Creating a TrilioVault Target to Store Backups, and point it to the same OVHcloud Object Storage bucket where your backup data is located. Also, please make sure that target browsing is enabled.

Next, verify and activate a new license as described in the TrilioVault Application Licensing section.

To get access to the web console user interface, please consult the Getting Access to the TVK Web Management Console section.

Then, navigate to Resource Management > TVK Namespace > Targets  (in the case of ovh/docs, the TVK Namespace is tvk).

Going further, browse the target and list the available backups by clicking the Actions button on the right. Then, select the Launch Browser option from the pop-up menu (for this to work, the target must have the enableBrowsing flag set to “true”).

Now click on the multi-ns-backup-plan item from the list, and then click and expand the multi-ns-backup item from the right sub-window, similar to:

To start the restore process, click the Restore button. A progress window will be displayed similar to the one below:

After a while, if the progress window looks like the one below, then the multi-namespace restore operation completed successfully.

First, verify all of the cluster Kubernetes resources (you should have everything in place):

kubectl get all --all-namespaces

In the next step, you will learn how to perform scheduled (or automatic) backups for your OVHcloud Managed Kubernetes Cluster applications.

Retention polices can be used for either Backup Plan or Cluster Backup Plan CRDs. A typical Policy manifest for the Retention type looks like this:

apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
  name: sample-ret-policy
spec:
  type: Retention
  retentionConfig:
    latest: 2
    weekly: 1
    dayOfWeek: Wednesday
    monthly: 1
    dateOfMonth: 15
    monthOfYear: March
    yearly: 1

Explanation for the above configuration:

• spec.type: Defines policy type. Can be Retention or Schedule.
• spec.retentionConfig: Describes the retention configuration, such as what interval to use for backup retention and how many to make.
• spec.retentionConfig.latest: Maximum number of the latest backups to be retained.
• spec.retentionConfig.weekly: Maximum number of backups to be retained in a week.
• spec.retentionConfig.dayOfWeek: Day of the week to maintain weekly backups.
• spec.retentionConfig.monthly: Maximum number of backups to be retained in a month.
• spec.retentionConfig.dateOfMonth: Date of the month to maintain monthly backups.
• spec.retentionConfig.monthOfYear: Month of the backup to retain for yearly backups.
• spec.retentionConfig.yearly: Maximum number of backups to be retained in a year.

The above retention policy translates to:

• On a weekly basis, keep one backup each Wednesday.
• On a monthly basis, keep one backup on the 15th day.
• On a yearly basis, keep one backup every March.
• Overall, I want to always have the 2 most recent backups available.

The basic flow for creating a retention policy resource goes the same way as with scheduled backups. You need a Backup Plan or a Cluster Backup Plan CRD defined to reference the retention policy, and then you need to have a Backup or Cluster Backup object to trigger the process.

A typical Cluster Backup Plan configuration with retention set looks like this:

apiVersion: triliovault.trilio.io/v1
kind: ClusterBackupPlan
metadata:
  name: multi-ns-backup-plan-15min-schedule-retention
  namespace: default
spec:
  backupConfig:
    target:
      name: trilio-ovh-s3-target
      namespace: default
    retentionPolicy:
        name: sample-ret-policy
        namespace: default
  backupComponents:
    - namespace: default
    - namespace: backend

Once you apply the Cluster Backup Plan, you can check it using the following command:

kubect get clusterbackupplan -n default

The output should look similar to the example below:

NAME                                            TARGET                 RETENTION POLICY    ...      STATUS
multi-ns-backup-plan-15min-schedule-retention   trilio-ovh-s3-target   sample-ret-policy   ...      Available  

Notice that it uses a retentionPolicy field to reference the policy in question. Of course, you can have a backup plan that has both types of policies set, so that it can perform scheduled backups as well as deal with retention strategies.

Having so many TVK resources (each one responsible for various operations such as scheduled backups, retention, etc.), something is likely to go wrong at some point in time. It means that some of the previously enumerated operations might fail due to various reasons, such as inaccessible storage, network issues for NFS, etc.

So, what happens is that your OVHcloud Managed Kubernetes Cluster will get crowded with many Kubernetes objects in a failed state.

You need a way to garbage collect all those objects in the end and release associated resources to avoid trouble in the future. Meet the Cleanup Policy CRD:

apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
  name: garbage-collect-policy
  namespace: tvk
spec:
  type: Cleanup
  cleanupConfig:
    backupDays: 5

The above cleanup policy must be defined in the TVK install namespace. Then, a cron job is created automatically for you that runs every 30 mins, and deletes failed backups based on the value specified for backupdays within the spec field.

This is a very neat feature that TVK provides to help you deal with this kind of situation.