How to Encrypt Kubernetes ETCD with OVHcloud KMS

Find out how to configure Kubernetes to encrypt ETCD storage with the OVHcloud KMS KMIP interface.

Objective

This guide explains how to set up the kube-apiserver encryption provider so Kubernetes clusters can encrypt and decrypt data at rest using OVHcloud KMS via the KMIP protocol.

Requirements

• An OVHcloud customer account.
• You must have ordered an OKMS domain.

Instructions

Installing the binary

The binary can be installed directly from Go packages.

go install github.com/ovh/okms-k8s-encryption-provider@latest

Or you can build from sources.

git clone https://github.com/ovh/okms-k8s-encryption-provider.git
cd okms-k8s-encryption-provider
go build -o okms-k8s-encryption-provider

Configuring OVHcloud KMS (OKMS)

To use OVHcloud KMS as an encryption provider for Kubernetes, you will need the following:

• An OVHcloud user and permissions to manage OKMS KMIP keys.
• An access certificate for your OKMS domain.
• A KMIP AES key in your OKMS.

Creating user and access rights

Create an IAM local user with access rights on your domain.

If you are using IAM policies instead, the user should have at least the following rights on the OKMS domain:

• okms:kmip:encrypt
• okms:kmip:decrypt
• okms:kmip:locate

Otherwise, the user should be a member of a group with the ADMIN role.

Alternatively, it is possible to create a user using the OVHcloud CLI:

ovhcloud iam user create --login "etcd-encryption" --group ADMIN --description "A user created for ETCD encryption" --password "xxxxxxxxx" --email "xxxxx@mycompany.com"

Creating access certificate

Create an OKMS access certificate and link the user previously created. 

Save the certificate cert.pem and the private key key.pem generated, as they will be required for the encryption provider configuration.

Creating KMIP AES key

To create a KMIP AES key, you can use the OKMS CLI:

Start by downloading the binary from the latest release or building from source.

Then you can create a key using:

okms-cli kmip create symmetric --alg aes --size 256

Keep the Key ID of the key generated. For the rest of the guide we'll use the Key ID 70001308-5674-43fe-93dd-6270ecac0710 as an example.

For more information on how to use the okms-cli, refer to the GitHub repository.

Configuring encryption provider

The encryption provider can be run on the kube-apiserver hosts directly with the following command line:

./okms-k8s-encryption-provider \
  --client-cert "~/.ovh-kms/cert.pem" \
  --client-key "~/.ovh-kms/key.pem" \
  --kmip-addr "us-east-vin.okms.ovh.us:5696" \
  --kmip-key-id "70001308-5674-43fe-93dd-6270ecac0710"

The encryption provider supports the following options:

Configuring Kubernetes

Based on the official Kubernetes guide for encrypting data with a KMS provider, add the following flags on your kube-apiserver:

  --encryption-provider-config=<path/to>/encryption-config.yaml
  # Optional: reload the file if it is updated
  --encryption-provider-config-automatic-reload=true

Make sure to mount the directory containing the Unix socket that the KMS server is listening on into the kube-apiserver.

An example of encryption-config.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - kms:
        name: okms-encryption-provider
        endpoint: unix:///var/run/okms_etcd_plugin.sock
        cachesize: 1000
        timeout: 3s
    - identity: {}

Validating configuration

Create a secret with kubectl create secret generic okms-test-secret -n default --from-literal=mykey=mydata and then check the contents of the secret in ETCD storage by running the following:

ETCDCTL_API=3 etcdctl \
    --key /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.key \
    --cert  /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt \
    --cacert /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt  \
    --endpoints "https://etcd-a.internal.${CLUSTER}:4001" get /registry/secrets/default/okms-test-secret

The output should be unreadable:

0m`�He.0�cryption-provider:�1x��%�B���#JP��J���*ȝ���΂@\n�96�^��ۦ�~0| *�H��
                    `q�*�J�.P��;&~��o#�O�8m��->8L��0�C3���A7�����~���f�V�ܬ���X��_��`�H#�D��z)+�81��qW��y��`�q��}1<LF, ��N��p����i*�aC#E�߸�s������s��l�?�a
�AźR������.��8H�4�O

Implementing key rotation

To rotate your key, you will need to run two encryption providers, each listening on a different Unix socket.

Below is an example encryption configuration file for all API servers prior to using the new key:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    # provider using old key
    - kms:
        name: okms-encryption-provider
        endpoint: unix:///var/run/kmsplugin/socket.sock
        cachesize: 1000
        timeout: 3s
    # provider using new key
    - kms:
        name: okms-encryption-provider-2
        endpoint: unix:///var/run/kmsplugin/socket2.sock
        cachesize: 1000
        timeout: 3s
    - identity: {}

After all API servers have been restarted and are able to decrypt using the new key, move the provider with the new key on top.

After all secrets have been re-encrypted with the new key, you can remove the old encryption provider.

Go further

Find out how to use Kubernetes External Secrets Operator with Secret Manager.