1. Articles
  2. Platform as a Service
  3. Managed Kubernetes (K8s)
  4. Technical Resources

Articles in this section

Managed Kubernetes - Known Limits

Avatar
Stephanie R.
  • Updated

Nodes and pods

We have tested our OVHcloud Managed Kubernetes service with up to 100 nodes and 100 pods per node. While we are fairly sure it can go further, we advise you to keep under those limits.

Nodepools with anti-affinity are limited to 5 nodes (but you can create multiple node pools with the same instance flavor if needed of course). A node can run up to 110 pods. This limit does not depend on node flavor.

In general, it is better to have several mid-size Kubernetes clusters than a monster-size one.

To ensure high availability for your services, it is recommended to possess the computation power capable of handling your workload even when one of your nodes becomes unavailable.

Please note that any operation requested to our services, like node deletions or rolling updates, will try to gracefully drain the nodes by respecting Pod Disruption Budgets for a maximum duration of 10 minutes. After this period, the nodes will be forcefully drained to ensure the smooth progress of the operation. This graceful node draining process only applies when there is at least one other node in the cluster.

Most worker nodes (be they added manually or through cluster autoscaler) are created within a few minutes, except GPU worker nodes (t1 and t2 flavors) where ready status can take up to a bit more than one hour.

Delivering a fully managed service, including OS and other component updates, you will neither need nor be able to SSH into your nodes.

 

Data persistence

If an incident is detected by the OVHcloud monitoring service, as part of auto-healing or in case of a version upgrade, the Nodes can be fully reinstalled.

We advise you to save your data in Persistent Volumes (PV), not to save data directly on Nodes if you don’t want to lose your data. Follow our guide about how to set up and manage Persistent Volumes on OVHcloud Managed Kubernetes for more information.

 

Load Balancer

Creating a Kubernetes Load Balancer service in a Managed Kubernetes cluster triggers the creation of a Public Cloud Load Balancer. The lifespan of the external Load Balancer (and thus the associated IP address) is linked to the lifespan of this Kubernetes resource.

There is a default quota of 200 external Load Balancers per OpenStack project (also named OpenStack tenant). This limit can be raised by contacting OVHcloud US support.

There is also a limit of 10 open ports on every Load Balancer, and these ports must be in a range between 6 and 65535. (Additionally, node-ports are using a default range of 30000 - 32767, allowing you to expose 2767 services/ports).

A Public Cloud Load Balancer has the following non-configurable timeouts:

  • 20 seconds for the backend connection to be established
  • 180 seconds for the client & server connections
The use of the UDP protocol is not supported.

 

OpenStack

Our Managed Kubernetes service is based on OpenStack, and your nodes and persistent volumes are built on it, using OVHcloud Public Cloud. As such, you can see them in the Compute > Instances section of your OVHcloud Public Cloud Control Panel. However, this does not mean that you can deal directly with these nodes and persistent volumes the same way you can do it for other Public Cloud instances.

The managed part of OVHcloud Managed Kubernetes Service means that we have configured those nodes and volumes to be part of our Managed Kubernetes.
Please refrain from manipulating them from the OVHcloud Public Cloud Control Panel (modifying ports left open, renaming volumes, resizing volumes, etc.), as you could break them.

There is also a limit of 20 Managed Kubernetes Services per OpenStack project.

 

Node naming

Due to known limitations currently present in the Kubelet service, be careful to set a unique name for all of your OpenStack instances running in your tenant including your “Managed Kubernetes Service” nodes and the instances that start directly on OpenStack through the OVHcloud Control Panel or API.

The usage of the period (.) character is forbidden in the node name. Please use the dash (-) character instead.

 

Ports

There are some ports that you shouldn’t block on your instances if you want to keep your OVHcloud Managed Kubernetes service running:

 

Ports to open from public network (INPUT)

  • TCP Port 22 (ssh): needed for node management by OVHcloud
  • TCP Ports from 30000 to 32767 (NodePort services port range): needed for Node Port and Load Balancer services
  • TCP Port 111 (rpcbind): needed only if you want to use the NFS client deployed on nodes managed by OVHcloud

 

Ports to open from instances to public network (OUTPUT)

  • TCP Port 443 (kubelet): needed for communication between the kubelets and the Kubernetes API server
  • TCP Port 80IP169.254.169.254/32 (init service): needed for OpenStack metadata service
  • TCP Ports from 25000 to 31999 (TLS tunnel): needed to tunnel traffic between pods and the Kubernetes API server
  • TCP Port 8090 (internal service): needed for node management by OVHcloud
  • UDP Port 123 (systemd-timesync): needed to allow NTP servers synchronization
  • TCP/UDP Port 53 (systemd-resolve): needed to allow domain name resolution
  • TCP Port 111 (rpcbind): needed only if you want to use the NFS client deployed on nodes managed by OVHcloud
  • TCP Port 4443 (metrics server): needed for communication between the metrics server and the Kubernetes API server

 

Ports to open from other worker nodes (INPUT/OUTPUT)

 

About OpenStack security groups

In case you want to apply OpenStack security groups to your nodes, it is mandatory to add the above ports in a ruleset concerning the 0.0.0.0/0 block.

NOTE: If you remove the default rules, accepting all input and output, when creating a new security group, make sure to allow the ports needed by your application, as well as the mandatory ports mentioned above.
To simplify your policy, you can add these rules which do not specify any port and will allow all internal traffic between pods and services within the cluster:
Direction Ether Type IP Protocol Port Range Remote IP Prefix Description
Ingress IPv4 TCP Any 10.2.0.0/16 Allow traffic from pods
Ingress IPv4 TCP Any 10.3.0.0/16 Allow traffic from services

This allows you to trust the internal traffic between pods and services within the cluster.

 

Private Networks

The vRack feature is currently available and compliant with our Managed Kubernetes Service.

To prevent any conflict, we advise you to keep the DHCP service running in your private network.

NOTE: If you create your subnet via the OVHcloud APIv6, please ensure that the option noGateway is checked if you do not have a gateway on this subnet. Not doing so will result in faulty Load Balancer services.

NOTE: If your cluster has been created using an OpenStack Private Network, you should not change this private network's name or the network's subnet name.

The OpenStack Cloud Controller Manager (CCM) uses the network name to create private network connectivity inside the cluster to link nodes to the private network. Changing either the private network name or the network's subnet name will have an impact on future nodes to be deployed as the CCM cannot find network information.

The CCM cannot fetch private network information on the OpenStack side to initialize networking on the freshly deployed nodes on the Kubernetes side.

Nodes will have an "uninitialized=true:NoSchedule" taint that prevents the kube-scheduler from deploying pods on those new uninitialized nodes. Nodes impacted by this use case don't have an External-IP as well.

 

Known non-compliant IP ranges

The following subnets are not compliant with the vRack feature and can generate some incoherent behaviors with our used overlay networks:

10.2.0.0/16 # Subnet used by pods
10.3.0.0/16 # Subnet used by services
172.17.0.0/16 # Subnet used by the Docker daemon

 

Cluster health

The command kubectl get componentstatus is reporting the scheduler, the controller manager, and the etcd service as unhealthy. This is a limitation due to our implementation of the Kubernetes control plane as the endpoints needed to report the health of these components are not accessible.

 

Persistent Volumes

Kubernetes Persistent Volume Claims resizing only allows you to expand volumes, not decrease them. If you try to decrease the storage size, you will get a message like:

The PersistentVolumeClaim is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

For more details, please refer to the Resizing Persistent Volumes documentation.

The Persistent Volumes are using our Cinder-based block-storage solution through Cinder CSI.
A worker node can have a maximum of 254 persistent volumes attached to it, and a persistent volume can only be attached to a single worker node.

 

Go further

For more information and tutorials, please see our other Managed Kubernetes support guides or explore the guides for other OVHcloud products and services.

Related articles