Kafka - Capabilities and Limitations

This page provides the technical capabilities and limitations of the Managed Databases (also called Cloud Databases) for Kafka offer.

We continuously improve our offers. You can follow and submit ideas to add to our roadmap.

Capabilities and limitations

Supported regions

To view Cloud Databases availability, please see our regions and availability webpage.

Entire database instances have to be in the same region.

Kafka versions

The Cloud Databases offer supports the following Kafka versions:

• Kafka 3.5
• Kafka 3.6

Please refer to our Cloud Databases - Lifecycle Policy guide for recommendations on version upgrades and end-of-life announcements of major versions. Additionally, you can follow Kafka Release Cycle on their official page.

Kafka clients

You can use any of the Kafka-recommended clients to access your cluster.

Kafka Connect and Kafka Mirrormaker 2 are available at OVHcloud.

Plans

Two plans are available:

• Business
• Enterprise

Here is an overview of the various plans' capabilities:

Your choice of plan affects the number of nodes your cluster can run, the SLA, and a few other features such as read replicas or backup retention.

Nodes and replicas

• Business: the cluster is delivered with 3 nodes by default.
• Enterprise: the cluster is delivered with 6 nodes by default.

License type

Kafka software is under the Apache 2 license, a liberal open-source license. More information can be found here.

Hardware resources

Here are the node types you can choose from:

Business plans

Enterprise plans

Right now, all nodes of a given cluster should be of the same type and distributed in the same region.

Flexible storage

You can increase the storage of your cluster up to the maximum allowed for a given reference. Please refer to our Cloud Databases - Resize Your Cluster Storage guide for more information.

Node template upgrade

You can upgrade the node template of your cluster to scale your hardware resources up. This operation causes no interruption of service but be aware that you will not be able to downgrade the node template afterwards.

Disk type

The type of storage available may vary according to the region your cluster lives in: see Availability of Public Cloud products for more information about block storage type availability depending on the region. Thus, your cluster may be backed by e.g. High Speed or High Speed Gen2 block storage.

Also, the performance characteristics of the various storage offerings may vary depending on e.g. the storage size your cluster uses: High Speed may offer better iops than High Speed Gen2 for some disk sizes. See our Block Storage webpage for more information about those performance characteristics.

Cloud Databases will select the most efficient disk type for your cluster depending on your cluster parameters.

Effective storage

The disk size listed above is the total disk size of the underlying machine. However, a small part of it goes towards the OS install.

We try hard to avoid "disk full" situations that could be harmful to cluster health. Therefore:

1. When reaching a concerning level of disk usage, a warning email is sent.
2. When reaching a concerning level of disk usage, the service is moved to the "DISK_FULL" state, and "read-only" mode, meaning no more writes can be done.
3. You then can upgrade to a higher service plan with more storage.

See our Cloud Databases - Handling «Disk Full» Situations guide for more information.

Features

Network

Kafka clusters are reachable through a random port, attributed during cluster creation. Once your cluster is in RUNNING status, the Service URI will display the port to use.

Public as well as private networking (vRack) can be used for all the offers.

Ingress and Egress traffic are included in the service plans and unmetered.

The database service's IP address is subject to change periodically. Thus, it is advised not to rely on these IPs for any configuration, such as connection or egress policy. Instead, utilize the provided DNS record and implement CIDR-based egress policies for more robust and flexible network management.

Private network considerations

Here are some considerations to take into account when using a private network:

• Network ports are created in the private network of your choice. Thus, further operations on that network might be restricted - e.g. you won’t be able to delete the network if you didn’t stop the Cloud Databases services first.
• When connecting from an outside subnet, the OpenStack IP gateway must be enabled in the subnet used for the Database service. The customer is responsible for any other custom network setup.
• Subnet sizing should include considerations for service nodes, other co-located services within the same subnet, and an allocation of additional available IP addresses for maintenance purposes. Failure to adequately size subnets could result in operational challenges and the malfunctioning of services.

Authorized IPs

Once your service is up and running, you will be able to specify IP addresses (or CIDR blocks) to authorize incoming traffic. Until then, your service will be unreachable.

Maximum simultaneous connections

The number of simultaneous connections in Cloud Databases for Kafka depends on the available total memory on the node. We allow approximately 100 connections per 4 GB of RAM, capped to a maximum of 1000 active connections.

So for example on a server with 7 GB memory, you will get approximately 200 connections, and with 15 GB memory, you will get 400 connections.

Schema registry

Schema registry is supported and can be activated on a running Kafka service by setting the schemaRegistry to true with an API call:

PUT /cloud/project/{serviceName}/database/kafka/{clusterId}

Advanced parameters

You can further customize your Kafka by using advanced parameters. See our Kafka - Advanced Parameters References guide for more information on the supported parameters.

Backups

Kafka is a streaming tool. We don't back up Kafka data.

Logs and metrics

Logs and metrics are available through the Control Panel and the API. Additionally, cross-service integration can be configured to leverage your logs and metrics in other Cloud Database services. You could then view your Kafka logs in OpenSearch and metrics in Grafana (metrics have to be exported first in a time series compatible engine such as PostgreSQL or M3db). See our Cloud Databases - Cross Service Integration guide for more information.

• Logs retention: 1000 lines of logs
• Metrics retention: 1 calendar month

Please note that if the database instance is deleted, logs and metrics are also automatically deleted.

Users and roles

The creation of users is allowed via the Control Panel and API.

You can specify a username for each user. By default, the role is admin.

Topics

The creation of topics is allowed via the Control Panel and API.

For each topic, you can specify:

• A topic name;
• The replication factor;
• The number of partitions;
• The size of partitions;
• The retention time;
• The minimum in-sync replication;
• The deletion policy.

Go further

For more information and tutorials, please see our other Cloud Databases support guides or explore the guides for other OVHcloud products and services.