Kafka® - Advanced parameters references – Support Guides

This guide lists some of the supported advanced parameters that allow you to configure your Cloud Analytics with Kafka® according to your use cases.

Requirements

A Public Cloud project in your OVHcloud account
A Kafka® service running on your OVHcloud Cloud Analytics (this guide can help you to meet this requirement)
Access to your OVHcloud Control Panel or the OVHcloud API

Instructions

NOTE: The management of advanced parameters is not supported via Terraform.

Using the OVHcloud Control Panel

Please refer to this guide to find out how to change your advanced parameters from the OVHcloud Control Panel.

Using API

Please refer to this guide to find out how to change your advanced parameters from the OVHcloud API.

Advanced parameters for Kafka®

You can find a complete list of configuration options via OVHcloud API. Below is a summary of some configuration options available for Kafka® service:

Parameters
kafka.auto_create_topics_enable `boolean` Enable auto-creation of topics. (Default: true)
kafka.compression_type `string` values: `gzip`, `lz4`, `producer`, `snappy`, `uncompressed`, `zstd` Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer.(Default: producer)
kafka.connections_max_idle_ms `long` min: `1000` max: `3600000` Idle connections timeout: the server socket processor threads close the connections that idle for longer than this. (Default: 600000 ms (10 minutes))
kafka.default_replication_factor `long` min: `1` max: `10` Replication factor for auto-created topics (Default: 3)
kafka.group_initial_rebalance_delay_ms `long` min: `0` max: `300000` The amount of time, in milliseconds, the group coordinator will wait for more consumers to join a new group before performing the first rebalance. A longer delay means potentially fewer rebalances, but increases the time until processing begins. The default value for this is 3 seconds. During development and testing it might be desirable to set this to 0 in order to not delay test execution time. (Default: 3000 ms (3 seconds))
kafka.group_max_session_timeout_ms `long` min: `0` max: `1800000` The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures. Default: 1800000 ms (30 minutes)
kafka.group_min_session_timeout_ms `long` min: `0` max: `60000` The minimum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures. (Default: 6000 ms (6 seconds))
kafka.log_cleaner_delete_retention_ms `long` min: `0` max: `315569260000` How long are delete records retained? (Default: 86400000 (1 day))
kafka.log_cleaner_max_compaction_lag_ms `long` min: `30000` max: `9223372036854776000` The maximum amount of time message will remain uncompacted. Only applicable for logs that are being compacted. (Default: 9223372036854775807 ms (Long.MAX_VALUE))
kafka.log_cleaner_min_cleanable_ratio `double` min: `0.2` max: `0.9` Controls log compactor frequency. Larger value means more frequent compactions but also more space wasted for logs. Consider setting log.cleaner.max.compaction.lag.ms to enforce compactions sooner, instead of setting a very high value for this option. (Default: 0.5)
kafka.log_cleaner_min_compaction_lag_ms `long` min: `0` max: `9223372036854776000` The minimum time a message will remain uncompacted in the log. Only applicable for logs that are being compacted. (Default: 0 ms)
kafka.log_cleanup_policy `string` values: `compact`, `compact,delete`, `delete` The default cleanup policy for segments beyond the retention window (Default: delete)
kafka.log_flush_interval_messages `long` min: `1` max: `9223372036854776000` The number of messages accumulated on a log partition before messages are flushed to disk (Default: 9223372036854775807 (Long.MAX_VALUE))
kafka.log_flush_interval_ms `long` min: `0` max: `9223372036854776000` The maximum time in ms that a message in any topic is kept in memory (page-cache) before flushed to disk. If not set, the value in log.flush.scheduler.interval.ms is used (Default: null)
kafka.log_index_interval_bytes `long` min: `0` max: `104857600` The interval with which Kafka adds an entry to the offset index (Default: 4096 bytes (4 kibibytes))
kafka.log_index_size_max_bytes `long` min: `1048576` max: `104857600` The maximum size in bytes of the offset index (Default: 10485760 (10 mebibytes))
kafka.log_local_retention_bytes `long` min: `-2` max: `9223372036854776000` The maximum size of local log segments that can grow for a partition before it gets eligible for deletion. If set to -2, the value of log.retention.bytes is used. The effective value should always be less than or equal to log.retention.bytes value. (Default: -2)
kafka.log_local_retention_ms `long` min: `-2` max: `9223372036854776000` The number of milliseconds to keep the local log segments before it gets eligible for deletion. If set to -2, the value of log.retention.ms is used. The effective value should always be less than or equal to log.retention.ms value. (Default: -2)
kafka.log_message_downconversion_enable `boolean` This configuration controls whether down-conversion of message formats is enabled to satisfy consume requests. (Default: true)
kafka.log_message_timestamp_difference_max_ms `long` min: `0` max: `9223372036854776000` The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message (Default: 9223372036854775807 (Long.MAX_VALUE))
kafka.log_message_timestamp_type `string` values: `CreateTime`, `LogAppendTime` Define whether the timestamp in the message is message create time or log append time. (Default: CreateTime)
kafka.log_preallocate `boolean` Should pre allocate file when create new segment? (Default: false)
kafka.log_retention_bytes `long` min: `-1` max: `9223372036854776000` The maximum size of the log before deleting messages (Default: -1)
kafka.log_retention_hours `long` min: `-1` max: `2147483647` The number of hours to keep a log file before deleting it (Default: 168 hours (1 week))
kafka.log_retention_ms `long` min: `-1` max: `9223372036854776000` The number of milliseconds to keep a log file before deleting it (in milliseconds), If not set, the value in log.retention.minutes is used. If set to -1, no time limit is applied. (Default: null, log.retention.hours applies)
kafka.log_roll_jitter_ms `long` min: `0` max: `9223372036854776000` The maximum jitter to subtract from logRollTimeMillis (in milliseconds). If not set, the value in log.roll.jitter.hours is used (Default: null)
kafka.log_roll_ms `long` min: `1` max: `9223372036854776000` The maximum time before a new log segment is rolled out (in milliseconds). (Default: null, log.roll.hours applies (Default: 168, 7 days))
kafka.log_segment_bytes `long` min: `10485760` max: `1073741824` The maximum size of a single log file (Default: 1073741824 bytes (1 gibibyte))
kafka.log_segment_delete_delay_ms `long` min: `0` max: `3600000` The amount of time to wait before deleting a file from the filesystem (Default: 60000 ms (1 minute))
kafka.max_connections_per_ip `long` min: `256` max: `2147483647` The maximum number of connections allowed from each ip address (Default: 2147483647).
kafka.max_incremental_fetch_session_cache_slots `long` min: `1000` max: `10000` The maximum number of incremental fetch sessions that the broker will maintain. (Default: 1000)
kafka.message_max_bytes `long` min: `0` max: `100001200` The maximum size of message that the server can receive. (Default: 1048588 bytes (1 mebibyte + 12 bytes))
kafka.min_insync_replicas `long` min: `1` max: `7` When a producer sets acks to 'all' (or '-1'), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. (Default: 1)
kafka.num_partitions `long` min: `1` max: `1000` Number of partitions for auto-created topics (Default: 1)
kafka.offsets_retention_minutes `long` min: `1` max: `2147483647` Log retention window in minutes for offsets topic (Default: 10080 minutes (7 days))
kafka.producer_purgatory_purge_interval_requests `long` min: `10` max: `10000` The purge interval (in number of requests) of the producer request purgatory (Default: 1000).
kafka.replica_fetch_max_bytes `long` min: `1048576` max: `104857600` The number of bytes of messages to attempt to fetch for each partition . This is not an absolute maximum, if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. (Default: 1048576 bytes (1 mebibytes))
kafka.replica_fetch_response_max_bytes `long` min: `10485760` max: `1048576000` Maximum bytes expected for the entire fetch response. Records are fetched in batches, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. As such, this is not an absolute maximum. (Default: 10485760 bytes (10 mebibytes))
kafka.sasl_oauthbearer_expected_audience `string` The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. (Default: null)
kafka.sasl_oauthbearer_expected_issuer `string` Optional setting for the broker to use to verify that the JWT was created by the expected issuer.(Default: null)
kafka.sasl_oauthbearer_jwks_endpoint_url `string` OIDC JWKS endpoint URL. By setting this the SASL SSL OAuth2/OIDC authentication is enabled. See also other options for SASL OAuth2/OIDC. (Default: null)
kafka.sasl_oauthbearer_sub_claim_name `string` Name of the scope from which to extract the subject claim from the JWT.(Default: sub)
kafka.socket_request_max_bytes `long` min: `10485760` max: `209715200` The maximum number of bytes in a socket request (Default: 104857600 bytes).
kafka.transaction_partition_verification_enable `boolean` Enable verification that checks that the partition has been added to the transaction before writing transactional records to the partition. (Default: true)
kafka.transaction_remove_expired_transaction_cleanup_interval_ms `long` min: `600000` max: `3600000` The interval at which to remove transactions that have expired due to transactional.id.expiration.ms passing (Default: 3600000 ms (1 hour)).
kafka.transaction_state_log_segment_bytes `long` min: `1048576` max: `2147483647` The transaction topic segment bytes should be kept relatively small in order to facilitate faster log compaction and cache loads (Default: 104857600 bytes (100 mebibytes)).
kafka_authentication_methods.certificate `boolean` Enable certificate/SSL authentication
kafka_authentication_methods.sasl `boolean` Enable SASL authentication

Go further

For more information and tutorials, please see our other Managed Databases & Analytics or Platform as a Service guides. You can also explore the guides for other OVHcloud products and services.

OVHcloud Managed Databases and Analytics:

Grafana® is a registered trademark of Grafana Labs and is used with the permission of Grafana Labs. OVH SAS and its subsidiaries are not affiliated with or endorsed by Grafana Labs.
Kafka® is a registered trademark of The Apache Software Foundation and has been licensed for use by OVHcloud, who has no affiliation with and is not endorsed by The Apache Software Foundation.
MongoDB® is a registered trademark of MongoDB, Inc.
MySQL® is a registered trademark of Oracle and/or its affiliates.
PostgreSQL® is a registered trademark of the PostgreSQL Community Association of Canada.