Skip to main content

Kafka® Connect - Advanced parameters references

Stephanie R.
Updated

This guide lists some of the supported advanced parameters that allow you to configure your Cloud Analytics with Kafka® Connect according to your use cases.

 

Requirements

 

Instructions

NOTE: The management of advanced parameters is not supported via Terraform.

Using the OVHcloud Control Panel

Please refer to this guide to find out how to change your advanced parameters from the OVHcloud Control Panel.

 

Using API

Please refer to this guide to find out how to change your advanced parameters from the OVHcloud API.

 

Advanced parameters for Kafka® Connect

You can find a complete list of configuration options via OVHcloud API. Below is a summary of some configuration options available for Kafka® Connect service:

Parameters
kafka_connect.connector_client_config_override_policy string
values: All, None
Defines what client configurations can be overridden by the connector. Default is None
kafka_connect.consumer_auto_offset_reset string
values: earliest, latest
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server. Default is earliest
kafka_connect.consumer_fetch_max_bytes long
min: 1048576 max: 104857600
Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not a absolute maximum.
kafka_connect.consumer_isolation_level string
values: read_committed, read_uncommitted
Transaction read isolation level. read_uncommitted is the default, but read_committed can be used if consume-exactly-once behavior is desired.
kafka_connect.consumer_max_partition_fetch_bytes long
min: 1048576 max: 104857600
Records are fetched in batches by the consumer.If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress.
kafka_connect.consumer_max_poll_interval_ms long
min: 1 max: 2147483647
The maximum delay in milliseconds between invocations of poll() when using consumer group management (defaults to 300000).
kafka_connect.consumer_max_poll_records long
min: 1 max: 10000
The maximum number of records returned in a single call to poll() (defaults to 500).
kafka_connect.offset_flush_interval_ms long
min: 1 max: 100000000
The interval at which to try committing offsets for tasks (defaults to 60000).
kafka_connect.offset_flush_timeout_ms long
min: 1 max: 2147483647
Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to offset storage before cancelling the process and restoring the offset data to be committed in a future attempt (defaults to 5000).
kafka_connect.producer_batch_size long
min: 0 max: 5242880
This setting gives the upper bound of the batch size to be sent. If there are fewer than this many bytes accumulated for this partition, the producer will 'linger' for the linger.ms time waiting for more records to show up. A batch size of zero will disable batching entirely (defaults to 16384).
kafka_connect.producer_buffer_memory long
min: 5242880 max: 134217728
The total bytes of memory the producer can use to buffer records waiting to be sent to the broker (defaults to 33554432).
kafka_connect.producer_compression_type string
values: gzip, lz4, none, snappy, zstd
Specify the default compression type for producers. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'none' which is the default and equivalent to no compression.
kafka_connect.producer_linger_ms long
min: 0 max: 5000
This setting gives the upper bound on the delay for batching: once there is batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if there are fewer than this many bytes accumulated for this partition the producer will 'linger' for the specified time waiting for more records to show up. Defaults to 0.
kafka_connect.producer_max_request_size long
min: 131072 max: 67108864
This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests.
kafka_connect.scheduled_rebalance_max_delay_ms long
min: 0 max: 600000
The maximum delay that is scheduled in order to wait for the return of one or more departed workers before rebalancing and reassigning their connectors and tasks to the group. During this period the connectors and tasks of the departed workers remain unassigned. Defaults to 5 minutes.
kafka_connect.session_timeout_ms long
min: 1 max: 2147483647
The timeout in milliseconds used to detect failures when using Kafka’s group management facilities (defaults to 10000).

 

Go further

For more information and tutorials, please see our other Managed Databases & Analytics or Platform as a Service guides. You can also explore the guides for other OVHcloud products and services.

OVHcloud Managed Databases and Analytics:

  • Grafana® is a registered trademark of Grafana Labs and is used with the permission of Grafana Labs. OVH SAS and its subsidiaries are not affiliated with or endorsed by Grafana Labs.
  • Kafka® is a registered trademark of The Apache Software Foundation and has been licensed for use by OVHcloud, who has no affiliation with and is not endorsed by The Apache Software Foundation.
  • MongoDB® is a registered trademark of MongoDB, Inc.
  • MySQL® is a registered trademark of Oracle and/or its affiliates.
  • PostgreSQL® is a registered trademark of the PostgreSQL Community Association of Canada.

Related articles