kafka Dead Letter Queue

In the kafka sidecar reactive consumer, user have ability to set Dead Letter Queue (DLQ) and sidecar will send those message (events) with process error to the DLQ directly:

Configuration

The following is the config values for DLQ in kafka-consumer.yml

# Indicator if the dead letter topDead Letter Queueic is enabled.
deadLetterEnabled: ${kafka-consumer.deadLetterEnabled:true}
# The extension of the dead letter queue(topic) that is added to the original topic to form the dead letter topic
deadLetterTopicExt: ${kafka-consumer.deadLetterTopicExt:.dlq}

By default, the DLQ name is the consumer topic name with ‘.dlq’ extension. But user can change the extension with different format.

If the event process with errors, user can set RecordProcessedResult by setting processed = ‘false’:

 RecordProcessedResult rpr = new RecordProcessedResult(record, false, sw.toString());

For kafka sidecar reactive consume, if the deadLetterEnabled is true and the RecordProcessedResult processed with error, it will send the record to Dead Letter Queue automatically.

The example can be found in the light-example-4j/kafka/sidecar-backend

Reference for confluent Dead Letter Queue

A Dead Letter Queue topic is autogenerated for Confluent Cloud sink connectors.

For Connect, errors that may occur are typically serialization and deserialization (serde) errors. In Confluent Cloud, the connector does not stop when serde errors occur. Instead, the connector continues processing records and sends the errors to a Dead Letter Queue (DLQ). You can use the record headers in a DLQ topic record to identify and troubleshoot an error when it occurs.

https://docs.confluent.io/cloud/current/connectors/dead-letter-queue.html

https://docs.confluent.io/platform/current/connect/concepts.html#dead-letter-queue