- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
questions about kafka and compression.type
- Labels:
-
Apache Kafka
Created 05-06-2022 06:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
is it possible to alter compression.type on a kafka topic while having applications on the topic running?
What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip.
In CM, I see a global setting.
Does it apply to KafkaMirrorMaker only?
Are producer and consumer applications somehow affected, when the global parameter changes.
Best regards
Jaro
Created 08-06-2022 10:28 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please check my inline answer to your queries:
1) is it possible to alter compression.type on a kafka topic while having applications on the topic running?
>> Yes, we would recommend stopping the producer and then alert compression type as whenever the Kafka broker gets a compressed batch of messages from a producer. It will decompress the data to validate if the compression type is different at the topic level and producer level and if compression is the same from both sides, then it will write directly to the topic/leader partition.
2) What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip.
>> You should use and test snappy or lz4 for best speed/compression ratio
3) Does it apply to KafkaMirrorMaker only?
>> Yes, as per the screenshot it will apply for the MM only.
4) Are producer and consumer applications somehow affected, when the global parameter changes.
>> If you are changing it from the Kafka side, then it should not affect the applications.
If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post.
Thank you.
Created 08-06-2022 10:28 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please check my inline answer to your queries:
1) is it possible to alter compression.type on a kafka topic while having applications on the topic running?
>> Yes, we would recommend stopping the producer and then alert compression type as whenever the Kafka broker gets a compressed batch of messages from a producer. It will decompress the data to validate if the compression type is different at the topic level and producer level and if compression is the same from both sides, then it will write directly to the topic/leader partition.
2) What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip.
>> You should use and test snappy or lz4 for best speed/compression ratio
3) Does it apply to KafkaMirrorMaker only?
>> Yes, as per the screenshot it will apply for the MM only.
4) Are producer and consumer applications somehow affected, when the global parameter changes.
>> If you are changing it from the Kafka side, then it should not affect the applications.
If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post.
Thank you.