how to set offset.storage to kafka in HDP 2.4 ?

[#1]

Hi

Almost evrything is in the title.. I want to switch from an offsets management in ZK to an offsets management in kafka. How to do this in HDP 2.4 ? I tried to add offsets.storage and offsets.storage.topic in the "custom kafka broker" and parameters are well taken into account :

[kafka@mykafka conf]$ grep offsets.storage *

server.properties:offsets.storage=kafka

server.properties:offsets.storage.topic=offset-topic

But it doesn't work... and nothing is recorded in my offset topic.

Thx in advance. Regards...

[#2]

did you try setting dual.commit.enabled to true?

[#3]

Yes I set it to true.

I made a mistake... It seems that it is offsets.storage (with S) and offset.storage.topic (w/o S)... See :http://kafka.apache.org/documentation.html

But even after having fixed this, it doesn't work. Did you achieve to us offset management within kafka ?

Regards

[#4 · Accepted solution]

@Régis LE BRETONNIC there is no such property as offsets.storage on kafka broker side i.e server.properties. offsets.storage is consumer side config.

If you are using new consumer api than it defaults to storing the offsets in kafka topic but if you are using old high-level consumer api than it defaults to using zookeeper as storage. To change that make sure you add offsets.storage=kafka in consumer.properties.

[#5]

Of course you're right. I was thinking that this property was inherited from the server but it is a consumer feature. It worked as wished now... I still have an issue when I want to personalize the offset topic :

[2016-06-08 05:20:02,664] WARN Property offset.storage.topic is not valid (kafka.utils.VerifiableProperties)

I will investigate, but if I ignore this property and use the default topic __consumer_offsets it works fine.

Thk a lot for your clarification.