Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to set offset.storage to kafka in HDP 2.4 ?

avatar
Contributor

Hi

Almost evrything is in the title.. I want to switch from an offsets management in ZK to an offsets management in kafka. How to do this in HDP 2.4 ? I tried to add offsets.storage and offsets.storage.topic in the "custom kafka broker" and parameters are well taken into account :

[kafka@mykafka conf]$ grep offsets.storage *

server.properties:offsets.storage=kafka

server.properties:offsets.storage.topic=offset-topic

But it doesn't work... and nothing is recorded in my offset topic.

Thx in advance. Regards...

1 ACCEPTED SOLUTION

avatar

@Régis LE BRETONNIC there is no such property as offsets.storage on kafka broker side i.e server.properties. offsets.storage is consumer side config.

If you are using new consumer api than it defaults to storing the offsets in kafka topic but if you are using old high-level consumer api than it defaults to using zookeeper as storage. To change that make sure you add offsets.storage=kafka in consumer.properties.

View solution in original post

4 REPLIES 4

avatar
Super Guru

did you try setting dual.commit.enabled to true?

avatar
Contributor

Yes I set it to true.

I made a mistake... It seems that it is offsets.storage (with S) and offset.storage.topic (w/o S)... See :http://kafka.apache.org/documentation.html

But even after having fixed this, it doesn't work. Did you achieve to us offset management within kafka ?

Regards

avatar

@Régis LE BRETONNIC there is no such property as offsets.storage on kafka broker side i.e server.properties. offsets.storage is consumer side config.

If you are using new consumer api than it defaults to storing the offsets in kafka topic but if you are using old high-level consumer api than it defaults to using zookeeper as storage. To change that make sure you add offsets.storage=kafka in consumer.properties.

avatar
Contributor

Of course you're right. I was thinking that this property was inherited from the server but it is a consumer feature. It worked as wished now... I still have an issue when I want to personalize the offset topic :

[2016-06-08 05:20:02,664] WARN Property offset.storage.topic is not valid (kafka.utils.VerifiableProperties)

I will investigate, but if I ignore this property and use the default topic __consumer_offsets it works fine.

Thk a lot for your clarification.