Support Questions
Find answers, ask questions, and share your expertise

how to set offset.storage to kafka in HDP 2.4 ?

Solved Go to solution

how to set offset.storage to kafka in HDP 2.4 ?

Hi

Almost evrything is in the title.. I want to switch from an offsets management in ZK to an offsets management in kafka. How to do this in HDP 2.4 ? I tried to add offsets.storage and offsets.storage.topic in the "custom kafka broker" and parameters are well taken into account :

[kafka@mykafka conf]$ grep offsets.storage *

server.properties:offsets.storage=kafka

server.properties:offsets.storage.topic=offset-topic

But it doesn't work... and nothing is recorded in my offset topic.

Thx in advance. Regards...

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: how to set offset.storage to kafka in HDP 2.4 ?

@Régis LE BRETONNIC there is no such property as offsets.storage on kafka broker side i.e server.properties. offsets.storage is consumer side config.

If you are using new consumer api than it defaults to storing the offsets in kafka topic but if you are using old high-level consumer api than it defaults to using zookeeper as storage. To change that make sure you add offsets.storage=kafka in consumer.properties.

View solution in original post

4 REPLIES 4
Highlighted

Re: how to set offset.storage to kafka in HDP 2.4 ?

did you try setting dual.commit.enabled to true?

Highlighted

Re: how to set offset.storage to kafka in HDP 2.4 ?

Yes I set it to true.

I made a mistake... It seems that it is offsets.storage (with S) and offset.storage.topic (w/o S)... See :http://kafka.apache.org/documentation.html

But even after having fixed this, it doesn't work. Did you achieve to us offset management within kafka ?

Regards

Highlighted

Re: how to set offset.storage to kafka in HDP 2.4 ?

@Régis LE BRETONNIC there is no such property as offsets.storage on kafka broker side i.e server.properties. offsets.storage is consumer side config.

If you are using new consumer api than it defaults to storing the offsets in kafka topic but if you are using old high-level consumer api than it defaults to using zookeeper as storage. To change that make sure you add offsets.storage=kafka in consumer.properties.

View solution in original post

Highlighted

Re: how to set offset.storage to kafka in HDP 2.4 ?

Of course you're right. I was thinking that this property was inherited from the server but it is a consumer feature. It worked as wished now... I still have an issue when I want to personalize the offset topic :

[2016-06-08 05:20:02,664] WARN Property offset.storage.topic is not valid (kafka.utils.VerifiableProperties)

I will investigate, but if I ignore this property and use the default topic __consumer_offsets it works fine.

Thk a lot for your clarification.