question Re: how to set offset.storage to kafka in HDP 2.4 ? in Archives of Support Questions (Read Only)

how to set offset.storage to kafka in HDP 2.4 ?

rlebretonnic — Tue, 07 Jun 2016 22:43:22 GMT

Almost evrything is in the title.. I want to switch from an offsets management in ZK to an offsets management in kafka. How to do this in HDP 2.4 ? I tried to add offsets.storage and offsets.storage.topic in the "custom kafka broker" and parameters are well taken into account :

[kafka@mykafka conf]$ grep offsets.storage *

server.properties:offsets.storage=kafka

server.properties:offsets.storage.topic=offset-topic

But it doesn't work... and nothing is recorded in my offset topic.

Thx in advance. Regards...

Re: how to set offset.storage to kafka in HDP 2.4 ?

rajkumar_singh — Tue, 07 Jun 2016 23:22:50 GMT

did you try setting dual.commit.enabled to true?

Re: how to set offset.storage to kafka in HDP 2.4 ?

rlebretonnic — Tue, 07 Jun 2016 23:25:48 GMT

Yes I set it to true.

I made a mistake... It seems that it is offsets.storage (with S) and offset.storage.topic (w/o S)... See :http://kafka.apache.org/documentation.html

But even after having fixed this, it doesn't work. Did you achieve to us offset management within kafka ?

Regards

Re: how to set offset.storage to kafka in HDP 2.4 ?

schintalapani — Wed, 08 Jun 2016 01:26:25 GMT

@Régis LE BRETONNIC there is no such property as offsets.storage on kafka broker side i.e server.properties. offsets.storage is consumer side config.

If you are using new consumer api than it defaults to storing the offsets in kafka topic but if you are using old high-level consumer api than it defaults to using zookeeper as storage. To change that make sure you add offsets.storage=kafka in consumer.properties.

Re: how to set offset.storage to kafka in HDP 2.4 ?

rlebretonnic — Wed, 08 Jun 2016 15:00:37 GMT

Of course you're right. I was thinking that this property was inherited from the server but it is a consumer feature. It worked as wished now... I still have an issue when I want to personalize the offset topic :

[2016-06-08 05:20:02,664] WARN Property offset.storage.topic is not valid (kafka.utils.VerifiableProperties)

I will investigate, but if I ignore this property and use the default topic __consumer_offsets it works fine.

Thk a lot for your clarification.