question Re: Trident (Opaque Transactional Spout): multiple topologies sourcing from same kafka topic. in Archives of Support Questions (Read Only)

Trident (Opaque Transactional Spout): multiple topologies sourcing from same kafka topic.

amber_kulkarni8 — Tue, 26 Jul 2016 18:03:45 GMT

So I want multiple storm topologies to read from the same kafka topic.

So suppose topic contains: A B C (data)

I want both the topologies to get A,B,C

But currently what is happening is messaged are getting distributed.(eg. topology 1 is getting A and other topology is getting B and C)

How do I achieve this is trident.

I am using OpaqueTridentKafkaSpout.

I think what needs to be done is both topologies need to specify different consumer group ids.

But could not find how to specify them in trident(TridentKafkaConfig)

ggolani — Tue, 26 Jul 2016 22:14:29 GMT

TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())

amber_kulkarni8 — Wed, 27 Jul 2016 13:18:54 GMT

Hey,

@ggolani Thanks for the reply.

TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())

We have already tried this and seems not be working. The javadoc says that the 3rd param is clientId. Is this same as group id ?

amber_kulkarni8 — Wed, 27 Jul 2016 17:29:28 GMT

Hey folks,

Turned out that I was using same name for TridentTopology#newStream in both topologies.

And it creates dir by this name in zk.

Thanks for folks on apache storm user group

sduggana — Thu, 28 Jul 2016 12:46:53 GMT

For more details about the conversation, you can go to mail thread here