Trident (Opaque Transactional Spout): multiple topologies sourcing from same kafka topic.

[#1]

So I want multiple storm topologies to read from the same kafka topic.

So suppose topic contains: A B C (data)

I want both the topologies to get A,B,C

But currently what is happening is messaged are getting distributed.(eg. topology 1 is getting A and other topology is getting B and C)

How do I achieve this is trident.

I am using OpaqueTridentKafkaSpout.

I think what needs to be done is both topologies need to specify different consumer group ids.

But could not find how to specify them in trident(TridentKafkaConfig)

[#2]

As per, https://svn.apache.org/repos/asf/storm/site/releases/0.10.1/javadocs/storm/kafka/trident/TridentKafk...

TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())

[#3]

Hey,

@ggolani Thanks for the reply.

TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())

We have already tried this and seems not be working. The javadoc says that the 3rd param is clientId. Is this same as group id ?

[#4 · Accepted solution]

Hey folks,

Turned out that I was using same name for TridentTopology#newStream in both topologies.

And it creates dir by this name in zk.

Thanks for folks on apache storm user group

[#5]

For more details about the conversation, you can go to mail thread here