Support Questions
Find answers, ask questions, and share your expertise

Trident (Opaque Transactional Spout): multiple topologies sourcing from same kafka topic.

So I want multiple storm topologies to read from the same kafka topic.

So suppose topic contains: A B C (data)

I want both the topologies to get A,B,C

But currently what is happening is messaged are getting distributed.(eg. topology 1 is getting A and other topology is getting B and C)

How do I achieve this is trident.

I am using OpaqueTridentKafkaSpout.

I think what needs to be done is both topologies need to specify different consumer group ids.

But could not find how to specify them in trident(TridentKafkaConfig)

1 ACCEPTED SOLUTION

Hey folks,

Turned out that I was using same name for TridentTopology#newStream in both topologies.

And it creates dir by this name in zk.

Thanks for folks on apache storm user group

View solution in original post

4 REPLIES 4

New Contributor

As per, https://svn.apache.org/repos/asf/storm/site/releases/0.10.1/javadocs/storm/kafka/trident/TridentKafk...

TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())

Hey,

@ggolani Thanks for the reply.

TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())

We have already tried this and seems not be working. The javadoc says that the 3rd param is clientId. Is this same as group id ?

Hey folks,

Turned out that I was using same name for TridentTopology#newStream in both topologies.

And it creates dir by this name in zk.

Thanks for folks on apache storm user group

Explorer

For more details about the conversation, you can go to mail thread here

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.