- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Trident (Opaque Transactional Spout): multiple topologies sourcing from same kafka topic.
- Labels:
-
Apache Kafka
-
Apache Storm
Created ‎07-26-2016 11:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So I want multiple storm topologies to read from the same kafka topic.
So suppose topic contains: A B C (data)
I want both the topologies to get A,B,C
But currently what is happening is messaged are getting distributed.(eg. topology 1 is getting A and other topology is getting B and C)
How do I achieve this is trident.
I am using OpaqueTridentKafkaSpout.
I think what needs to be done is both topologies need to specify different consumer group ids.
But could not find how to specify them in trident(TridentKafkaConfig)
Created ‎07-27-2016 10:29 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey folks,
Turned out that I was using same name for TridentTopology#newStream in both topologies.
And it creates dir by this name in zk.
Thanks for folks on apache storm user group
Created ‎07-26-2016 03:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())
Created ‎07-27-2016 06:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey,
@ggolani Thanks for the reply.
TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "test-topic",UUID.randomUUID().toString())
We have already tried this and seems not be working. The javadoc says that the 3rd param is clientId. Is this same as group id ?
Created ‎07-27-2016 10:29 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey folks,
Turned out that I was using same name for TridentTopology#newStream in both topologies.
And it creates dir by this name in zk.
Thanks for folks on apache storm user group
Created ‎07-28-2016 05:46 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
For more details about the conversation, you can go to mail thread here
