Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Read from sharded rabbitmq and write to partitioned kafka with only once semantics.

Solved Go to solution
Highlighted

Read from sharded rabbitmq and write to partitioned kafka with only once semantics.

New Contributor

Problem : Read from sharded rabbitmq and write to kafka with only once semantics.

==> I am planning to use Trident for the same.

  • I have used RabbitMq spout (storm) which has implemented BaseRichSpout (https://github.com/ppat/storm-rabbitmq)
    1. I can see that it is a non transactional spout, and hence "would not" support only once semantics of Trident. Am I correct?
  • If I need to build an OpaqueTransactionalSpout for Rabbitmq, Can you give me some hints on how to start or some examples to look into
  • I wanted to write unit test to check if my Trident Topology supports "Exactly once semantic". Are there any sample Unit tests/examples. The reason I want to write the tests because "Exactly once semantics" depend on Spout also not just on Trident or Storm with my understanding.
  • I have to support sharded Rabbitmq and hence how to connect to more than one queue. I am thinking of using Stream.merge(List<Streams>) after creating once stream for each queue, Is this good idea to do?. Let me know your thoughts on the same.
  • https://nathanmarz.github.io/storm/doc/storm/tride...
  • @Ram Sriharsha

    1 ACCEPTED SOLUTION

    Accepted Solutions

    Re: Read from sharded rabbitmq and write to partitioned kafka with only once semantics.

    New Contributor
    • Storm does not support the storm-rabbitmq spout that you mentioning, it does not ship with apache storm distro so I have not looked at the code.
    • Here is a JMS spout that you may be able to use as is with RabitMQ if RAbitMQ supports JMS https://github.com/hortonworks/storm/tree/2.3-mai... (Note that this is available in HWX repo but not under apache)
    • For an example of OpaqueSpout take a look at kafka-spout. https://github.com/apache/storm/blob/master/extern...
    • The unit test part for exactly once is kind of hard in my opinion and there are no current examples.
    • I am not sure if I understand the sharing queue's question completely. Your spout could read from N queues while outputting all the messages to a single stream, that is completely up to your spout. The merge can be useful if you want to create one spout -> one queue mapping and then merge all the output streams into a single stream that other processing units (trident states) subscribe to.
    2 REPLIES 2

    Re: Read from sharded rabbitmq and write to partitioned kafka with only once semantics.

    New Contributor
    • Storm does not support the storm-rabbitmq spout that you mentioning, it does not ship with apache storm distro so I have not looked at the code.
    • Here is a JMS spout that you may be able to use as is with RabitMQ if RAbitMQ supports JMS https://github.com/hortonworks/storm/tree/2.3-mai... (Note that this is available in HWX repo but not under apache)
    • For an example of OpaqueSpout take a look at kafka-spout. https://github.com/apache/storm/blob/master/extern...
    • The unit test part for exactly once is kind of hard in my opinion and there are no current examples.
    • I am not sure if I understand the sharing queue's question completely. Your spout could read from N queues while outputting all the messages to a single stream, that is completely up to your spout. The merge can be useful if you want to create one spout -> one queue mapping and then merge all the output streams into a single stream that other processing units (trident states) subscribe to.

    Re: Read from sharded rabbitmq and write to partitioned kafka with only once semantics.

    New Contributor
    • Out of the box Rabbitmq can support a single delivery of a message (Exactly Once), it really depends on how the RabbitMQ routing topology has been created. For instance, if you have a single exchange and (direct) queue then it's impossible for another queue to receive the message.
    • Instead merging streams, you may want to use RabbitMQ to do this, of course it depends on your RabbitMQ topology once again.