When you kill the topology you can specify a time that storm should wait before removing the topology completely. The default value for this wait time is 30 second. Can you verify that you are actually seeing the topology even after 30 seconds have elapsed or if you specify wait time explicitly , are you seeing the topology even after that time has elapsed? ... View more

No way to read based on timestamp right now, there is a discussion in the community https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message ... View more

Storm does not support the storm-rabbitmq spout that you mentioning, it does not ship with apache storm distro so I have not looked at the code. Here is a JMS spout that you may be able to use as is with RabitMQ if RAbitMQ supports JMS https://github.com/hortonworks/storm/tree/2.3-mai... (Note that this is available in HWX repo but not under apache) For an example of OpaqueSpout take a look at kafka-spout. https://github.com/apache/storm/blob/master/extern... The unit test part for exactly once is kind of hard in my opinion and there are no current examples. I am not sure if I understand the sharing queue's question completely. Your spout could read from N queues while outputting all the messages to a single stream, that is completely up to your spout. The merge can be useful if you want to create one spout -> one queue mapping and then merge all the output streams into a single stream that other processing units (trident states) subscribe to. ... View more

No way to kerberize Kafka in 2.2 If you are running storm-2.2 you should still be able to run a topology that uses storm-kafka connector's version from version 2.3 which should be able to read from a secure kafka cluster (HDP or not HDP). ... View more

You must specify "hdfs.storm.dir" in the topology config which should map to a hdfs directory where these files are stored. In addition hdfs-site.xml and core-site.xml must be part of nimbus's classpath. By default the hdfs jars are not shipped with storm so the user will have to set this up on all nimbus hosts and add these jars in nimbus's classpath using nimbus.childopts. In general it is better for now to use the default code distributor as HDFSCodeDistributor is written as an example and it is a class that will probably still evolve in near future. ... View more