question Re: Storm and Kafka Development Cycle - Best Practices in Archives of Support Questions (Read Only)

Storm and Kafka Development Cycle - Best Practices

wfloyd — Thu, 24 Sep 2015 04:19:47 GMT

Wanted to ask what process our Storm and Kafka developers & testers follow when they’re developing new topologies? The process of building a Storm/Kafka topology on laptop, copying files to cluster or sandbox, and testing can be time consuming and difficult to iterate quickly. Do you have any tips you could pass on for efficient development process:

Do you develop on the same machine (linux or mac) which is also running your Storm/Kafka cluster?
Do you instead rely on JUnit tests to instantiate and test Storm bolts on their own and manually trigger Tuples through the execute() method (example here)?

Thanks for any tips you can pass along to help speed up our development.

Re: Storm and Kafka Development Cycle - Best Practices

schintalapani — Thu, 24 Sep 2015 04:25:21 GMT

Its fairly easy to setup kafka on local machine.

Download kafka from here https://kafka.apache.org/downloads.html.

Unzip kafka_2.10-0.8.2.0.tgz

run ./bin/zookeeper-server-start.sh config/zookeeper.properties

run ./bin/kafka-server-start.sh config/server.properties

once you've kafka up and running you can use storm LocalCluster to deploy and test your topology. More details on LocalCluster https://storm.apache.org/documentation/Local-mode.html .

Re: Storm and Kafka Development Cycle - Best Practices

andrewg — Thu, 24 Sep 2015 04:28:34 GMT

Doesn't local storm instance not work for them? E.g. most of online examples check to see if topology is being launched in the same JVM or a distributed cluster.

One of my customers has built a web utility to be able to remotely submit topology jar to the cluster. I'd imagine it's easy to build with any REST framework.

This might be a good idea to suggest

Re: Storm and Kafka Development Cycle - Best Practices

gvetticaden1 — Thu, 24 Sep 2015 04:37:50 GMT

I have used and built some testing utilities to be able to create unit tests and integration tests of topologies that use kafka, solr and other components. These provide the capabilities to deploy a topology from your desktop machine to your dev cluster, etc..

Check out the following github links for examples:

The test utilities it uses can be found here:

Also check out the Hortonworks Gallery, it has a cool project that is being featured called Mini Clusters by @skumpf@hortonworks.com that has some great testing utilities to run clusters in local mode.

Re: Storm and Kafka Development Cycle - Best Practices

gvetticaden1 — Fri, 25 Sep 2015 19:56:37 GMT

I plan to move these this from my github to the Gallery

Re: Storm and Kafka Development Cycle - Best Practices

wsalazar — Sat, 26 Sep 2015 00:30:43 GMT

I have had success setting up an Eclipse project for Storm/Kafka and using the tools there to semi automate the build and deployment. Anyone else tried that?

Re: Storm and Kafka Development Cycle - Best Practices

pwendorf — Tue, 29 Sep 2015 22:24:37 GMT

I found using some simple ssh scripts made it quicker for me:

mvn package

ssh root@storm-server <<'ENDSSH'
storm kill <current topology name>
ENDSSH

scp target/<topology>.jar root@<storm server>:
scp src/main/resources/config.properties root@<storm server>:


ssh root@<storm server> <<'ENDSSH'
storm jar <topology>.jar <main class> config.properties
ENDSSH

Re: Storm and Kafka Development Cycle - Best Practices

aervits — Mon, 05 Oct 2015 19:55:46 GMT

using localcluster utility from storm works pretty well. I also used examples from https://github.com/nathanmarz/storm-starter/blob/master/test/jvm/storm/starter/bolt/RollingCountBoltTest.java for my own unit tests. MockTuple is the way to go