question Re: Collocate Storm and Kafka in Support Questions

Collocate Storm and Kafka

alinazemian — Mon, 13 Mar 2017 19:03:27 GMT

Let's say Storm is compute intensive and Kafka IO intensive. In this case, do you suggest to collocate Storm and Kafka on the same node? I have seen Storm-Zookeeper collocation and separate Kafka

Re: Collocate Storm and Kafka

aervits — Mon, 13 Mar 2017 19:44:44 GMT

it is best to keep them separate, here are two threads for you to review with findings from the field.

https://community.hortonworks.com/questions/10868/best-practices-for-storm-deployment-on-a-hadoop-cl.html

https://community.hortonworks.com/articles/550/unofficial-storm-and-kafka-best-practices-guide.html

Re: Collocate Storm and Kafka

alinazemian — Mon, 13 Mar 2017 19:53:20 GMT

Have you done any benchmarking for this purpose?

Re: Collocate Storm and Kafka

aervits — Mon, 13 Mar 2017 21:17:45 GMT

@Ali benchmarks are based on many factors, your setup may differ from other deployments. In my experience, separating the two would give optimal performance as you can see from one of the responses in the links I provided. It really depends on your volumes, to be cost-effective sure you can colocate but when your application becomes mission critical, you will regret making those decisions.