Collocate Storm and Kafka

[#1]

Let's say Storm is compute intensive and Kafka IO intensive. In this case, do you suggest to collocate Storm and Kafka on the same node? I have seen Storm-Zookeeper collocation and separate Kafka

[#2 · Accepted solution]

it is best to keep them separate, here are two threads for you to review with findings from the field.

https://community.hortonworks.com/questions/10868/best-practices-for-storm-deployment-on-a-hadoop-cl...

https://community.hortonworks.com/articles/550/unofficial-storm-and-kafka-best-practices-guide.html

[#3]

Have you done any benchmarking for this purpose?

[#4]

@Ali benchmarks are based on many factors, your setup may differ from other deployments. In my experience, separating the two would give optimal performance as you can see from one of the responses in the links I provided. It really depends on your volumes, to be cost-effective sure you can colocate but when your application becomes mission critical, you will regret making those decisions.