Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kafka installation best practices in HDF

avatar
Expert Contributor

HI,

We are going to install HDP(2.6) and HDF(2,1) in our site for now. 3 Serves in HDP and 2 servers in HDF.

Since zookeeper is not recommend to be in the same servers as Kafka and Nifi, so we are planning to have zookeeper in just HDF and Kafka in HDF servers. I have two questions regarding this architecture :

- Is the architecture the right approach to go ?

- What is the best practices for Kafka initial configuration and installationKafka.

Thanks,

SJ

1 ACCEPTED SOLUTION

avatar

Hi @Sanaz Janbakhsh I'm not sure I understand your first question, ideally the zookeeper instances will be running on separate hardware to the Kafka nodes, however it is possible to co-locate them as long as you have enough bandwidth on the nodes, and preferably dedicate spindles to ZK.

In terms of best practices overall I'd suggest you download the slides and watch the video from this session at the recent DataWorks summit Munich: Apache Kafka Best Practices

Session video: https://www.youtube.com/watch?v=maD_7ZdyuAU

Session slides: https://www.slideshare.net/HadoopSummit/apache-kafka-best-practices

I hope this helps!

View solution in original post

1 REPLY 1

avatar

Hi @Sanaz Janbakhsh I'm not sure I understand your first question, ideally the zookeeper instances will be running on separate hardware to the Kafka nodes, however it is possible to co-locate them as long as you have enough bandwidth on the nodes, and preferably dedicate spindles to ZK.

In terms of best practices overall I'd suggest you download the slides and watch the video from this session at the recent DataWorks summit Munich: Apache Kafka Best Practices

Session video: https://www.youtube.com/watch?v=maD_7ZdyuAU

Session slides: https://www.slideshare.net/HadoopSummit/apache-kafka-best-practices

I hope this helps!