Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

can kafka multiple nodes work by itself without any cluster?

avatar
Expert Contributor

my project need to stream data from kafka to mongodb, so we did setup HDP cluster and kafka multiple nodes within.

After all, I feel we may process all data by using kafka/zookeeper alone, no need to have cluster. any one can tell if kafka stream can work itself without in a cluster?

if yes, anythine I need to aware?

1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

Thank you very much Wang for confirm the zk and kf and work alone.

yes, with HDP, kafka and zookeeper is better administrated and monitored. I did setup setup kafka cluster and mongodb with HDP, it seemed very easy steps.

However, I tried to save some money for our company, so I came up with this question. thank you for confirm it.

avatar
Contributor

@Robin Dong

The HDP is also free, and ambari agent don't consume much resource.

Feel free to use it.

avatar
Expert Contributor

yes, you are right HDP is license free. however the cluster install needs master and slave infrastructure, so all the HDFS and data name, name node, yarn alone with some mandatory like Hive, pig, tez need installed and maintained.

All of these cost a lots on AWS/EC2.

in other hand, without HDP cluster, the monitoring, upgrade, version compatiblility and security on kafka, spark, zookeeper may bring issues in long run. admin have to deal with it.

I am looking for use case of zookeeper/kafka cluster and spark to compare to see if it is worth to do so.

How do you think?

avatar
Contributor

@Robin Dong

That why I suggest you to use HDF, which you can only install zookeeper and kafka.

avatar
Expert Contributor

Thank you so much.