How to install a Ambari managed Kafka client

[#1]

Hi,

Ambari has no option to install a 'Kafka Client'. So if you want to have the Kafka shell scripts kafka-topics.sh etc plus Ambari managed kafka configs there seems to be no other option then to install a full Kafka broker on the node.

Is there a way to have a Kafka client or broker that is fully synced with Kafka config by Ambari, but without it ever receiving Kafka topics on disk?

[#2]

I think that is not possible at the moment.

[#3 · Accepted solution]

Well, no. The best you can do is to install Kafka on your node (yum install kafka), and copy /etc/kafka/conf files from one of your brokers. Then whenever you change configuration of your Kafka you need to update conf files on your node. If that doesn't happen often you can do it manually, or you can configure rsync to do it for you.

[#4]

@Predrag Minovic @Balint Molnar thanks for your input.

I thought of the option to do the minimal (manual) kafka install on the node, but the overhead to manually sync the kafka config from there onwards did not look that appealing to me.

So instead, I installed a Ambari managed Kafka broker on the node. I prevent that broker from ever having any topics by introducing a special CG for that node with auto.create.topics.enable set to 'false'.

[#5]

auto.create.topics.enable=true means that Kafka will create a topic automatically when you send messages to a non existing topic. So, in production Kafka you usually set that to false. It doesn't mean that your Kafka instance won't store any topics. And even if you can achieve that somehow, your broker will be running in vain, wasting resources. If you want automatism, I'd go with rsync. You can call it a primitive solution but works great.

[#6]

why to install a broker?just yum install kafka and rest of the configuration will be managed by command line/customer conf files...you donot need any configuration for client

[#7]

Is there a way to setup a Kafka container without ambari and then add kafka broker to ambari ?