Support Questions
Find answers, ask questions, and share your expertise

How to install a Ambari managed Kafka client

Super Collaborator

Hi,

Ambari has no option to install a 'Kafka Client'. So if you want to have the Kafka shell scripts kafka-topics.sh etc plus Ambari managed kafka configs there seems to be no other option then to install a full Kafka broker on the node.

Is there a way to have a Kafka client or broker that is fully synced with Kafka config by Ambari, but without it ever receiving Kafka topics on disk?

1 ACCEPTED SOLUTION

Well, no. The best you can do is to install Kafka on your node (yum install kafka), and copy /etc/kafka/conf files from one of your brokers. Then whenever you change configuration of your Kafka you need to update conf files on your node. If that doesn't happen often you can do it manually, or you can configure rsync to do it for you.

View solution in original post

6 REPLIES 6

Explorer

I think that is not possible at the moment.

Well, no. The best you can do is to install Kafka on your node (yum install kafka), and copy /etc/kafka/conf files from one of your brokers. Then whenever you change configuration of your Kafka you need to update conf files on your node. If that doesn't happen often you can do it manually, or you can configure rsync to do it for you.

Super Collaborator

@Predrag Minovic @Balint Molnar thanks for your input.

I thought of the option to do the minimal (manual) kafka install on the node, but the overhead to manually sync the kafka config from there onwards did not look that appealing to me.

So instead, I installed a Ambari managed Kafka broker on the node. I prevent that broker from ever having any topics by introducing a special CG for that node with auto.create.topics.enable set to 'false'.

auto.create.topics.enable=true means that Kafka will create a topic automatically when you send messages to a non existing topic. So, in production Kafka you usually set that to false. It doesn't mean that your Kafka instance won't store any topics. And even if you can achieve that somehow, your broker will be running in vain, wasting resources. If you want automatism, I'd go with rsync. You can call it a primitive solution but works great.

Explorer

why to install a broker?just yum install kafka and rest of the configuration will be managed by command line/customer conf files...you donot need any configuration for client

Contributor

Is there a way to setup a Kafka container without ambari and then add kafka broker to ambari ?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.