Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to install a Ambari managed Kafka client

SOLVED Go to solution
Highlighted

How to install a Ambari managed Kafka client

Super Collaborator

Hi,

Ambari has no option to install a 'Kafka Client'. So if you want to have the Kafka shell scripts kafka-topics.sh etc plus Ambari managed kafka configs there seems to be no other option then to install a full Kafka broker on the node.

Is there a way to have a Kafka client or broker that is fully synced with Kafka config by Ambari, but without it ever receiving Kafka topics on disk?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to install a Ambari managed Kafka client

Well, no. The best you can do is to install Kafka on your node (yum install kafka), and copy /etc/kafka/conf files from one of your brokers. Then whenever you change configuration of your Kafka you need to update conf files on your node. If that doesn't happen often you can do it manually, or you can configure rsync to do it for you.

5 REPLIES 5

Re: How to install a Ambari managed Kafka client

New Contributor

I think that is not possible at the moment.

Re: How to install a Ambari managed Kafka client

Well, no. The best you can do is to install Kafka on your node (yum install kafka), and copy /etc/kafka/conf files from one of your brokers. Then whenever you change configuration of your Kafka you need to update conf files on your node. If that doesn't happen often you can do it manually, or you can configure rsync to do it for you.

Re: How to install a Ambari managed Kafka client

Super Collaborator

@Predrag Minovic @Balint Molnar thanks for your input.

I thought of the option to do the minimal (manual) kafka install on the node, but the overhead to manually sync the kafka config from there onwards did not look that appealing to me.

So instead, I installed a Ambari managed Kafka broker on the node. I prevent that broker from ever having any topics by introducing a special CG for that node with auto.create.topics.enable set to 'false'.

Re: How to install a Ambari managed Kafka client

auto.create.topics.enable=true means that Kafka will create a topic automatically when you send messages to a non existing topic. So, in production Kafka you usually set that to false. It doesn't mean that your Kafka instance won't store any topics. And even if you can achieve that somehow, your broker will be running in vain, wasting resources. If you want automatism, I'd go with rsync. You can call it a primitive solution but works great.

Re: How to install a Ambari managed Kafka client

New Contributor

why to install a broker?just yum install kafka and rest of the configuration will be managed by command line/customer conf files...you donot need any configuration for client