Kafka Parcels

[nphung]

Hello,

I have updated CDM and CDH to version 5.2.0 + installed the Kafka parcels via http://archive-primary.cloudera.com/cloudera-labs/kafka/parcels/latest/. However, I can't see the Kafka services in the "Add a service" menu in CDM. The parcel for Kafka is not integrated yet ? Or have I done something wrong ?

I haven't tried with uploading the jar yet to /opt/cloudera/csd. Should I put the jar only on the cloudera manager host or on every single master/nodes ? After that, I will be able to install kafka via "Add a service", right ?

Thanks.

[jarcec · Accepted solution]

Hi sir,

the Parcel is vehicle to ship actual Kafka bits to your Hadoop cluster. Parcel by itself do not provide any new service in CM. You have to download the CSD and install it into CM. Please follow the linked installation instructions. By default, this is done by copying the CSD jar to /opt/cloudera/csd directory on your CM node only. 

Jarcec

[nphung]

Thank you, it works now. So the CSD jar to /opt/cloudera/csd is a plugin for cloudera manager and the parcels is the installation need to be deployed on nodes like the others parcels services. 

When I'm running some Kafka sample, I had to change chroot path in CDM because the quickstart at http://kafka.apache.org/documentation.html#quickstart expected it to be in / instead of /kafka in zookeeper path else you'll have this :

[/opt/cloudera/parcels/CLABS_KAFKA/lib/kafka/bin]# kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic toto
Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
	at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409)
	at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:480)
	at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:81)
	at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:154)
	at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496)
	at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99)
	at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416)
	at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
	... 8 more

[jarcec]

That is correct, Cloudera default configuration will use zookeeper chroot to "/kafka", so that we are not putting a lot of random znodes directly to the root. Adding "/kafka" to zookeeper URL in the examples should be sufficient to get them running.

[gshapira]

While changing chroot is not something we recommend, it is possible to change it to "/" instead of "/kafka".

Make sure you restart the brokers after making the change so they can register in the "/" directory. 

[dinoamaral]

I've already copied the jar file to /opt/cloudera/csd and followed the links available "Add on Services", "Parcels" at Cloudera's site. CLABS_KAFKA 0.8.1.1-1.kafka1.0.0.p0.14 is activated, but the Kfaka still does not appear at "Add Service" in the Cloudera Manager. I also have CDM and CDH 5.2 running OK !!!

Anyone to give me a hand ?

Best Regards

[gshapira]

Did you restart Cloudera Manager Service in CM UI?

[dinoamaral]

First of all: Thks for your quick response !!!

I've already restarted CM, but i'll try again and later, i'll post here !!! 

[dinoamaral]

Tks a lot !!!

Now it appears to me ... "Kafka-Cloudera-Labs" !!!

Again

Best Regards

[NeoHwang]

I am using CDH5.10, and I want to activate kafka Parcels by CM,But failed.The CM said:incompatible cdh version, at least 5.2+，I was very confused for my cdh version is 5.10,can you help me?