Created 10-16-2014 11:58 PM
Hello,
I have updated CDM and CDH to version 5.2.0 + installed the Kafka parcels via http://archive-primary.cloudera.com/cloudera-labs/kafka/parcels/latest/. However, I can't see the Kafka services in the "Add a service" menu in CDM. The parcel for Kafka is not integrated yet ? Or have I done something wrong ?
I haven't tried with uploading the jar yet to /opt/cloudera/csd. Should I put the jar only on the cloudera manager host or on every single master/nodes ? After that, I will be able to install kafka via "Add a service", right ?
Thanks.
Created 10-17-2014 06:47 AM
Hi sir,
the Parcel is vehicle to ship actual Kafka bits to your Hadoop cluster. Parcel by itself do not provide any new service in CM. You have to download the CSD and install it into CM. Please follow the linked installation instructions. By default, this is done by copying the CSD jar to /opt/cloudera/csd directory on your CM node only.
Jarcec
Created 10-17-2014 06:47 AM
Hi sir,
the Parcel is vehicle to ship actual Kafka bits to your Hadoop cluster. Parcel by itself do not provide any new service in CM. You have to download the CSD and install it into CM. Please follow the linked installation instructions. By default, this is done by copying the CSD jar to /opt/cloudera/csd directory on your CM node only.
Jarcec
Created 10-21-2014 07:51 AM
Thank you, it works now. So the CSD jar to /opt/cloudera/csd is a plugin for cloudera manager and the parcels is the installation need to be deployed on nodes like the others parcels services.
When I'm running some Kafka sample, I had to change chroot path in CDM because the quickstart at http://kafka.apache.org/documentation.html#quickstart expected it to be in / instead of /kafka in zookeeper path else you'll have this :
[/opt/cloudera/parcels/CLABS_KAFKA/lib/kafka/bin]# kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic toto Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:480) at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:81) at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:154) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86) at kafka.admin.TopicCommand$.main(TopicCommand.scala:50) at kafka.admin.TopicCommand.main(TopicCommand.scala) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496) at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ... 8 more
Created 10-21-2014 08:49 AM
That is correct, Cloudera default configuration will use zookeeper chroot to "/kafka", so that we are not putting a lot of random znodes directly to the root. Adding "/kafka" to zookeeper URL in the examples should be sufficient to get them running.
Created 10-21-2014 08:59 AM
While changing chroot is not something we recommend, it is possible to change it to "/" instead of "/kafka".
Make sure you restart the brokers after making the change so they can register in the "/" directory.
Created 11-06-2014 10:43 AM
I've already copied the jar file to /opt/cloudera/csd and followed the links available "Add on Services", "Parcels" at Cloudera's site. CLABS_KAFKA 0.8.1.1-1.kafka1.0.0.p0.14 is activated, but the Kfaka still does not appear at "Add Service" in the Cloudera Manager. I also have CDM and CDH 5.2 running OK !!!
Anyone to give me a hand ?
Best Regards
Created 11-06-2014 10:56 AM
Did you restart Cloudera Manager Service in CM UI?
Created 11-06-2014 03:56 PM
First of all: Thks for your quick response !!!
I've already restarted CM, but i'll try again and later, i'll post here !!!
Created 11-07-2014 04:06 AM
Tks a lot !!!
Now it appears to me ... "Kafka-Cloudera-Labs" !!!
Again
Best Regards
Created 07-18-2017 11:59 PM
I am using CDH5.10, and I want to activate kafka Parcels by CM,But failed.The CM said:incompatible cdh version, at least 5.2+,I was very confused for my cdh version is 5.10,can you help me?