Cloudera Labs
Provide feedback on Cloudera Labs
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka Parcels

Solved Go to solution

Kafka Parcels

New Contributor

Hello,

 

I have updated CDM and CDH to version 5.2.0 + installed the Kafka parcels via http://archive-primary.cloudera.com/cloudera-labs/kafka/parcels/latest/. However, I can't see the Kafka services in the "Add a service" menu in CDM. The parcel for Kafka is not integrated yet ? Or have I done something wrong ?

 

I haven't tried with uploading the jar yet to /opt/cloudera/csd. Should I put the jar only on the cloudera manager host or on every single master/nodes ? After that, I will be able to install kafka via "Add a service", right ?

 

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Kafka Parcels

Cloudera Employee

Hi sir,

the Parcel is vehicle to ship actual Kafka bits to your Hadoop cluster. Parcel by itself do not provide any new service in CM. You have to download the CSD and install it into CM. Please follow the linked installation instructions. By default, this is done by copying the CSD jar to /opt/cloudera/csd directory on your CM node only. 

 

Jarcec

9 REPLIES 9

Re: Kafka Parcels

Cloudera Employee

Hi sir,

the Parcel is vehicle to ship actual Kafka bits to your Hadoop cluster. Parcel by itself do not provide any new service in CM. You have to download the CSD and install it into CM. Please follow the linked installation instructions. By default, this is done by copying the CSD jar to /opt/cloudera/csd directory on your CM node only. 

 

Jarcec

Re: Kafka Parcels

New Contributor

Thank you, it works now. So the CSD jar to /opt/cloudera/csd is a plugin for cloudera manager and the parcels is the installation need to be deployed on nodes like the others parcels services. 

 

When I'm running some Kafka sample, I had to change chroot path in CDM because the quickstart at http://kafka.apache.org/documentation.html#quickstart expected it to be in / instead of /kafka in zookeeper path else you'll have this :

 

[/opt/cloudera/parcels/CLABS_KAFKA/lib/kafka/bin]# kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic toto
Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
	at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409)
	at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:480)
	at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:81)
	at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:154)
	at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496)
	at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99)
	at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416)
	at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
	... 8 more

 

Re: Kafka Parcels

Cloudera Employee

That is correct, Cloudera default configuration will use zookeeper chroot to "/kafka", so that we are not putting a lot of random znodes directly to the root. Adding "/kafka" to zookeeper URL in the examples should be sufficient to get them running.

Re: Kafka Parcels

Contributor

While changing chroot is not something we recommend, it is possible to change it to "/" instead of "/kafka".

Make sure you restart the brokers after making the change so they can register in the "/" directory. 

Re: Kafka Parcels

Explorer

I've already copied the jar file to /opt/cloudera/csd and followed the links available "Add on Services", "Parcels" at Cloudera's site. CLABS_KAFKA 0.8.1.1-1.kafka1.0.0.p0.14 is activated, but the Kfaka still does not appear at "Add Service" in the Cloudera Manager. I also have CDM and CDH 5.2 running OK !!!

Anyone to give me a hand ?

Best Regards

Re: Kafka Parcels

Contributor

Did you restart Cloudera Manager Service in CM UI?

Re: Kafka Parcels

Explorer

First of all: Thks for your quick response !!!

I've already restarted CM, but i'll try again and later, i'll post here !!! 

Re: Kafka Parcels

Explorer

Tks a lot !!!

Now it appears to me ... "Kafka-Cloudera-Labs" !!!

Again

Best Regards

Highlighted

Re: Kafka Parcels

New Contributor

I am using CDH5.10, and I want to activate kafka Parcels by CM,But failed.The CM said:incompatible cdh version, at least 5.2+,I was very confused for my cdh version is 5.10,can you help me?

Don't have an account?
Coming from Hortonworks? Activate your account here