Reply
New Contributor
Posts: 4
Registered: ‎09-24-2014
Accepted Solution

Kafka Parcels

Hello,

 

I have updated CDM and CDH to version 5.2.0 + installed the Kafka parcels via http://archive-primary.cloudera.com/cloudera-labs/kafka/parcels/latest/. However, I can't see the Kafka services in the "Add a service" menu in CDM. The parcel for Kafka is not integrated yet ? Or have I done something wrong ?

 

I haven't tried with uploading the jar yet to /opt/cloudera/csd. Should I put the jar only on the cloudera manager host or on every single master/nodes ? After that, I will be able to install kafka via "Add a service", right ?

 

Thanks.

Cloudera Employee
Posts: 20
Registered: ‎07-08-2013

Re: Kafka Parcels

Hi sir,

the Parcel is vehicle to ship actual Kafka bits to your Hadoop cluster. Parcel by itself do not provide any new service in CM. You have to download the CSD and install it into CM. Please follow the linked installation instructions. By default, this is done by copying the CSD jar to /opt/cloudera/csd directory on your CM node only. 

 

Jarcec

New Contributor
Posts: 4
Registered: ‎09-24-2014

Re: Kafka Parcels

Thank you, it works now. So the CSD jar to /opt/cloudera/csd is a plugin for cloudera manager and the parcels is the installation need to be deployed on nodes like the others parcels services. 

 

When I'm running some Kafka sample, I had to change chroot path in CDM because the quickstart at http://kafka.apache.org/documentation.html#quickstart expected it to be in / instead of /kafka in zookeeper path else you'll have this :

 

[/opt/cloudera/parcels/CLABS_KAFKA/lib/kafka/bin]# kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic toto
Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
	at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413)
	at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409)
	at kafka.utils.ZkUtils$.getChildren(ZkUtils.scala:480)
	at kafka.utils.ZkUtils$.getSortedBrokerList(ZkUtils.scala:81)
	at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:154)
	at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:86)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:50)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496)
	at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:99)
	at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416)
	at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
	... 8 more

 

Cloudera Employee
Posts: 20
Registered: ‎07-08-2013

Re: Kafka Parcels

That is correct, Cloudera default configuration will use zookeeper chroot to "/kafka", so that we are not putting a lot of random znodes directly to the root. Adding "/kafka" to zookeeper URL in the examples should be sufficient to get them running.

Cloudera Employee
Posts: 31
Registered: ‎10-16-2013

Re: Kafka Parcels

While changing chroot is not something we recommend, it is possible to change it to "/" instead of "/kafka".

Make sure you restart the brokers after making the change so they can register in the "/" directory. 

Explorer
Posts: 23
Registered: ‎02-21-2014

Re: Kafka Parcels

I've already copied the jar file to /opt/cloudera/csd and followed the links available "Add on Services", "Parcels" at Cloudera's site. CLABS_KAFKA 0.8.1.1-1.kafka1.0.0.p0.14 is activated, but the Kfaka still does not appear at "Add Service" in the Cloudera Manager. I also have CDM and CDH 5.2 running OK !!!

Anyone to give me a hand ?

Best Regards

Cloudera Employee
Posts: 31
Registered: ‎10-16-2013

Re: Kafka Parcels

Did you restart Cloudera Manager Service in CM UI?

Explorer
Posts: 23
Registered: ‎02-21-2014

Re: Kafka Parcels

First of all: Thks for your quick response !!!

I've already restarted CM, but i'll try again and later, i'll post here !!! 

Explorer
Posts: 23
Registered: ‎02-21-2014

Re: Kafka Parcels

Tks a lot !!!

Now it appears to me ... "Kafka-Cloudera-Labs" !!!

Again

Best Regards

Highlighted
New Contributor
Posts: 2
Registered: ‎07-18-2017

Re: Kafka Parcels

I am using CDH5.10, and I want to activate kafka Parcels by CM,But failed.The CM said:incompatible cdh version, at least 5.2+,I was very confused for my cdh version is 5.10,can you help me?

Announcements

Currently incubating in Cloudera Labs:

Envelope
HTrace
Ibis
Impyla
Livy
Oryx
Phoenix
Spark Runner for Beam SDK
Time Series for Spark
YCSB