Support Questions

Find answers, ask questions, and share your expertise

adding a Kafka service failed

Explorer

Dear All:

 

We'd like to add a Kafka service on cloudera manager failed, here's the step to reproduce:

1. install Kafka parcel and active it successfully

2. adding a service from cloudera manager failed , on the destination broker list and source broker list:

  • Missing required value: Destination Broker List
  • Missing required value: Source Broker List

Please take a look at the attached , any advice would be appreciated

 

Regards,

JoeactKafka.gifKafkaError.gif

 

1 ACCEPTED SOLUTION

Rising Star

Hi, please perform the following actions:

 

1) Fill in "Source Brokers List" --> "nameOfClouderaKafkaBrokerServer.yourdomain.com:9092". This is the Server (or Servers) where you configured the Kafka Broker (NOT the MirrorMaker).

 

2) Fill in "Destination Brokers List" --> "nameOfRemoteBrokerServer.otherdomain.com:9092". This is supposed to be a remote Cluster that will receive Topics sent over by your MirrorMaker. If you have one, put in that one. Otherwise just put in another Server in your network, whatever Server.

 

Please note that both this Server Names must be FQDN and resolvable by your DNS (or hosts file), otherwise you'll get other errors. Also the format with the trailing Port Number is mandatory!

 

3) Click "Continue". Service will NOT start (error). Do not navigate away from that screen

 

4) Open another Cloudera Manager in another browser pane. You should now see "Kafka" in the list of Services (red, but it should be there). Click on the Kafka Service and then "Configure".

 

5) Search for the "java heap space" Configuration Property. The standard Java Heap Space you'll find already set up should be 50 MBytes. Put in at least 256 MBytes. The original value is simply not enough.

 

6) Now search for the "whitelist" Configuration Property. In the field, put in "(?!x)x" (without the quotation marks). That's a regular expression that does not match anything. Given that apparently a Whitelist is mandatory for the Mirrormaker Service to start, and I'm assuming you don't want to replicate any topics remotely right now, just put in something that won't replicate anything e.g. that regular expression.

 

7) Save the changes and go back to the original Configuration Screen on the othe browser pane. Click "Retry", or wathever, or even exit that screen and manually restart the Kafka Service in Cloudera Manager.

 

That should work, at least it did for me! HTH

View solution in original post

19 REPLIES 19

Explorer

Hi joeyang,

Which Kafka version is it? (1.4 or 2.0)
Does the parcel tab show kafka service as activated?
While adding the service it just asks for broker instances and mirrormaker is optional. 
At what stage do you get the error?

Explorer

Hi, Siddesh2104

 

Thanks for prompt response,

1. csd(kafka-1.2.0.jar) from http://archive.cloudera.com/csds/kafka/
    parcel(KAFKA-2.0.1-1.2.0.1.p0.5-trusty.parcel) from: http://archive.cloudera.com/kafka/parcels/latest/

3. yes, parcel has been activated as attached

4. the red color warning message prevent us from clicking "continue", we got stuck on the page as attached, nothing happened after we pressed "continue"

   

Any more advice would be appreciated.

 

Thanks

Joe

snap.gif

 

snap.gif

 

Explorer

While adding the Kafka service, have you added mirrormaker role as well?
If yes, can you try without adding it?

Explorer

yes, without selecting mirror , we can press continue, then we hit the following errors as attached,

 

any further advice would be appreciated

 

snap.gif

Explorer
Hi, any more advicde would be appreciated...

Rising Star

Hi, please perform the following actions:

 

1) Fill in "Source Brokers List" --> "nameOfClouderaKafkaBrokerServer.yourdomain.com:9092". This is the Server (or Servers) where you configured the Kafka Broker (NOT the MirrorMaker).

 

2) Fill in "Destination Brokers List" --> "nameOfRemoteBrokerServer.otherdomain.com:9092". This is supposed to be a remote Cluster that will receive Topics sent over by your MirrorMaker. If you have one, put in that one. Otherwise just put in another Server in your network, whatever Server.

 

Please note that both this Server Names must be FQDN and resolvable by your DNS (or hosts file), otherwise you'll get other errors. Also the format with the trailing Port Number is mandatory!

 

3) Click "Continue". Service will NOT start (error). Do not navigate away from that screen

 

4) Open another Cloudera Manager in another browser pane. You should now see "Kafka" in the list of Services (red, but it should be there). Click on the Kafka Service and then "Configure".

 

5) Search for the "java heap space" Configuration Property. The standard Java Heap Space you'll find already set up should be 50 MBytes. Put in at least 256 MBytes. The original value is simply not enough.

 

6) Now search for the "whitelist" Configuration Property. In the field, put in "(?!x)x" (without the quotation marks). That's a regular expression that does not match anything. Given that apparently a Whitelist is mandatory for the Mirrormaker Service to start, and I'm assuming you don't want to replicate any topics remotely right now, just put in something that won't replicate anything e.g. that regular expression.

 

7) Save the changes and go back to the original Configuration Screen on the othe browser pane. Click "Retry", or wathever, or even exit that screen and manually restart the Kafka Service in Cloudera Manager.

 

That should work, at least it did for me! HTH

Explorer

yes, Frozenwave's solution works,  thanks 

Explorer

 A quick test failed with the following error, any advice would be appreciated:
root@cloudera01:/opt/cloudera/parcels/KAFKA-2.0.1-1.2.0.1.p0.5/lib/kafka/bin# ./kafka-topics.sh --zookeeper cloudera01.pscsi.com:2181 --list
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/kafka/common/config/ConfigDef$Validator : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:288)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:41)
at kafka.admin.TopicCommand.main(TopicCommand.scala)

Explorer
after setting java home to 1.7, there's no error and nothering happened as follows:
root@cloudera01:~# kafka-topics.sh --zookeeper cloudera01.pscsi.com:2181 --list
root@cloudera01:~# kafka-topics --zookeeper cloudera01.pscsi.com:2181 --list
root@cloudera01:~#

any advice would be appreciated.

Explorer
we followed this official link to test kafka:
http://www.cloudera.com/documentation/kafka/latest/topics/kafka_command_line.html

not sure why ther's no any error message, it got back to prompt

Explorer
it works with the following command:
kafka-topics --create --zookeeper cloudera01 --topic ggz --partitions 1 --replication-factor 1
kafka-console-producer --broker-list cloudera02:9092 --topic ggz
kafka-console-consumer --zookeeper cloudera01:2181 --topic ggz --from-beginning

Explorer

Hi,

 

I followed same process till here.

Still getting errors as below. Could you please suggest what else i can do.

 

Image 1.png

Image 2.png

New Contributor

Did you solve the problem? 

Explorer

Hi Joe,

 

nameOfClouderaKafkaBrokerServer.yourdomain.com:9092

 

What are the actual values we can give for nameOfClouderaKafkaBrokerServer? yourdomain?

Is port number 9092 same always?

 

Please ignore .. i see this questions is answered in Page 2 🙂 Very good topic. 

 

Thanks,

Chandra

 

New Contributor

Fantastic post bro

New Contributor
Could you please give me what exactly to put into these fields on CDH kafka configuring.

1) Fill in "Source Brokers List" --> "nameOfClouderaKafkaBrokerServer.yourdomain.com:9092". This is the Server (or Servers) where you configured the Kafka Broker (NOT the MirrorMaker).



2) Fill in "Destination Brokers List" --> "nameOfRemoteBrokerServer.otherdomain.com:9092". This is supposed to be a remote Cluster that will receive Topics sent over by your MirrorMaker. If you have one, put in that one. Otherwise just put in another Server in your network, whatever Server.

New Contributor
I have given Destination Broker List as
quickstart.cloudera:9092

Source Broker List as
quickstart.cloudera:9092

New Contributor

Yeah its worked for me. Thanks for the solution

New Contributor

thank you very much it really helped me

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.