Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kafka.properties override for listeners property fails

avatar
Explorer

Setting the property

listeners=PLAINTEXT://host1:9092,PLAINTEXT://host2:9092,PLAINTEXT://host3:9092,PLAINTEXT://host4:9092,

Fails with exception

java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port
	at scala.Predef$.require(Predef.scala:219)
	at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:903)
	at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:911)
	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:864)
	at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:698)
	at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:695)
	at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
	at kafka.Kafka$.main(Kafka.scala:58)
	at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
	at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)

As long as each listener is on a different machine, why should " each listener must have a different port"

14 REPLIES 14

avatar

Cloudera Team

 

can I get the confirmation on this Product Limitations pls.

 

thanks

Deepak

avatar

That documentation is correct. The flume kafka client version that is in CDH5.7 and lower isn't capable of communicating with secure kafka. You need to upgrade to CDH5.8 or higher version of flume in order to be able to connect to secure kafka, and use these steps to configure flume: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_flume_kafka_security_confg.htm...


-pd

avatar

Thanks PD for the confirmation

 

Appreciate that

 

can you please let me know technically what change we are going to do  in Flume or Kafka side to make it work over kerberos on versions > 5.7

avatar

I want to know why the current version is not suppoirting and what change has been done to make it work ?

avatar

As I stated in my recent comment, the flume kafka client was upgraded as a part of the CDH5.8 upgrade to be able to use the new consumer API, which supports secure communication with kerberos. Versions prior to CDH5.8 use the old api which doesn't support kerberos or SSL.

You will have to upgrade to get this new functionality, or run flume outside of Cloudera Manager, using tarballs or RPM's.

-pd