Kafka.properties override for listeners property fails

[#1]

Setting the property

listeners=PLAINTEXT://host1:9092,PLAINTEXT://host2:9092,PLAINTEXT://host3:9092,PLAINTEXT://host4:9092,

Fails with exception

java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port
	at scala.Predef$.require(Predef.scala:219)
	at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:903)
	at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:911)
	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:864)
	at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:698)
	at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:695)
	at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
	at kafka.Kafka$.main(Kafka.scala:58)
	at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
	at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)

As long as each listener is on a different machine, why should " each listener must have a different port"

[#2]

Cloudera Team

can I get the confirmation on this Product Limitations pls.

thanks

Deepak

[#3]

That documentation is correct. The flume kafka client version that is in CDH5.7 and lower isn't capable of communicating with secure kafka. You need to upgrade to CDH5.8 or higher version of flume in order to be able to connect to secure kafka, and use these steps to configure flume: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_flume_kafka_security_confg.htm...


-pd

[#4]

Thanks PD for the confirmation

Appreciate that

can you please let me know technically what change we are going to do  in Flume or Kafka side to make it work over kerberos on versions > 5.7

[#5]

I want to know why the current version is not suppoirting and what change has been done to make it work ?

[#6]

As I stated in my recent comment, the flume kafka client was upgraded as a part of the CDH5.8 upgrade to be able to use the new consumer API, which supports secure communication with kerberos. Versions prior to CDH5.8 use the old api which doesn't support kerberos or SSL.

You will have to upgrade to get this new functionality, or run flume outside of Cloudera Manager, using tarballs or RPM's.

-pd