ISSUE with kafka while enabling kerberos

[#1]

i am using kafka version 0.9.0

using two kafka brokers.

After we enable the kerberos, the kafka producer starts showing leadernotfound exception where as the describe command shows that leader is assigned to the topic.

Also,i noticed something weird about kafka znode as follows:

If we try to get the information for kafka broker znode after enabling kerberos, it shows as below:

get /brokers/ids/1002 {"jmx_port":-1,"timestamp":"1490888565689","endpoints":["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1}

cZxid = 0x1300000241

ctime = Thu Mar 30 21:12:45 IST 2017

mZxid = 0x1300000241

mtime = Thu Mar 30 21:12:45 IST 2017

pZxid = 0x1300000241 cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x15b1439a498001a

dataLength = 140

numChildren = 0

The host and the port information is not getting configured. I tried configuring these(advertised.host.name and advertised.port) through ambari as well as setting proprty in server.properties file and restarting the kafka manually but it is not getting reflected.

But, if i change the protocol from PLAINTEXTSASL/SASL_PLAINTEXT to simply PLAINTEXT then the host and port information is gets updated correctly as below.

get /brokers/ids/1002

{"jmx_port":-1,"timestamp":"1490888108079","endpoints":["PLAINTEXT://0.0.0.0:9092"],"host":"xxxx.domain.com","version":2,"port":9092}

cZxid = 0x1300000230

ctime = Thu Mar 30 21:05:08

IST 2017 mZxid = 0x1300000230

mtime = Thu Mar 30 21:05:08 IST 2017

pZxid = 0x1300000230 cversion = 0 dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x25b1439a48c001a

dataLength = 161

numChildren = 0

can anyone help?

TIA.

[#2]

@Sumit Sharma

- Are you using custom producer or kafka-console-producer script ? Post the exact command used for producing messages to topic.

- Are you facing this issue for a specific topic or for all topics ?

- As which user you are running producer ?

- Please post the output of klist command as a user which is executing producer code/script.

- Output of describe command.

- What is the version of jdk you are running ?

[#3]

@amankumbare

Thanks for responding, Sumit and me work in the same team .

It is happening for all topics and jdk we using is 7 .

The main issue here is the content of the znode of the broker has no host and port information updated .

["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1}

-Is it the expected behavior ?

-And it happens when listener value is PLAINTEXTSASL://xxxx.domain.com:9092 and everything is fine when it is just PLAINTEXT .

Thanks in advance ,

Param.

[#4 · Accepted solution]

@Param NC

@Sumit Sharma This is a know issue. What is the version of HDP you are using ?

After the cluster is kerberized, the default value of ‘host’ is set to ‘null’ and ‘port’ is set to ‘-1’ which becomes invalid to the legacy Kafka clients. For example:

[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1479492220088","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":null,"version":2,"port":-1}

The default ‘host’ and ‘port’ here in Zookeeper are used to maintain backward compatibility for older Kafka clients, but only PLAINTEXT protocol is supported as default.

If the broker does not listen on PLAINTEXT protocol, for example PLAINTEXTSASL after kerberizing, an empty endpoint (that is, "host":null,"port”:-1) will be registered and older clients will break.

Could you please try to manually change the ‘host’ and ‘port’ to actual value.

For example:

[zk: localhost:2181(CONNECTED) 2] set /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1481503998915","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":"test.support.com","version":3,"port":6667}

[#5]

@Param NC

@Sumit Sharma

The znode values are expected post enabling kerberos and that should not cause any issue.

What is the actual issue you are facing, can post the the error trace here ?

[#6]

@amankumbare

thanks ! this worked for me.

[#7]

@Param NC

I have tried above commands For me get is working but while changing port and host, manually using the above command facing below error

Authentication is not valid : /brokers/ids/1001

[#8]

try to switch to a kafka user : su kafka
and retry again