question Re: ISSUE with kafka while enabling kerberos in Support Questions

ISSUE with kafka while enabling kerberos

sumitsharma8692 — Thu, 30 Mar 2017 23:20:37 GMT

i am using kafka version 0.9.0

using two kafka brokers.

After we enable the kerberos, the kafka producer starts showing leadernotfound exception where as the describe command shows that leader is assigned to the topic.

Also,i noticed something weird about kafka znode as follows:

If we try to get the information for kafka broker znode after enabling kerberos, it shows as below:

get /brokers/ids/1002 {"jmx_port":-1,"timestamp":"1490888565689","endpoints":["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1}

cZxid = 0x1300000241

ctime = Thu Mar 30 21:12:45 IST 2017

mZxid = 0x1300000241

mtime = Thu Mar 30 21:12:45 IST 2017

pZxid = 0x1300000241 cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x15b1439a498001a

dataLength = 140

numChildren = 0

The host and the port information is not getting configured. I tried configuring these(advertised.host.name and advertised.port) through ambari as well as setting proprty in server.properties file and restarting the kafka manually but it is not getting reflected.

But, if i change the protocol from PLAINTEXTSASL/SASL_PLAINTEXT to simply PLAINTEXT then the host and port information is gets updated correctly as below.

get /brokers/ids/1002

{"jmx_port":-1,"timestamp":"1490888108079","endpoints":["PLAINTEXT://0.0.0.0:9092"],"host":"xxxx.domain.com","version":2,"port":9092}

cZxid = 0x1300000230

ctime = Thu Mar 30 21:05:08

IST 2017 mZxid = 0x1300000230

mtime = Thu Mar 30 21:05:08 IST 2017

pZxid = 0x1300000230 cversion = 0 dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x25b1439a48c001a

dataLength = 161

numChildren = 0

can anyone help?

TIA.

Re: ISSUE with kafka while enabling kerberos

amankumbare — Fri, 31 Mar 2017 11:32:27 GMT

@Sumit Sharma

- Are you using custom producer or kafka-console-producer script ? Post the exact command used for producing messages to topic.

- Are you facing this issue for a specific topic or for all topics ?

- As which user you are running producer ?

- Please post the output of klist command as a user which is executing producer code/script.

- Output of describe command.

- What is the version of jdk you are running ?

Re: ISSUE with kafka while enabling kerberos

parameswarnc — Fri, 31 Mar 2017 12:06:05 GMT

@amankumbare

Thanks for responding, Sumit and me work in the same team .

It is happening for all topics and jdk we using is 7 .

The main issue here is the content of the znode of the broker has no host and port information updated .

["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1}

-Is it the expected behavior ?

-And it happens when listener value is PLAINTEXTSASL://xxxx.domain.com:9092 and everything is fine when it is just PLAINTEXT .

Thanks in advance ,

Param.

Re: ISSUE with kafka while enabling kerberos

amankumbare — Fri, 31 Mar 2017 12:31:37 GMT

@Param NC

@Sumit Sharma This is a know issue. What is the version of HDP you are using ?

After the cluster is kerberized, the default value of ‘host’ is set to ‘null’ and ‘port’ is set to ‘-1’ which becomes invalid to the legacy Kafka clients. For example:

[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1479492220088","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":null,"version":2,"port":-1}

The default ‘host’ and ‘port’ here in Zookeeper are used to maintain backward compatibility for older Kafka clients, but only PLAINTEXT protocol is supported as default.

If the broker does not listen on PLAINTEXT protocol, for example PLAINTEXTSASL after kerberizing, an empty endpoint (that is, "host":null,"port”:-1) will be registered and older clients will break.

Could you please try to manually change the ‘host’ and ‘port’ to actual value.

For example:

[zk: localhost:2181(CONNECTED) 2] set /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1481503998915","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":"test.support.com","version":3,"port":6667}

Re: ISSUE with kafka while enabling kerberos

sandyy006 — Fri, 31 Mar 2017 19:14:49 GMT

@Param NC

@Sumit Sharma

The znode values are expected post enabling kerberos and that should not cause any issue.

What is the actual issue you are facing, can post the the error trace here ?

Re: ISSUE with kafka while enabling kerberos

sumitsharma8692 — Fri, 31 Mar 2017 20:12:05 GMT

@amankumbare

thanks ! this worked for me.

Re: ISSUE with kafka while enabling kerberos

satish_p — Thu, 11 Jul 2019 14:27:54 GMT

@Param NC

I have tried above commands For me get is working but while changing port and host, manually using the above command facing below error

Authentication is not valid : /brokers/ids/1001

Re: ISSUE with kafka while enabling kerberos

interbooy — Thu, 26 Aug 2021 12:24:22 GMT

try to switch to a kafka user : su kafka
and retry again