Created 03-30-2017 04:20 PM
i am using kafka version 0.9.0
using two kafka brokers.
After we enable the kerberos, the kafka producer starts showing leadernotfound exception where as the describe command shows that leader is assigned to the topic.
Also,i noticed something weird about kafka znode as follows:
If we try to get the information for kafka broker znode after enabling kerberos, it shows as below:
get /brokers/ids/1002 {"jmx_port":-1,"timestamp":"1490888565689","endpoints":["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1}
cZxid = 0x1300000241
ctime = Thu Mar 30 21:12:45 IST 2017
mZxid = 0x1300000241
mtime = Thu Mar 30 21:12:45 IST 2017
pZxid = 0x1300000241 cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15b1439a498001a
dataLength = 140
numChildren = 0
The host and the port information is not getting configured. I tried configuring these(advertised.host.name and advertised.port) through ambari as well as setting proprty in server.properties file and restarting the kafka manually but it is not getting reflected.
But, if i change the protocol from PLAINTEXTSASL/SASL_PLAINTEXT to simply PLAINTEXT then the host and port information is gets updated correctly as below.
get /brokers/ids/1002
{"jmx_port":-1,"timestamp":"1490888108079","endpoints":["PLAINTEXT://0.0.0.0:9092"],"host":"xxxx.domain.com","version":2,"port":9092}
cZxid = 0x1300000230
ctime = Thu Mar 30 21:05:08
IST 2017 mZxid = 0x1300000230
mtime = Thu Mar 30 21:05:08 IST 2017
pZxid = 0x1300000230 cversion = 0 dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x25b1439a48c001a
dataLength = 161
numChildren = 0
can anyone help?
TIA.
Created 03-31-2017 05:31 AM
@Sumit Sharma This is a know issue. What is the version of HDP you are using ?
After the cluster is kerberized, the default value of ‘host’ is set to ‘null’ and ‘port’ is set to ‘-1’ which becomes invalid to the legacy Kafka clients. For example:
[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1479492220088","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":null,"version":2,"port":-1}
The default ‘host’ and ‘port’ here in Zookeeper are used to maintain backward compatibility for older Kafka clients, but only PLAINTEXT protocol is supported as default.
If the broker does not listen on PLAINTEXT protocol, for example PLAINTEXTSASL after kerberizing, an empty endpoint (that is, "host":null,"port”:-1) will be registered and older clients will break.
Could you please try to manually change the ‘host’ and ‘port’ to actual value.
For example:
[zk: localhost:2181(CONNECTED) 2] set /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1481503998915","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":"test.support.com","version":3,"port":6667}
Created 03-31-2017 04:32 AM
- Are you using custom producer or kafka-console-producer script ? Post the exact command used for producing messages to topic.
- Are you facing this issue for a specific topic or for all topics ?
- As which user you are running producer ?
- Please post the output of klist command as a user which is executing producer code/script.
- Output of describe command.
- What is the version of jdk you are running ?
Created 03-31-2017 05:06 AM
Thanks for responding, Sumit and me work in the same team .
It is happening for all topics and jdk we using is 7 .
The main issue here is the content of the znode of the broker has no host and port information updated .
["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1}
-Is it the expected behavior ?
-And it happens when listener value is PLAINTEXTSASL://xxxx.domain.com:9092 and everything is fine when it is just PLAINTEXT .
Thanks in advance ,
Param.
Created 03-31-2017 05:31 AM
@Sumit Sharma This is a know issue. What is the version of HDP you are using ?
After the cluster is kerberized, the default value of ‘host’ is set to ‘null’ and ‘port’ is set to ‘-1’ which becomes invalid to the legacy Kafka clients. For example:
[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1479492220088","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":null,"version":2,"port":-1}
The default ‘host’ and ‘port’ here in Zookeeper are used to maintain backward compatibility for older Kafka clients, but only PLAINTEXT protocol is supported as default.
If the broker does not listen on PLAINTEXT protocol, for example PLAINTEXTSASL after kerberizing, an empty endpoint (that is, "host":null,"port”:-1) will be registered and older clients will break.
Could you please try to manually change the ‘host’ and ‘port’ to actual value.
For example:
[zk: localhost:2181(CONNECTED) 2] set /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1481503998915","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":"test.support.com","version":3,"port":6667}
Created 03-31-2017 12:14 PM
The znode values are expected post enabling kerberos and that should not cause any issue.
What is the actual issue you are facing, can post the the error trace here ?
Created 03-31-2017 01:12 PM
thanks ! this worked for me.
Created 07-11-2019 07:27 AM
I have tried above commands For me get is working but while changing port and host, manually using the above command facing below error
Authentication is not valid : /brokers/ids/1001
Created 08-26-2021 05:24 AM
try to switch to a kafka user : su kafka
and retry again