Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

[#1]

I followed Producing Events/Messages to Kafka on a Secured Cluster.

I am setting export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf"

and passing --security-protocol SASL_PLAINTEXT my command looks like

./bin/kafka-console-producer.sh --broker-list <Brokker-hosts>:6667  --topic test  --security-protocol SASL_PLAINTEXT

kafka_client_jaas.conf:

KafkaClient {

com.sun.security.auth.module.Krb5LoginModule required

useTicketCache=true

renewTicket=true

serviceName="kafka";

};

kafka_jaas.conf:

KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    storeKey=true
    useTicketCache=false
    serviceName="kafka"
    principal="kafka/_host@EXAMPLE.COM";
    };
    KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=true
    renewTicket=true
    serviceName="kafka";
    };
    Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    storeKey=true
    useTicketCache=false
    serviceName="zookeeper"
    principal="kafka/_host@EXAMPLE.COM";
    };
    com.sun.security.jgss.krb5.initiate {
    com.sun.security.auth.module.Krb5LoginModule required
    renewTGT=false
    doNotPrompt=true
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    storeKey=true
    useTicketCache=false
    serviceName="kafka"
    principal="kafka/_host@EXAMPLE.COM";
    };

When I run this I get the prompt to type my message but then I get:

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 307 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 308 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 309 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 310 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 311 : {test=LEADER_NOT_AVAILABLE}

my Kafka version is : 1.0.0

I made sure that topic "test" exists and I can get the leader ids when I run describe

How can I resolve this issue?

[#2]

@hoda moradi

Can you check these 2 properties in server.properties

Please follow the below steps.

1. Add the following lines in server.properties for the brokers file:

   listeners=PLAINTEXT://host.name:port
   advertised.listeners=PLAINTEXT://host.name:port 
   

   where host.name is the IP address or host name of the Kafka broker.

2. Restart the Kafka brokers and test.

[#3]

@Geoffrey Shelton Okot in the server.properties I see

listeners=SASL_PLAINTEXT://host.name:6667
advertised.listeners=SASL_PLAINTEXT://host.name:6667

Do I need to change them? The cluster is secured and we are using SASL_PLAINTEXT not PLAINTEXT

[#4]

@hoda moradi

Have you secured your kafka with SSL and Keberos? Was it working before?

[#5]

@Geoffrey Shelton Okot Yes the Kafka cluster is secured with SASL and Kerberos. We just did this so it is the first time we are testing it. We followed Hortonwork's documentation to secure the cluster.

[#6]

@hoda moradi

Okay I am already seeing issues with your kafka_jaas.conf there are too many entries. Can tokenize your server.properties and share the entries

• listeners
• advertised.listeners
• sasl.enabled.mechanisms
• sasl.kerberos.service.name

Is it an HDP cluster if so version or standalone kafka cluster (how many nodes)

[#7]

@Geoffrey Shelton Okot

It is a HDP cluster version: 2.6.5.4-1. I have a Kafka cluster with 6 brokers.

listeners=SASL_PLAINTEXT://host.name:6667
advertised.listeners=SASL_PLAINTEXT://host.name:6667
sasl.enabled.mechanisms=GSSAPI

I do not see "sasl.kerberos.service.name" in server.properties I do see it in kafka_jaas.conf and kafka_client_jaas.conf being set to 'kafka'

[#8]

@hoda moradi

Your kafka_jaas.conf and contradicting entries 4 in number can you back up the current file and re-adjust the one I have attached on all the brokers if multimode.

Below is functioning SSL, Kerberos config

#########################################################
# server.properties
#########################################################
listeners=PLAINTEXT://0.0.0.0:9092,SSL:0.0.0.0:9093,SASL_SSL://0.0.0.0:9094
advertised.listeners=PLAINTEXT://FQDN_Broker:9092,SSL://FQDN_Broker:9093,SASL_SSL://FQDN_Broker:9092

sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka 

Client

#########################################################
# kafka_client_jaas.conf:
#########################################################
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
}; 

Server

#########################################################
# kafka_jaas.conf
#########################################################
KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    useTicketCache=false
    serviceName="kafka"
    principal="kafka/_host@EXAMPLE.COM";
    }; 

After these steps restart the Kafka broker(s) please revert

[#9]

We are using SASL and Kerberos not SSL. Do you have any functioning SASL and Kerberos config?

[#10]

@hoda moradi

Just omit the SSL_SASL entry in the server.properties

listeners=PLAINTEXT://0.0.0.0:9092,SASL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://FQDN_Broker:9092,SASL://FQDN_Broker:9093

HTH

[#11]

@hoda moradi

Any updates?

[#12]

@Geoffrey Shelton Okot

No I still get the same error.

[#13]

@hoda moradi

Can you share the below configs?

• Listener entries in server.properties
• kafka_client_jaas.conf
  
• kafka_jaas.conf

• /var/log/kafka/

  server.log

Thanks

[#14]

@ hoda moradi

Any updates