question Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE} in Support Questions

Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

hoda_moradi2014 — Fri, 08 Feb 2019 04:04:16 GMT

I followed Producing Events/Messages to Kafka on a Secured Cluster.

I am setting export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf"

and passing --security-protocol SASL_PLAINTEXT my command looks like

./bin/kafka-console-producer.sh --broker-list <Brokker-hosts>:6667  --topic test  --security-protocol SASL_PLAINTEXT

kafka_client_jaas.conf:

KafkaClient {

com.sun.security.auth.module.Krb5LoginModule required

useTicketCache=true

renewTicket=true

serviceName="kafka";

};

kafka_jaas.conf:

KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    storeKey=true
    useTicketCache=false
    serviceName="kafka"
    principal="kafka/_host@EXAMPLE.COM";
    };
    KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=true
    renewTicket=true
    serviceName="kafka";
    };
    Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    storeKey=true
    useTicketCache=false
    serviceName="zookeeper"
    principal="kafka/_host@EXAMPLE.COM";
    };
    com.sun.security.jgss.krb5.initiate {
    com.sun.security.auth.module.Krb5LoginModule required
    renewTGT=false
    doNotPrompt=true
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    storeKey=true
    useTicketCache=false
    serviceName="kafka"
    principal="kafka/_host@EXAMPLE.COM";
    };

When I run this I get the prompt to type my message but then I get:

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 307 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 308 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 309 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 310 : {test=LEADER_NOT_AVAILABLE}

19/02/07 13:35:52 WARN NetworkClient: Error while fetching metadata with correlation id 311 : {test=LEADER_NOT_AVAILABLE}

my Kafka version is : 1.0.0

I made sure that topic "test" exists and I can get the leader ids when I run describe

How can I resolve this issue?

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Fri, 08 Feb 2019 04:27:01 GMT

@hoda moradi

Can you check these 2 properties in server.properties

Please follow the below steps.

Add the following lines in server.properties for the brokers file:
```
listeners=PLAINTEXT://host.name:port
advertised.listeners=PLAINTEXT://host.name:port 
```
where host.name is the IP address or host name of the Kafka broker.
Restart the Kafka brokers and test.

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

hoda_moradi2014 — Fri, 08 Feb 2019 04:39:20 GMT

@Geoffrey Shelton Okot in the server.properties I see

listeners=SASL_PLAINTEXT://host.name:6667
advertised.listeners=SASL_PLAINTEXT://host.name:6667

Do I need to change them? The cluster is secured and we are using SASL_PLAINTEXT not PLAINTEXT

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Fri, 08 Feb 2019 05:11:34 GMT

@hoda moradi

Have you secured your kafka with SSL and Keberos? Was it working before?

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

hoda_moradi2014 — Fri, 08 Feb 2019 05:14:01 GMT

@Geoffrey Shelton Okot Yes the Kafka cluster is secured with SASL and Kerberos. We just did this so it is the first time we are testing it. We followed Hortonwork's documentation to secure the cluster.

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Fri, 08 Feb 2019 05:40:02 GMT

@hoda moradi

Okay I am already seeing issues with your kafka_jaas.conf there are too many entries. Can tokenize your server.properties and share the entries

listeners
advertised.listeners
sasl.enabled.mechanisms
sasl.kerberos.service.name

Is it an HDP cluster if so version or standalone kafka cluster (how many nodes)

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

hoda_moradi2014 — Fri, 08 Feb 2019 05:52:44 GMT

@Geoffrey Shelton Okot

It is a HDP cluster version: 2.6.5.4-1. I have a Kafka cluster with 6 brokers.

listeners=SASL_PLAINTEXT://host.name:6667
advertised.listeners=SASL_PLAINTEXT://host.name:6667
sasl.enabled.mechanisms=GSSAPI

I do not see "sasl.kerberos.service.name" in server.properties I do see it in kafka_jaas.conf and kafka_client_jaas.conf being set to 'kafka'

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Fri, 08 Feb 2019 06:29:51 GMT

@hoda moradi

Your kafka_jaas.conf and contradicting entries 4 in number can you back up the current file and re-adjust the one I have attached on all the brokers if multimode.

Below is functioning SSL, Kerberos config

#########################################################
# server.properties
#########################################################
listeners=PLAINTEXT://0.0.0.0:9092,SSL:0.0.0.0:9093,SASL_SSL://0.0.0.0:9094
advertised.listeners=PLAINTEXT://FQDN_Broker:9092,SSL://FQDN_Broker:9093,SASL_SSL://FQDN_Broker:9092

sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka

Client

#########################################################
# kafka_client_jaas.conf:
#########################################################
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
};

Server

#########################################################
# kafka_jaas.conf
#########################################################
KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka.service.keytab"
    useTicketCache=false
    serviceName="kafka"
    principal="kafka/_host@EXAMPLE.COM";
    };

After these steps restart the Kafka broker(s) please revert

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

hoda_moradi2014 — Fri, 08 Feb 2019 06:47:39 GMT

We are using SASL and Kerberos not SSL. Do you have any functioning SASL and Kerberos config?

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Fri, 08 Feb 2019 07:06:09 GMT

@hoda moradi

Just omit the SSL_SASL entry in the server.properties

listeners=PLAINTEXT://0.0.0.0:9092,SASL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://FQDN_Broker:9092,SASL://FQDN_Broker:9093

HTH

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Fri, 08 Feb 2019 17:35:27 GMT

@hoda moradi

Any updates?

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

hoda_moradi2014 — Fri, 08 Feb 2019 22:34:36 GMT

@Geoffrey Shelton Okot

No I still get the same error.

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Fri, 08 Feb 2019 23:36:58 GMT

@hoda moradi

Can you share the below configs?

Listener entries in server.properties
kafka_client_jaas.conf
kafka_jaas.conf
/var/log/kafka/
server.log

Thanks

Re: Kafka producer/consumer with kerberos authentication not working, Error while fetching metadata with correlation id 299 : {test=LEADER_NOT_AVAILABLE}

Shelton — Sun, 10 Mar 2019 15:36:55 GMT

@ hoda moradi

Any updates