not able to produce messages on kafka broker after configuring it for encryption and using protocol SASL_SSL

[#1]

Will use keytab

Commit Succeeded

a

s

d

f

[2018-09-11 03:08:16,347] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 03:09:16,349] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 03:10:16,349] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. a s ddd ffffffffffffffffffff gggggggggggggggggggggggg [2018-09-11 03:11:16,350] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 03:12:16,351] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 03:13:16,352] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 03:14:16,352] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 03:15:16,353] ERROR Error when sending message to topic test with key: null, value: 20 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 03:16:16,353] ERROR Error when sending message to topic test with key: null, value: 24 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

[#2]

@Ankita Ghate

Are you passing the "--security-protocol" option while running the producer? Can you please share the exact command which you are using to start the producer?

Example:

# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list <BROKER_1>:6667,<BROKER_2>:6667 --topic test --security-protocol PLAINTEXTSASL

Please refer to the NOTE in the following doc saying : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/secure-kafka-produce-ev...
Add "--security-protocol" SASL_PLAINTEXT to the kafka-console-producer.sh runtime options.

[#3]

@Jay Kumar SelenSharma thanks for the response, this issue was solved.

I am getting below error on producer while producing message,

Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 18 86 c2 46 01 01 00 00 6b 61 66 6b 61 2d 63 6c 69 65 6e 74 2f 75 62 75 6e 74 75 32 36 2e 6d 73 74 6f 72 6d 2e 63 6f 6d 40 4d 53 54 4f 52 4d 2e 43 4f 4d 46 80 3d 15 92 45 c2 58 cd 12 11 76 ] [2018-09-14 11:12:39,775] WARN Error while fetching metadata with correlation id 1 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:39,874] WARN Error while fetching metadata with correlation id 2 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:39,978] WARN Error while fetching metadata with correlation id 3 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,082] WARN Error while fetching metadata with correlation id 4 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,186] WARN Error while fetching metadata with correlation id 5 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,290] WARN Error while fetching metadata with correlation id 6 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,394] WARN Error while fetching metadata with correlation id 7 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) ^C[2018-09-14 11:12:40,409] WARN [Principal=kafka-client/kafka1.example.com@EXAMPLE.COMa]: TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)

[#4]

@Ankita Ghate

Good to know that your original issue reported in the thread is resolved. I see that your current issue is slightly different. It is always recommended to open separate HCC thread for different issues so that One HCC thread has one Specific solution / answer. This helps us HCC users to quickly find the solution for originally reported issues. Long threads with multiple issues addressed in Single thread causes users to get easily confused.

I will suggest you to please mark this HCC thread as Answered (Accepted) by clicking the Accept button and then open a new thread for the recent issue.

[#5]

@Ankita Ghate

Thank you for opening a new HCC thread for the current issue.

For the "UNKNOWN_TOPIC_OR_PARTITION" issue i have responded to: https://community.hortonworks.com/questions/221979/not-able-to-produce-messages-on-kafka-broker-afte...