not able to produce messages on kafka broker after enabling kerberos, encryption and autorization

[#1]

I am getting below error on producer while producing message,

Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 18 86 c2 46 01 01 00 00 6b 61 66 6b 61 2d 63 6c 69 65 6e 74 2f 75 62 75 6e 74 75 32 36 2e 6d 73 74 6f 72 6d 2e 63 6f 6d 40 4d 53 54 4f 52 4d 2e 43 4f 4d 46 80 3d 15 92 45 c2 58 cd 12 11 76 ] [2018-09-14 11:12:39,775] WARN Error while fetching metadata with correlation id 1 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:39,874] WARN Error while fetching metadata with correlation id 2 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:39,978] WARN Error while fetching metadata with correlation id 3 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,082] WARN Error while fetching metadata with correlation id 4 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,186] WARN Error while fetching metadata with correlation id 5 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,290] WARN Error while fetching metadata with correlation id 6 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-09-14 11:12:40,394] WARN Error while fetching metadata with correlation id 7 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) ^C[2018-09-14 11:12:40,409] WARN [Principal=kafka-client/kafka1.example.com@EXAMPLE.COMa]: TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)

[#2]

@Ankita Ghate

Can you please check if "kafka.acl" is setup proplery? This is just to verify if you have provided proper read permission on the group and topic? as we see "UNKNOWN_TOPIC_OR_PARTITION"

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/kafka-acl-examples.html

# kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --list

Also have you enabled Kafka Ranger Plugin? In that case have you setup the ranger policies correctly?

- Also can you please try restarting your Kafka Brokers and try once again?

[#3]

@Jay Kumar SenSharma thanks for the response.

kafka.acl are set properly for producer and consumer for test topic still giving "UNKNOWN_TOPIC_OR_PARTITION"

[#4]

Below is the list of kafka-acls on test topic,

Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 27 d9 c3 5b 01 01 00 00 6b 61 66 6b 61 2f 75 62 75 6e 74 75 32 36 2e 6d 73 74 6f 72 6d 2e 63 6f 6d 40 4d 53 54 4f 52 4d 2e 43 4f 4d 9c d0 cc bf 71 74 35 93 38 71 59 a0 ] Current ACLs for resource `Topic:test`: user:root has Allow permission for operations: Read from hosts: kafka1.example.com user:deepak has Allow permission for operations: Read from hosts: kafka1.example.com user:root has Allow permission for operations: Write from hosts: kafka1.example.com user:deepak has Allow permission for operations: Write from hosts: kafka1.example.com

[#5]

@Jay Kumar SenSharma any solution to this?

[#6]

@Ankita Ghate

I think you need to follow these post configuration steps for the client jass file Configuring kafka Authentication with Kerberos