Issue producing data into Kerborized Kafka cluster

[#1]

Hi,

I'm trying to produce data into a kerborized 3 node kafka cluster with SimpleAclAuthorizer.

When I run:

[root@domain bin]# ./kafka-console-producer.sh --broker-list host.domain.net:6667 --topic topic1 --security-protocol SASL_PLAINTEXT 
Test
[2017-04-11 09:07:43,821] WARN Error while fetching metadata with correlation id 0 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,022] WARN Error while fetching metadata with correlation id 1 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,122] WARN Error while fetching metadata with correlation id 2 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,223] WARN Error while fetching metadata with correlation id 3 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,323] WARN Error while fetching metadata with correlation id 4 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,423] WARN Error while fetching metadata with correlation id 5 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,523] WARN Error while fetching metadata with correlation id 6 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,624] WARN Error while fetching metadata with correlation id 7 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:43,821] WARN Error while fetching metadata with correlation id 0 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,022] WARN Error while fetching metadata with correlation id 1 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,122] WARN Error while fetching metadata with correlation id 2 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,223] WARN Error while fetching metadata with correlation id 3 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,323] WARN Error while fetching metadata with correlation id 4 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,423] WARN Error while fetching metadata with correlation id 5 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,523] WARN Error while fetching metadata with correlation id 6 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,624] WARN Error while fetching metadata with correlation id 7 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

I can see my topic if I run:

[root@domain bin]# ./kafka-topics.sh --list --zookeeper host.domain.net:2181
topic1 

I can see the ACL's I've applied:

[root@domain bin]# ./kafka-acls.sh --list --authorizer-properties zookeeper.connect=host.domain.net:2181 --topic topic1
Current ACLs for resource `Topic:topic1`:
User:nifi/host.domain.net has Allow permission for operations: Write from hosts: *
User:nifi/host.domain.net has Allow permission for operations: Write from hosts: * 

I've run kinit:

[root@domain bin]# klist 
Ticket cache: FILE:/tmp/krb5cc_...
Default principal: nifi/host.domain.net@DOMAIN.NET
Valid starting     Expires            Service principal
04/11/17 08:59:42  04/11/17 18:59:42  krbtgt/DOMAIN.NET@DOMAIN.NET
        renew until 04/18/17 08:59:42 Default principal: nifi/host.domain.net@DOMAIN.NET
Valid starting     Expires            Service principal
04/11/17 08:59:42  04/11/17 18:59:42  krbtgt/DOMAIN.NET@DOMAIN.NET
        renew until 04/18/17 08:59:42 

Thanks in advance for any help.

Ollie

[#2]

@Oliver Fletcher

Producing user should also have Allow permission on DESCRIBE operation. You can use "--producer" , "--consumer" options of kafka-acls.sh script to give ACLs for producer or consumer role.

[#3]

Can you please let me know the final answer to this issue ported by @Oliver Fletcher I am facing the same issue with HDP 2.6 kerberized cluster with Kafka.

[#4]

@Sugi Narayana I encountered the same issue with HDP 2.6.4, Kerberized Kafka with SimpleAclAuthorizer, and addressed the issue by referring this thread. I used following commands to give an user required privileges for producer and consumer:

# Added to publish
./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=zk-host:2181 --topic topic-name --producer --add --allow-principal User:UserName
# Added to consume
./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=zk-host:2181 --topic topic-name --consumer --group group-name --add --allow-principal User:UserName

Hope this helps.