Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Issue producing data into Kerborized Kafka cluster

Explorer

Hi,

I'm trying to produce data into a kerborized 3 node kafka cluster with SimpleAclAuthorizer.

When I run:

[root@domain bin]# ./kafka-console-producer.sh --broker-list host.domain.net:6667 --topic topic1 --security-protocol SASL_PLAINTEXT 
Test
[2017-04-11 09:07:43,821] WARN Error while fetching metadata with correlation id 0 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,022] WARN Error while fetching metadata with correlation id 1 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,122] WARN Error while fetching metadata with correlation id 2 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,223] WARN Error while fetching metadata with correlation id 3 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,323] WARN Error while fetching metadata with correlation id 4 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,423] WARN Error while fetching metadata with correlation id 5 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,523] WARN Error while fetching metadata with correlation id 6 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,624] WARN Error while fetching metadata with correlation id 7 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:43,821] WARN Error while fetching metadata with correlation id 0 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,022] WARN Error while fetching metadata with correlation id 1 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,122] WARN Error while fetching metadata with correlation id 2 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,223] WARN Error while fetching metadata with correlation id 3 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,323] WARN Error while fetching metadata with correlation id 4 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,423] WARN Error while fetching metadata with correlation id 5 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,523] WARN Error while fetching metadata with correlation id 6 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2017-04-11 09:07:44,624] WARN Error while fetching metadata with correlation id 7 : {topic1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

I can see my topic if I run:

[root@domain bin]# ./kafka-topics.sh --list --zookeeper host.domain.net:2181
topic1 

I can see the ACL's I've applied:

[root@domain bin]# ./kafka-acls.sh --list --authorizer-properties zookeeper.connect=host.domain.net:2181 --topic topic1
Current ACLs for resource `Topic:topic1`:
User:nifi/host.domain.net has Allow permission for operations: Write from hosts: *
User:nifi/host.domain.net has Allow permission for operations: Write from hosts: * 

I've run kinit:

[root@domain bin]# klist 
Ticket cache: FILE:/tmp/krb5cc_...
Default principal: nifi/host.domain.net@DOMAIN.NET
Valid starting     Expires            Service principal
04/11/17 08:59:42  04/11/17 18:59:42  krbtgt/DOMAIN.NET@DOMAIN.NET
        renew until 04/18/17 08:59:42 Default principal: nifi/host.domain.net@DOMAIN.NET
Valid starting     Expires            Service principal
04/11/17 08:59:42  04/11/17 18:59:42  krbtgt/DOMAIN.NET@DOMAIN.NET
        renew until 04/18/17 08:59:42 

Thanks in advance for any help.

Ollie

3 REPLIES 3

Explorer
@Oliver Fletcher

Producing user should also have Allow permission on DESCRIBE operation. You can use "--producer" , "--consumer" options of kafka-acls.sh script to give ACLs for producer or consumer role.

New Contributor

Can you please let me know the final answer to this issue ported by @Oliver Fletcher I am facing the same issue with HDP 2.6 kerberized cluster with Kafka.

@Sugi Narayana I encountered the same issue with HDP 2.6.4, Kerberized Kafka with SimpleAclAuthorizer, and addressed the issue by referring this thread. I used following commands to give an user required privileges for producer and consumer:

# Added to publish
./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=zk-host:2181 --topic topic-name --producer --add --allow-principal User:UserName
# Added to consume
./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=zk-host:2181 --topic topic-name --consumer --group group-name --add --allow-principal User:UserName

Hope this helps.