kafka send message error

[#1]

After enable ranger kafka plugin, execute command "/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list bigdata001:6667 --topic test5", but when I input content to send message, the result returns as follows:

[2017-03-30 17:06:45,507] WARN Error while fetching metadata with correlation id 0 : {test5=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2017-03-30 17:06:45,507] ERROR Error when sending message to topic test5 with key: null, value: 7 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test5]
[2017-03-30 17:11:45,563] WARN Error while fetching metadata with correlation id 1 : {test5=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient

[#2]

@Zhao Chaofeng

Please login to Ranger UI http://RANGER_HOST:6080/index.html#!/policymanager/resource

Check the policies defined for Kafka (specially for topics) that you are not restricting access.

.

Purely at Kafka level you can check the permissions using the following utility:

# bin/kafka-acls.sh --list --topic <TOPIC_NAME>

.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_secure-kafka-ambari/content/kafka-acl-exa...

[#3]

Yes, I defined a policy in ranger as follows:

[#4]

# bin/kafka-acls.sh --list --topic test5

After executing the command, there is no acls for topic test5.

[#5 · Accepted solution]

@Zhao Chaofeng

Please refer below links and see if it helps :

https://community.hortonworks.com/articles/17059/apache-ranger-and-kafka-1.html

https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin#KafkaPlugin-CanIauthorizeraccesstoKa...?

[#6]

OK, thank you for your reply.

[#7]

@sunil thupakulaI

i am also getting the same error when i try to publish/consume messages.

I have created a topic with one partition, using kafka user , though i have 3 brokers in my kafka cluster, i have also authorized using ranger policy . When ever i alter the topic partitions to more than 1 then i get below error.

ERROR Error when sending message to topic partition_change1 with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [partition_change1]

I am facing this issue for quit sometime, i need to fix this as i have some business impact. ASAP.

[#8]

@sunil thupakula

kafka server logs:

Retrying leaderEpoch request for partition hello-4 as the leader reported an error: CLUSTER_AUTHORIZATION_FAILED (kafka.server.ReplicaFetcherThread) [2018-05-24 10:11:55,890] INFO [ReplicaFetcher replicaId=1004, leaderId=1003, fetcherId=0] Retrying leaderEpoch request for partition hello-4 as the leader reported an error: CLUSTER_AUTHORIZATION_FAILED (kafka.server.ReplicaFetcherThread)

Any idea, why i am not getting cluster authorization to kafka topics ?

[#9]

@sunil thupakula

More Logs :

Kafka Server Logs:

Retrying leaderEpoch request for partition hello-4 as the leader reported an error: CLUSTER_AUTHORIZATION_FAILED (kafka.server.ReplicaFetcherThread) [2018-05-24 10:11:55,890] INFO [ReplicaFetcher replicaId=1004, leaderId=1003, fetcherId=0] Retrying leaderEpoch request for partition hello-4 as the leader reported an error: CLUSTER_AUTHORIZATION_FAILED (kafka.server.ReplicaFetcherThread)

[#10]

Make Sure that HDFS Service checkbox is checked  in ClouderaManager>Kafka>Configuration

For me after i checked HDFS Service it was resolved