question Re: kafka send message error in Archives of Support Questions (Read Only)

kafka send message error

amazon20062007 — Thu, 30 Mar 2017 16:33:45 GMT

After enable ranger kafka plugin, execute command "/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list bigdata001:6667 --topic test5", but when I input content to send message, the result returns as follows:

[2017-03-30 17:06:45,507] WARN Error while fetching metadata with correlation id 0 : {test5=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2017-03-30 17:06:45,507] ERROR Error when sending message to topic test5 with key: null, value: 7 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test5]
[2017-03-30 17:11:45,563] WARN Error while fetching metadata with correlation id 1 : {test5=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient

Re: kafka send message error

jsensharma — Thu, 30 Mar 2017 17:01:18 GMT

@Zhao Chaofeng

Please login to Ranger UI http://RANGER_HOST:6080/index.html#!/policymanager/resource

Check the policies defined for Kafka (specially for topics) that you are not restricting access.

Purely at Kafka level you can check the permissions using the following utility:

# bin/kafka-acls.sh --list --topic <TOPIC_NAME>

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_secure-kafka-ambari/content/kafka-acl-examples.html

Re: kafka send message error

amazon20062007 — Sun, 18 Aug 2019 09:07:45 GMT

Yes, I defined a policy in ranger as follows:

Re: kafka send message error

amazon20062007 — Thu, 30 Mar 2017 17:17:51 GMT

# bin/kafka-acls.sh --list --topic test5

After executing the command, there is no acls for topic test5.

Re: kafka send message error

amankumbare — Fri, 31 Mar 2017 12:14:27 GMT

@Zhao Chaofeng

Please refer below links and see if it helps :

https://community.hortonworks.com/articles/17059/apache-ranger-and-kafka-1.html

https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin#KafkaPlugin-CanIauthorizeraccesstoKafkaoveranon-securechannelviaRanger?

Re: kafka send message error

amazon20062007 — Fri, 31 Mar 2017 12:27:34 GMT

OK, thank you for your reply.

Re: kafka send message error

sunilkumar_thup — Thu, 24 May 2018 02:10:26 GMT

@sunil thupakulaI

i am also getting the same error when i try to publish/consume messages.

I have created a topic with one partition, using kafka user , though i have 3 brokers in my kafka cluster, i have also authorized using ranger policy . When ever i alter the topic partitions to more than 1 then i get below error.

ERROR Error when sending message to topic partition_change1 with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [partition_change1]

I am facing this issue for quit sometime, i need to fix this as i have some business impact. ASAP.

Re: kafka send message error

sunilkumar_thup — Thu, 24 May 2018 19:58:44 GMT

@sunil thupakula

kafka server logs:

Retrying leaderEpoch request for partition hello-4 as the leader reported an error: CLUSTER_AUTHORIZATION_FAILED (kafka.server.ReplicaFetcherThread) [2018-05-24 10:11:55,890] INFO [ReplicaFetcher replicaId=1004, leaderId=1003, fetcherId=0] Retrying leaderEpoch request for partition hello-4 as the leader reported an error: CLUSTER_AUTHORIZATION_FAILED (kafka.server.ReplicaFetcherThread)

Any idea, why i am not getting cluster authorization to kafka topics ?

Re: kafka send message error

sunilkumar_thup — Thu, 24 May 2018 19:58:45 GMT

@sunil thupakula

More Logs :

Kafka Server Logs:

Re: kafka send message error

VenkataAvinash — Sun, 14 Apr 2024 12:00:48 GMT

Make Sure that HDFS Service checkbox is checked in ClouderaManager>Kafka>Configuration

For me after i checked HDFS Service it was resolved