Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Am getting GroupAuthorizationException: Not authorized to access group: spark-executor error while running SPARK-KAFKA consumer

avatar
New Contributor

Hello Team,

 

am able to produce message to topic, but getting below error while running spark-kafka consumer , using com.sun.security.auth.module.Krb5LoginModule , jaas & krb5.conf file.

running code through eclipse ,

am able to consume message when i run in terminal to using /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --topic <tn> --from-beginning --bootstrap-server <> --consumer.config <path>/consumer.properties

exact error message :

ERROR org.apache.spark.executor.Executor - Exception in task 2.0 in stage 0.0 (TID 2)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-executor-<groupID>

Env Used :

compile(group: 'org.apache.spark', name: 'spark-streaming_2.11', version: '2.3.1')
compile(group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: '2.3.1')

compile(group: 'org.apache.spark' , name: 'spark-core_2.11', version: '2.3.2')

scala version : 2.11.8

 

val kafkaParam = Map[String, Object](
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> broker,
ConsumerConfig.GROUP_ID_CONFIG -> groupId,
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer",
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"
,CommonClientConfigs.SECURITY_PROTOCOL_CONFIG -> "SASL_PLAINTEXT"
)

 

 

 

1 REPLY 1

avatar
Expert Contributor

Hello @Venkat451 ,

 

Thank you for posting your query.

 

From the error message shared (as below) I see the executor failing while it is trying to attach itself with the consumer group, more specifically, it is getting Authorisation exception while attaching to the group.

 

ERROR org.apache.spark.executor.Executor - Exception in task 2.0 in stage 0.0 (TID 2)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-executor-<groupID>

If you have authorisation mechanisms such as (sentry, kafka ACLs, Ranger) enabled on your cluster please grant necessary permissions to the consumer group

 

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/kafka-acl-examples.html

Thanks,
Satz