Am getting GroupAuthorizationException: Not authorized to access group: spark-executor error while running SPARK-KAFKA consumer

[#1]

Hello Team,

am able to produce message to topic, but getting below error while running spark-kafka consumer , using com.sun.security.auth.module.Krb5LoginModule , jaas & krb5.conf file.

running code through eclipse ,

am able to consume message when i run in terminal to using /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --topic <tn> --from-beginning --bootstrap-server <> --consumer.config <path>/consumer.properties

exact error message :

ERROR org.apache.spark.executor.Executor - Exception in task 2.0 in stage 0.0 (TID 2)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-executor-<groupID>

Env Used :

compile(group: 'org.apache.spark', name: 'spark-streaming_2.11', version: '2.3.1')
compile(group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: '2.3.1')

compile(group: 'org.apache.spark' , name: 'spark-core_2.11', version: '2.3.2')

scala version : 2.11.8

val kafkaParam = Map[String, Object](
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> broker,
ConsumerConfig.GROUP_ID_CONFIG -> groupId,
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer",
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"
,CommonClientConfigs.SECURITY_PROTOCOL_CONFIG -> "SASL_PLAINTEXT"
)

[#2]

Hello @Venkat451 ,

Thank you for posting your query.

From the error message shared (as below) I see the executor failing while it is trying to attach itself with the consumer group, more specifically, it is getting Authorisation exception while attaching to the group.

ERROR org.apache.spark.executor.Executor - Exception in task 2.0 in stage 0.0 (TID 2)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-executor-<groupID>

If you have authorisation mechanisms such as (sentry, kafka ACLs, Ranger) enabled on your cluster please grant necessary permissions to the consumer group

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/kafka-acl-examples.html