Incorrect zookeeper ACL settings was throwing No leader exception on Producing/Consuming

[#1]

Problem Description:

On deleting few kafka topics and creating same topics again there Not Leader for this partition exception while Producing/Consuming and was reading old meta data.

Getting Below error:

17/03/10 10:45:35 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: org.apache.spark.SparkException: Couldn't find leaders for Set([akrnohij-wng-fp9,47], [akrnohij-wng-fp4,224], [akrnohij-wng-fp11,172], [akrnohij-wng-fp1,84], [akrnohij-wng-fp10,117], [akrnohij-wng-fp10,168], [akrnohij-wng-fp4,176], [akrnohij-wng-fp1,136], [akrnohij-wng-fp9,167], [akrnohij-wng-fp2,174], [akrnohij-wng-fp11,74], [akrnohij-wng-fp3,189], [akrnohij-wng-fp11,200], [akrnohij-wng-fp11,168], [akrnohij-wng-fp4,149], [akrnohij-wng-fp7,127], [akrnohij-wng-fp6,39], [akrnohij-wng-fp10,133], [akrnohij-wng-fp9,171], [akrnohij-wng-fp5,175], [akrnohij-wng-fp7,181] 

Cause:

The zookeeper ACL for state znode was not correct :

getAcl /brokers/topics/akrnohij-wng-fp4/partitions/135/state 

‘auth,’ 

: cdrwa, 

‘world,’anyone 

:r

The following property was set in zookeeper env :

-Dzookeeper.skipACL=yes

Solution:

1. Deleted the topic manually as deletion was stuck
2. Removed the property -Dzookeeper.skipACL=yes
3. Restart zookeeper and Kafka services

NOTE- It is dangerous to use -Dzookeeper.skipACL=yes property and instead it is recommended to use kafka service principal if in case you need to delete the znode for kafka topics.