Incorrect zookeeper ACL settings was throwing No leader exception on Producing/Consuming
Problem Description:
On deleting few kafka topics and creating same topics again there Not Leader for this partition exception while Producing/Consuming and was reading old meta data.
Getting Below error:
17/03/10 10:45:35 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: org.apache.spark.SparkException: Couldn't find leaders for Set([akrnohij-wng-fp9,47], [akrnohij-wng-fp4,224], [akrnohij-wng-fp11,172], [akrnohij-wng-fp1,84], [akrnohij-wng-fp10,117], [akrnohij-wng-fp10,168], [akrnohij-wng-fp4,176], [akrnohij-wng-fp1,136], [akrnohij-wng-fp9,167], [akrnohij-wng-fp2,174], [akrnohij-wng-fp11,74], [akrnohij-wng-fp3,189], [akrnohij-wng-fp11,200], [akrnohij-wng-fp11,168], [akrnohij-wng-fp4,149], [akrnohij-wng-fp7,127], [akrnohij-wng-fp6,39], [akrnohij-wng-fp10,133], [akrnohij-wng-fp9,171], [akrnohij-wng-fp5,175], [akrnohij-wng-fp7,181]
Cause:
The zookeeper ACL for state znode was not correct :
getAcl /brokers/topics/akrnohij-wng-fp4/partitions/135/state
‘auth,’
: cdrwa,
‘world,’anyone
:r
The following property was set in zookeeper env :
-Dzookeeper.skipACL=yes
Solution:
- Deleted the topic manually as deletion was stuck
- Removed the property -Dzookeeper.skipACL=yes
- Restart zookeeper and Kafka services
NOTE- It is dangerous to use -Dzookeeper.skipACL=yes property and instead it is recommended to use kafka service principal if in case you need to delete the znode for kafka topics.
