Since we have activated the HA namenode in Ambari 188.8.131.52 we are not able to run queries in spark with sparkhive (sqlcontext) in our kerberized cluster.
It keeps raising this error :
Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, [datanode fqdn]): java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "[datanode fqdn]/[datanode 9 address]"; destination host is: "[namenode fqdn]":8020;
Funny thing is that sometimes it works, for some users , then it fails, then works again etc ...
Note that we are now using Ambari 2.2 and HDP 2.3.4 and it did not solve the issue.