We are actually using Microsoft HDI with ESP but I am wondering if anyone has had similar experience.
From our zeppelin notebook, when I read data to a Dataset (spark.read.text) it works but when I try to read it to an RDD (sc.textFile), I get an authentication exception for failure to find Kerberos tgt
Note that while sc.textFile failed in zeppelin, it works well from spark-shell. Moreover, as a workaround for Zeppelin, we can first read the data to Dataset and convert it to RDD by "spark.read.text(path).rdd".
I looked at some answers here in the community. I added livy.spark.yarn.security.credentials.hiveserver2.enabled but it doesn't make any difference. Some mentioned that we need to enable 'user impersonate' but I got other errors from that and I am not sure if this is the right direction since only certain functions don't work.