We are running a Kerberized HDP 3.1, upgraded from HDP 2. In Zeppelin, all our Spark code runs as the zeppelin user. In HDP 2, other users could still view the YARN UI and Spark UI in order to track job progress. However, in HDP 3.1 it seems that only YARN jobs for the logged in user are displayed, while the Spark UI gives a 403. Is it possible to revert to the HDP 2 behaviour?
I can add users to yarn.admin.acl but this is far from ideal, and I haven't found a similar workaround for Spark. Unfortunately we can't use the spark.livy interpreter in Zeppelin because it doesn't support the ZeppelinContext.