We are running a Kerberized HDP 3.1, upgraded from HDP 2. In Zeppelin, all our Spark code runs as the zeppelin user. In HDP 2, other users could still view the YARN UI and Spark UI in order to track job progress. However, in HDP 3.1 it seems that only YARN jobs for the logged in user are displayed, while the Spark UI gives a 403. Is it possible to revert to the HDP 2 behaviour? I can add users to yarn.admin.acl but this is far from ideal, and I haven't found a similar workaround for Spark. Unfortunately we can't use the spark.livy interpreter in Zeppelin because it doesn't support the ZeppelinContext.
... View more
I'm upgrading our cluster from HDP 2.6.3 to 3.1.0. We currently use two separate postgres instances for Ranger and Hive. HDP 3.1.0 no longer supports postgres 9.2, so it needs to be upgraded. My question is when (precisely) should I do this? I've actually already upgraded both to postgres 10.6 (at the same time as I had to for ambari which is a third postgres instance), and upgraded Ambari to 22.214.171.124. However, the Hive metastore is failing to start with an odd postgres error - I think because postgres 10.6 isn't supported by HDP 2.6.3. Shall I revert my postgres upgrade until I have upgraded HDP to 3.1.0? If so, precisely when should I upgrade postgres? Thanks in advance
... View more