Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

KNOX with Spark 2.x History UI not working (works with Spark 1.6)

Highlighted

Re: KNOX with Spark 2.x History UI not working (works with Spark 1.6)

Rising Star

I m not sure about what you mean by working earlier. Without setting that parameter proxies are not supported.

Highlighted

Re: KNOX with Spark 2.x History UI not working (works with Spark 1.6)

What I meant was prior to version 2.3 that parameter setting was not required.We have a working version of HDP 2.6.2 without the same. Also coming to the my main question then "what we are doing here by setting the export SPARK_HISTORY_OPTS="-Dspark.ui.proxyBase=/gateway/default/sparkhistory" is a essential requirement of 2.3 code"? Its not a patch which we are applying to temporarily to fix something. Pls confirm. If you have more details or documentation about this variable.Kindly route me to that.

Re: KNOX with Spark 2.x History UI not working (works with Spark 1.6)

Rising Star

I am sorry but I couldn't find proper documentation about spark.ui.proxyBase. Anyway, this property just tells Spark that you are accessing its UI through a proxy and the base path at which the proxy forwards requests to Spark is that one. This is a valid property which has been there for a while, so I am actually a bit surprised that you were able to proxy Spark History server through Knox (or any other proxy) without setting it in HDP 2.6.2.

That is not a patch and it is the supposed way for Spark to work, I confirm that. Though, in HDP 3 that setting will not be needed anymore (if and only if the proxy in front of Spark History Server is Knox or it is providing the header mentioned in the JIRA) thanks to SPARK-24209 and the related Knox work.

Don't have an account?
Coming from Hortonworks? Activate your account here