I m not sure about what you mean by working earlier. Without setting that parameter proxies are not supported.
What I meant was prior to version 2.3 that parameter setting was not required.We have a working version of HDP 2.6.2 without the same. Also coming to the my main question then "what we are doing here by setting the export SPARK_HISTORY_OPTS="-Dspark.ui.proxyBase=/gateway/default/sparkhistory" is a essential requirement of 2.3 code"? Its not a patch which we are applying to temporarily to fix something. Pls confirm. If you have more details or documentation about this variable.Kindly route me to that.
I am sorry but I couldn't find proper documentation about spark.ui.proxyBase. Anyway, this property just tells Spark that you are accessing its UI through a proxy and the base path at which the proxy forwards requests to Spark is that one. This is a valid property which has been there for a while, so I am actually a bit surprised that you were able to proxy Spark History server through Knox (or any other proxy) without setting it in HDP 2.6.2.
That is not a patch and it is the supposed way for Spark to work, I confirm that. Though, in HDP 3 that setting will not be needed anymore (if and only if the proxy in front of Spark History Server is Knox or it is providing the header mentioned in the JIRA) thanks to SPARK-24209 and the related Knox work.