Spark2 history server UI not opening in HDP 3.1

[#1]

We have installed Spark2 in HDP 3.1 but when we are trying to access spark2 history server UI,we are getting below issue.

HTTP ERROR 403
Problem accessing /. Reason:
java.lang.IllegalArgumentException

log: spark-spark-org.apache.spark.deploy.history.HistoryServer-1-xxxxxx.visa.com.out

19/08/28 13:01:21 DEBUG AuthenticationFilter: Request [http://xxxxxxxx:18081/favicon.ico] triggering authentication. handler: class org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler
19/08/28 13:01:21 DEBUG AuthenticationFilter: Authentication exception: java.lang.IllegalArgumentException
org.apache.hadoop.security.authentication.client.AuthenticationException: java.lang.IllegalArgumentException
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:306)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:536)
        at org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:448)
        at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.spark_project.jetty.server.Server.handle(Server.java:539)
        at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333)
        at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:365)
        at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:358)
        at org.apache.hadoop.security.authentication.util.KerberosUtil.getTokenServerName(KerberosUtil.java:291)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:285)
        ... 22 more

HDP - HDP 3.1

Spark 2.3

Kerberized cluster

Could you please help on this.

Thank you

[#2]

@sampathkumar_ma 

We see the error caused by as following:

Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:365)

So can you please let us know which JDK are you using to run the spark? 

# ps -ef | grep -i spark
# java -version

.Since when are you noticing this error?  Any recent changes made to the host/config? 

[#3]

Hi @jsensharma ,

Please find the java version below:

lrwxrwxrwx. 1 root root  28 Jul 31 16:20 latest -> /usr/java/jdk1.8.0_221-amd64

$ ps -ef | grep spark | grep -v grep 
/usr/java/jdk1.8.0_221-amd64/bin/java -Dhdp.version=3.1.0.0-78 -cp /usr/hdp/current/spark2-historyserver/conf/:/usr/hdp/current/spark2-historyserver/jars/*:/usr/hdp/3.1.0.0-78/hadoop/conf/ -Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params=type=kerberos,kerberos.principal=HTTP/xxxx.xxxx.com@CORPxx.xxx.COM,kerberos.keytab=/etc/security/keytabs/spnego.service.keytab -Xmx2048m org.apache spark.deploy.history.HistoryServer

$java -version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b27, mixed mode)

 Thanks

[#4]

Hi @jsensharma ,

This is the first time we are deploying spark2 in our cluster. Thanks

[#5 · Accepted solution]

Hi @jsensharma,

After changing SPARK_HISTORY_OPTS in Advanced spark2-env as below, spark2 history server UI started working. 

Do you think below config change is the fix for this issue? please advise. Thanks

From:

export SPARK_HISTORY_OPTS='-Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal={{spnego_principal}},kerberos.keytab={{spnego_keytab}}"'

 To:

export SPARK_HISTORY_OPTS='-Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal={{spnego_principal}},kerberos.keytab={{spnego_keytab}}"'

[#6]

Hi,

This seems to be a Kerberos authentication problem. is this issue happens only for a SHS2 UI or does it happens for other URLs too? 

You could do a1. kinit username
2. kinit -kt service.keytab
3. send keytab file along with the submit command

Thanks

AKR