question Re: Spark2 history server UI not opening in HDP 3.1 in Support Questions

Spark2 history server UI not opening in HDP 3.1

sampathkumar_ma — Wed, 28 Aug 2019 17:16:41 GMT

We have installed Spark2 in HDP 3.1 but when we are trying to access spark2 history server UI,we are getting below issue.

HTTP ERROR 403 Problem accessing /. Reason: java.lang.IllegalArgumentException

log: spark-spark-org.apache.spark.deploy.history.HistoryServer-1-xxxxxx.visa.com.out

19/08/28 13:01:21 DEBUG AuthenticationFilter: Request [http://xxxxxxxx:18081/favicon.ico] triggering authentication. handler: class org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler 19/08/28 13:01:21 DEBUG AuthenticationFilter: Authentication exception: java.lang.IllegalArgumentException org.apache.hadoop.security.authentication.client.AuthenticationException: java.lang.IllegalArgumentException at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:306) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:536) at org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:448) at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.spark_project.jetty.server.Server.handle(Server.java:539) at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:275) at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:365) at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:358) at org.apache.hadoop.security.authentication.util.KerberosUtil.getTokenServerName(KerberosUtil.java:291) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:285) ... 22 more

HDP - HDP 3.1 Spark 2.3 Kerberized cluster

Could you please help on this.

Thank you

Re: Spark2 history server UI not opening in HDP 3.1

jsensharma — Wed, 28 Aug 2019 13:14:20 GMT

@sampathkumar_ma

We see the error caused by as following:

Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:275) at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:365)

So can you please let us know which JDK are you using to run the spark?

# ps -ef | grep -i spark # java -version

.Since when are you noticing this error? Any recent changes made to the host/config?

Re: Spark2 history server UI not opening in HDP 3.1

sampathkumar_ma — Wed, 28 Aug 2019 13:24:00 GMT

Hi @jsensharma ,

Please find the java version below:

lrwxrwxrwx. 1 root root 28 Jul 31 16:20 latest -> /usr/java/jdk1.8.0_221-amd64

$ ps -ef | grep spark | grep -v grep /usr/java/jdk1.8.0_221-amd64/bin/java -Dhdp.version=3.1.0.0-78 -cp /usr/hdp/current/spark2-historyserver/conf/:/usr/hdp/current/spark2-historyserver/jars/*:/usr/hdp/3.1.0.0-78/hadoop/conf/ -Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params=type=kerberos,kerberos.principal=HTTP/xxxx.xxxx.com@CORPxx.xxx.COM,kerberos.keytab=/etc/security/keytabs/spnego.service.keytab -Xmx2048m org.apache spark.deploy.history.HistoryServer

$java -version java version "1.8.0_221" Java(TM) SE Runtime Environment (build 1.8.0_221-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.221-b27, mixed mode)

Thanks

Re: Spark2 history server UI not opening in HDP 3.1

sampathkumar_ma — Wed, 28 Aug 2019 13:50:58 GMT

Hi @jsensharma ,

This is the first time we are deploying spark2 in our cluster. Thanks

Re: Spark2 history server UI not opening in HDP 3.1

sampathkumar_ma — Wed, 28 Aug 2019 18:07:28 GMT

Hi @jsensharma,

After changing SPARK_HISTORY_OPTS in Advanced spark2-env as below, spark2 history server UI started working.

Do you think below config change is the fix for this issue? please advise. Thanks

From:

export SPARK_HISTORY_OPTS='-Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal={{spnego_principal}},kerberos.keytab={{spnego_keytab}}"'

To:

export SPARK_HISTORY_OPTS='-Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal={{spnego_principal}},kerberos.keytab={{spnego_keytab}}"'

Re: Spark2 history server UI not opening in HDP 3.1

AKR — Sun, 05 Jan 2020 12:15:05 GMT

Hi,

This seems to be a Kerberos authentication problem. is this issue happens only for a SHS2 UI or does it happens for other URLs too?

You could do a1. kinit username
2. kinit -kt service.keytab
3. send keytab file along with the submit command

Thanks

AKR