Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark + LLAP problems after upgrade to HDP 3.0

SOLVED Go to solution

Spark + LLAP problems after upgrade to HDP 3.0

Contributor

I’m upgrading one of our clusters right now to HDP 3.0 and the upgrade itself worked fine. But after the upgrade, I just can’t get Spark with LLAP to work. This is not a new feature for us, as we have been using this for as long as the support have been there.

As there is some changes in the configuration, I’ve followed and change the config according to both
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_hivewarehouse...
and
https://github.com/hortonworks-spark/spark-llap/tree/master

The testcode I’m running is the following

spark-shell --master yarn --deploy-mode client --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar

import com.hortonworks.hwc.HiveWarehouseSession
import com.hortonworks.hwc.HiveWarehouseSession._
val hive = HiveWarehouseSession.session(spark).build()
hive.showDatabases().show(100)

The error I get is the following.

java.lang.RuntimeException:
java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not open
client transport with JDBC Uri: jdbc:hive2://<server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
Could not establish connection to jdbc:hive2:// <server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
HTTP Response code: 401)

The Hive server show the following

2018-08-17T07:28:50,759
INFO  [HiveServer2-HttpHandler-Pool:
Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(146)) -
Could not validate cookie sent, will try to generate a new cookie
2018-08-17T07:28:50,759 INFO [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(399)) - Failed to authenticate with http/_HOST kerberos principal, trying with hive/_HOST kerberos principal
2018-08-17T07:28:50,760 ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(407)) - Failed to authenticate with hive/_HOST kerberos principal
2018-08-17T07:28:50,760 ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(210)) - Error:
org.apache.hive.service.auth.HttpAuthenticationException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:408) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:160) [hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
... 25 more
Caused by: org.apache.hive.service.auth.HttpAuthenticationException: Kerberos authentication failed:
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:464) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
... 25 more
Caused by: org.ietf.jgss.GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
at sun.security.jgss.GSSHeader.<init>(GSSHeader.java:97) ~[?:1.8.0_112]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:306) ~[?:1.8.0_112]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_112]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:452) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
... 25 more

I can see that it complains about the Kerberos ticket, but I do have a valid key in my session. Running any other Kerberos access like beeline works fine from the same session.

Does anybody have any clue about this error?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Spark + LLAP problems after upgrade to HDP 3.0

New Contributor

@Berry Österlund

What setting do you have for spark.security.credentials.hiveserver2.enabled?

Please try setting it to false for client-mode on kerberized cluster.

Also make sure this is set, if it is not already, spark.sql.hive.hiveserver2.jdbc.url.principal.

8 REPLIES 8

Re: Spark + LLAP problems after upgrade to HDP 3.0

@Berry Österlund

I think the problem might be related to some missing configurations, please check you have set all as per:

https://github.com/hortonworks-spark/spark-llap

HTH

Re: Spark + LLAP problems after upgrade to HDP 3.0

Contributor

Thanks for the answer. But I have verified those setting atleast ten times now, and they are correct as far as I can see. This cluster worked with Spark + LLAP (even in Livy) with HDP 2.6.5, and most of these settings are the same.

Re: Spark + LLAP problems after upgrade to HDP 3.0

New Contributor

@Berry Österlund

What setting do you have for spark.security.credentials.hiveserver2.enabled?

Please try setting it to false for client-mode on kerberized cluster.

Also make sure this is set, if it is not already, spark.sql.hive.hiveserver2.jdbc.url.principal.

Re: Spark + LLAP problems after upgrade to HDP 3.0

Contributor

Setting spark.security.credentials.hiveserver2.enabled to false solved the problem. I can now use spark with LLAP in both Java and Python. Just R missing now. Will try to find out how to do it there aswell. Thanks for the help!

Re: Spark + LLAP problems after upgrade to HDP 3.0

Contributor

where is the setting "spark.security.credentials.hiveserver2.enabled" updated, Spark config in Ambari or Hive?

Re: Spark + LLAP problems after upgrade to HDP 3.0

New Contributor

@chandramouli muthukumaran

This is updated in the Spark2 config

,

@chandramouli muthukumaran

This is updated in Spark2 config.

Highlighted

Re: Spark + LLAP problems after upgrade to HDP 3.0

Contributor

Thanks Eric Wohlstadter. Do we add this as a custom spark2 defaults propoerty in the config

Re: Spark + LLAP problems after upgrade to HDP 3.0

New Contributor

@chandramouli muthukumaran

Yes, that's right.