Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark + LLAP problems after upgrade to HDP 3.0

avatar
Rising Star

I’m upgrading one of our clusters right now to HDP 3.0 and the upgrade itself worked fine. But after the upgrade, I just can’t get Spark with LLAP to work. This is not a new feature for us, as we have been using this for as long as the support have been there.

As there is some changes in the configuration, I’ve followed and change the config according to both
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_hivewarehouse...
and
https://github.com/hortonworks-spark/spark-llap/tree/master

The testcode I’m running is the following

spark-shell --master yarn --deploy-mode client --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar

import com.hortonworks.hwc.HiveWarehouseSession
import com.hortonworks.hwc.HiveWarehouseSession._
val hive = HiveWarehouseSession.session(spark).build()
hive.showDatabases().show(100)

The error I get is the following.

java.lang.RuntimeException:
java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not open
client transport with JDBC Uri: jdbc:hive2://<server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
Could not establish connection to jdbc:hive2:// <server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
HTTP Response code: 401)

The Hive server show the following

2018-08-17T07:28:50,759
INFO  [HiveServer2-HttpHandler-Pool:
Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(146)) -
Could not validate cookie sent, will try to generate a new cookie
2018-08-17T07:28:50,759 INFO [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(399)) - Failed to authenticate with http/_HOST kerberos principal, trying with hive/_HOST kerberos principal
2018-08-17T07:28:50,760 ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(407)) - Failed to authenticate with hive/_HOST kerberos principal
2018-08-17T07:28:50,760 ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(210)) - Error:
org.apache.hive.service.auth.HttpAuthenticationException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:408) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:160) [hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
... 25 more
Caused by: org.apache.hive.service.auth.HttpAuthenticationException: Kerberos authentication failed:
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:464) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
... 25 more
Caused by: org.ietf.jgss.GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
at sun.security.jgss.GSSHeader.<init>(GSSHeader.java:97) ~[?:1.8.0_112]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:306) ~[?:1.8.0_112]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_112]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:452) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
... 25 more

I can see that it complains about the Kerberos ticket, but I do have a valid key in my session. Running any other Kerberos access like beeline works fine from the same session.

Does anybody have any clue about this error?

1 ACCEPTED SOLUTION

avatar
New Contributor

@Berry Österlund

What setting do you have for spark.security.credentials.hiveserver2.enabled?

Please try setting it to false for client-mode on kerberized cluster.

Also make sure this is set, if it is not already, spark.sql.hive.hiveserver2.jdbc.url.principal.

View solution in original post

8 REPLIES 8

avatar

@Berry Österlund

I think the problem might be related to some missing configurations, please check you have set all as per:

https://github.com/hortonworks-spark/spark-llap

HTH

avatar
Rising Star

Thanks for the answer. But I have verified those setting atleast ten times now, and they are correct as far as I can see. This cluster worked with Spark + LLAP (even in Livy) with HDP 2.6.5, and most of these settings are the same.

avatar
New Contributor

@Berry Österlund

What setting do you have for spark.security.credentials.hiveserver2.enabled?

Please try setting it to false for client-mode on kerberized cluster.

Also make sure this is set, if it is not already, spark.sql.hive.hiveserver2.jdbc.url.principal.

avatar
Rising Star

Setting spark.security.credentials.hiveserver2.enabled to false solved the problem. I can now use spark with LLAP in both Java and Python. Just R missing now. Will try to find out how to do it there aswell. Thanks for the help!

avatar
Expert Contributor

where is the setting "spark.security.credentials.hiveserver2.enabled" updated, Spark config in Ambari or Hive?

avatar
New Contributor

@chandramouli muthukumaran

This is updated in the Spark2 config

,

@chandramouli muthukumaran

This is updated in Spark2 config.

avatar
Expert Contributor

Thanks Eric Wohlstadter. Do we add this as a custom spark2 defaults propoerty in the config

avatar
New Contributor

@chandramouli muthukumaran

Yes, that's right.