I’m
upgrading one of our clusters right now to HDP 3.0 and the upgrade itself
worked fine. But after the upgrade, I just can’t get Spark with LLAP to work. This
is not a new feature for us, as we have been using this for as long as the
support have been there. As there is
some changes in the configuration, I’ve followed and change the config
according to both
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_hivewarehouseconnector_for_handling_apache_spark_data.html
and
https://github.com/hortonworks-spark/spark-llap/tree/master The
testcode I’m running is the following spark-shell
--master yarn --deploy-mode client --jars
/usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar import com.hortonworks.hwc.HiveWarehouseSession
import com.hortonworks.hwc.HiveWarehouseSession._
val hive = HiveWarehouseSession.session(spark).build()
hive.showDatabases().show(100) The error I
get is the following. java.lang.RuntimeException:
java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not open
client transport with JDBC Uri: jdbc:hive2://<server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
Could not establish connection to jdbc:hive2:// <server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
HTTP Response code: 401) The Hive
server show the following 2018-08-17T07:28:50,759
INFO [HiveServer2-HttpHandler-Pool:
Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(146)) -
Could not validate cookie sent, will try to generate a new cookie 2018-08-17T07:28:50,759
INFO [HiveServer2-HttpHandler-Pool:
Thread-175]: thrift.ThriftHttpServlet
(ThriftHttpServlet.java:doKerberosAuth(399)) - Failed to authenticate with
http/_HOST kerberos principal, trying with hive/_HOST kerberos principal 2018-08-17T07:28:50,760
ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet
(ThriftHttpServlet.java:doKerberosAuth(407)) - Failed to authenticate with
hive/_HOST kerberos principal 2018-08-17T07:28:50,760
ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet
(ThriftHttpServlet.java:doPost(210)) - Error: org.apache.hive.service.auth.HttpAuthenticationException:
java.lang.reflect.UndeclaredThrowableException at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:408)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:160)
[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
[javax.servlet-api-3.1.0.jar:3.1.0] at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
[javax.servlet-api-3.1.0.jar:3.1.0] at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_112] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_112] at java.lang.Thread.run(Thread.java:745)
[?:1.8.0_112] Caused by:
java.lang.reflect.UndeclaredThrowableException at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] ... 25 more Caused by:
org.apache.hive.service.auth.HttpAuthenticationException: Kerberos
authentication failed: at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:464)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at
org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112] at
javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] ... 25 more Caused by:
org.ietf.jgss.GSSException: Defective token detected (Mechanism level:
GSSHeader did not find the right tag) at
sun.security.jgss.GSSHeader.<init>(GSSHeader.java:97) ~[?:1.8.0_112] at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:306)
~[?:1.8.0_112] at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
~[?:1.8.0_112] at
org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:452)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at
org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112] at
javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] ... 25 more I can see
that it complains about the Kerberos ticket, but I do have a valid key in my
session. Running any other Kerberos access like beeline works fine from the
same session. Does
anybody have any clue about this error?
... View more
