Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Spark + LLAP problems after upgrade to HDP 3.0

Labels:
avatar
author-rank berry_osterlund
Rising Star

Created ‎08-17-2018 08:51 AM

I’m upgrading one of our clusters right now to HDP 3.0 and the upgrade itself worked fine. But after the upgrade, I just can’t get Spark with LLAP to work. This is not a new feature for us, as we have been using this for as long as the support have been there.

As there is some changes in the configuration, I’ve followed and change the config according to both
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_hivewarehouse...
and
https://github.com/hortonworks-spark/spark-llap/tree/master

The testcode I’m running is the following

spark-shell --master yarn --deploy-mode client --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar

import com.hortonworks.hwc.HiveWarehouseSession
import com.hortonworks.hwc.HiveWarehouseSession._
val hive = HiveWarehouseSession.session(spark).build()
hive.showDatabases().show(100)

The error I get is the following.

java.lang.RuntimeException:
java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not open
client transport with JDBC Uri: jdbc:hive2://<server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
Could not establish connection to jdbc:hive2:// <server>:10501/;transportMode=http;httpPath=cliservice;auth=delegationToken:
HTTP Response code: 401)

The Hive server show the following

2018-08-17T07:28:50,759
INFO  [HiveServer2-HttpHandler-Pool:
Thread-175]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(146)) -
Could not validate cookie sent, will try to generate a new cookie
2018-08-17T07:28:50,759
INFO  [HiveServer2-HttpHandler-Pool:
Thread-175]: thrift.ThriftHttpServlet
(ThriftHttpServlet.java:doKerberosAuth(399)) - Failed to authenticate with
http/_HOST kerberos principal, trying with hive/_HOST kerberos principal
2018-08-17T07:28:50,760
ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet
(ThriftHttpServlet.java:doKerberosAuth(407)) - Failed to authenticate with
hive/_HOST kerberos principal
2018-08-17T07:28:50,760
ERROR [HiveServer2-HttpHandler-Pool: Thread-175]: thrift.ThriftHttpServlet
(ThriftHttpServlet.java:doPost(210)) - Error:
org.apache.hive.service.auth.HttpAuthenticationException:
java.lang.reflect.UndeclaredThrowableException
  at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:408)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:160)
[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  at
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
[javax.servlet-api-3.1.0.jar:3.1.0]
  at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
[javax.servlet-api-3.1.0.jar:3.1.0]
  at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_112]
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_112]
  at java.lang.Thread.run(Thread.java:745)
[?:1.8.0_112]
Caused by:
java.lang.reflect.UndeclaredThrowableException
  at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
  at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  ... 25 more
Caused by:
org.apache.hive.service.auth.HttpAuthenticationException: Kerberos
authentication failed:
  at org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:464)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  at
org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  at
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
  at
javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
   at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
  at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  ... 25 more
Caused by:
org.ietf.jgss.GSSException: Defective token detected (Mechanism level:
GSSHeader did not find the right tag)
  at
sun.security.jgss.GSSHeader.<init>(GSSHeader.java:97) ~[?:1.8.0_112]
  at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:306)
~[?:1.8.0_112]
  at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
~[?:1.8.0_112]
  at
org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:452)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  at
org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:413)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  at
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
  at
javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
  at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:405)
~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
  ... 25 more

I can see that it complains about the Kerberos ticket, but I do have a valid key in my session. Running any other Kerberos access like beeline works fine from the same session.

Does anybody have any clue about this error?

6,906 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank ewohlstadter
New Contributor

Created ‎08-17-2018 06:24 PM

@Berry Österlund

What setting do you have for spark.security.credentials.hiveserver2.enabled?

Please try setting it to false for client-mode on kerberized cluster.

Also make sure this is set, if it is not already, spark.sql.hive.hiveserver2.jdbc.url.principal.

View solution in original post

5,523 Views
0 Kudos
8 REPLIES 8

avatar
author-rank falbani author-rank
Guru

Created ‎08-17-2018 11:42 AM

@Berry Österlund

I think the problem might be related to some missing configurations, please check you have set all as per:

https://github.com/hortonworks-spark/spark-llap

HTH

5,523 Views
0 Kudos

avatar
author-rank berry_osterlund
Rising Star

Created ‎08-17-2018 12:02 PM

Thanks for the answer. But I have verified those setting atleast ten times now, and they are correct as far as I can see. This cluster worked with Spark + LLAP (even in Livy) with HDP 2.6.5, and most of these settings are the same.

5,523 Views
0 Kudos

avatar
author-rank ewohlstadter
New Contributor

Created ‎08-17-2018 06:24 PM

@Berry Österlund

What setting do you have for spark.security.credentials.hiveserver2.enabled?

Please try setting it to false for client-mode on kerberized cluster.

Also make sure this is set, if it is not already, spark.sql.hive.hiveserver2.jdbc.url.principal.

5,524 Views
0 Kudos

avatar
author-rank berry_osterlund
Rising Star

Created ‎08-20-2018 09:03 AM

Setting spark.security.credentials.hiveserver2.enabled to false solved the problem. I can now use spark with LLAP in both Java and Python. Just R missing now. Will try to find out how to do it there aswell. Thanks for the help!

5,523 Views
0 Kudos

avatar
author-rank Chandra
Expert Contributor

Created ‎12-14-2018 08:24 PM

where is the setting "spark.security.credentials.hiveserver2.enabled" updated, Spark config in Ambari or Hive?

5,523 Views
0 Kudos

avatar
author-rank ewohlstadter
New Contributor

Created ‎12-14-2018 08:32 PM

@chandramouli muthukumaran

This is updated in the Spark2 config

,

@chandramouli muthukumaran

This is updated in Spark2 config.

5,523 Views
0 Kudos

avatar
author-rank Chandra
Expert Contributor

Created ‎12-14-2018 08:39 PM

Thanks Eric Wohlstadter. Do we add this as a custom spark2 defaults propoerty in the config

5,523 Views
0 Kudos

avatar
author-rank ewohlstadter
New Contributor

Created ‎12-15-2018 06:37 PM

@chandramouli muthukumaran

Yes, that's right.

5,523 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streams Messaging - Kubernetes Operator ...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
View More Announcements