question Re: Issue with Spark on Hive. in Archives of Support Questions (Read Only)

Issue with Spark on Hive.

visakh_nair — Tue, 03 Nov 2015 15:17:31 GMT

I can run Queries using hive with the map reduce engine, but when running hive on spark it fails giving an expection:

Failed to execute spark task, with exception: 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'

Re: Issue with Spark on Hive.

jstraub — Tue, 03 Nov 2015 16:50:32 GMT

What is your HDP, Spark and Hive version?

Re: Issue with Spark on Hive.

nsabharwal — Tue, 03 Nov 2015 20:46:23 GMT

@visakh.nair@hortonworks.com

I am able to reproduce this

HDP 2.3.2

https://issues.apache.org/jira/browse/HIVE-9970

and https://issues.apache.org/jira/browse/HIVE-7980

I am working on finding the fix

Re: Issue with Spark on Hive.

nsabharwal — Tue, 03 Nov 2015 23:02:07 GMT

15/11/03 06:56:07 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
	

15/11/03 06:56:17 INFO util.Utils: Shutdown hook called

Re: Issue with Spark on Hive.

nsabharwal — Wed, 04 Nov 2015 01:47:49 GMT

@visakh.nair@hortonworks.com

Please see this "Work in progress"

Re: Issue with Spark on Hive.

vshukla — Wed, 04 Nov 2015 05:04:47 GMT

We don't support Spark on Hive, try Hive on Tez.

We do plan to support SparkSQL when Spark 1.5.1 becomes GA on HDP.

Re: Issue with Spark on Hive.

umaravikumar — Thu, 16 Jun 2016 23:26:26 GMT

Do you have any update on this issue : 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'

Re: Issue with Spark on Hive.

saravanan_c_moh — Wed, 06 Jul 2016 19:21:22 GMT

Do you know whether hive on spark supported in HDP2.4 and above with Hive as 1.2

Re: Issue with Spark on Hive.

vshukla — Wed, 06 Jul 2016 20:51:45 GMT

We still don't support Hive on Spark. We support Hive on Tez.

We also support SparkSQL.

Re: Issue with Spark on Hive.

m_a_vervuurt — Wed, 06 Jul 2016 22:22:40 GMT

I can indeed as vshukla is saying recommend using SparkSQL with the HiveSqlContext.