question Re: Error while executing hive query on Spark as execution engine in Archives of Support Questions (Read Only)

Error while executing hive query on Spark as execution engine

sangirawat1 — Wed, 27 Jan 2016 21:40:58 GMT

I am using HDP 2.3.2

I used following commands to set spark as hive's execution engine

set hive.execution.engine=spark;

and executed query as

select count(*) from tablename;

I got following error,

java.lang.NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;
        at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<init>(ScalaNumberDeserializersModule.scala:49)
        at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<clinit>(ScalaNumberDeserializersModule.scala)
        at com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule$class.$init$(ScalaNumberDeserializersModule.scala:61)
        at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:19)
        at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:35)
        at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
        at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:78)
        at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
        at org.apache.spark.SparkContext.withScope(SparkContext.scala:681)
        at org.apache.spark.SparkContext.hadoopRDD(SparkContext.scala:956)
        at org.apache.spark.api.java.JavaSparkContext.hadoopRDD(JavaSparkContext.scala:428)
        at org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generateMapInput(SparkPlanGenerator.java:188)
        at org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generateParentTran(SparkPlanGenerator.java:134)
        at org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:106)
        at org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient.execute(LocalHiveSparkClient.java:130)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:64)
        at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:107)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1655)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1414)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;

Please suggest some fixes. thanks

Re: Error while executing hive query on Spark as execution engine

aervits — Wed, 27 Jan 2016 21:55:54 GMT

@sangeeta rawat can you elaborate the need to use spark as hive exec engine? Hortonworks does not support this execution model. Where does Tez fall short for you that you think Spark can help?

Re: Error while executing hive query on Spark as execution engine

nsabharwal — Wed, 27 Jan 2016 22:24:47 GMT

@sangeeta rawat

It's not supported yet. You can leverage sparksql to access hive

Re: Error while executing hive query on Spark as execution engine

nsabharwal — Wed, 27 Jan 2016 22:29:42 GMT

@sangeeta rawat

https://spark.apache.org/docs/1.0.2/sql-programming-guide.html

Re: Error while executing hive query on Spark as execution engine

harshaljoshi_mc — Thu, 28 Jan 2016 12:34:52 GMT

Thanks @Neeraj Sabharwal for quick response 🙂

Re: Error while executing hive query on Spark as execution engine

nsabharwal — Thu, 28 Jan 2016 12:37:00 GMT

@Harshal Joshi

Please accept the best answer to close the thread and good to have you here

Re: Error while executing hive query on Spark as execution engine

harshaljoshi_mc — Thu, 28 Jan 2016 12:40:03 GMT

Thanks @Artem Ervits for quick response. 🙂

Re: Error while executing hive query on Spark as execution engine

sangirawat1 — Thu, 28 Jan 2016 12:43:45 GMT

@Artem Ervits Thanks for the prompt reply, Its not about Tez has fallen short, I was just analysing the difference between Different execution engines for Hive.

Re: Error while executing hive query on Spark as execution engine

sangirawat1 — Thu, 28 Jan 2016 12:45:17 GMT

@Neeraj Sabharwal Thanks for the prompt reply

Re: Error while executing hive query on Spark as execution engine

zssspagobi — Wed, 28 Sep 2016 18:54:39 GMT

I am facing the same issue

et hive.execution.engine=spark;

and executed query as

select count(*) from tablename;

getting same error ,

but select * from table name works fine

so what is the best solution to run

select count(*) from tablename; in hive shell