Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error With Hive Query Running on Spark

Error With Hive Query Running on Spark

I am trying to run a hive query using spark engine the query works when using the map reduce engine but I prefer SPark.


Here is a link to the query.


https://paste.ofcode.org/33QE3uXDGWkdQsQbtsthEn5

Error message below.


I was thinking this message is coming either from a small typo or some misconfiguration with spark and hive.

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:241)

at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:227)

at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:255)

at org.apache.hive.beeline.Commands.executeInternal(Commands.java:989)

at org.apache.hive.beeline.Commands.execute(Commands.java:1180)

at org.apache.hive.beeline.Commands.sql(Commands.java:1094)

at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1180)

at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1013)

at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:922)

at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:518)

at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:226)

at org.apache.hadoop.util.RunJar.main(RunJar.java:141)

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:187)

at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:271)

at org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:439)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:416)

at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:282)

at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:501)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)

at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:747)

at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.getSparkMemoryAndCores(SetSparkReducerParallelism.java:157)

at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:117)

at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)

at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runJoinOptimizations(SparkCompiler.java:313)

at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:124)

at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:101)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10316)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10109)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:560)

at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)

at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1345)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:185)




Thanks

2 REPLIES 2
Highlighted

Re: Error With Hive Query Running on Spark

I Resolved the issue by modifying the hive on spark configurations. It wasnt a query error but the hive on spark settings which was causing the issue. Got to be very careful with those.

Highlighted

Re: Error With Hive Query Running on Spark

New Contributor

Facing similar issue.

What are all the parameters did you modified...

 

Don't have an account?
Coming from Hortonworks? Activate your account here