Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error With Hive Query Running on Spark

Highlighted

Error With Hive Query Running on Spark

New Contributor

I am trying to run a hive query using spark engine the query works when using the map reduce engine but I prefer SPark.


Here is a link to the query.


https://paste.ofcode.org/33QE3uXDGWkdQsQbtsthEn5

Error message below.


I was thinking this message is coming either from a small typo or some misconfiguration with spark and hive.

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:241)

at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:227)

at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:255)

at org.apache.hive.beeline.Commands.executeInternal(Commands.java:989)

at org.apache.hive.beeline.Commands.execute(Commands.java:1180)

at org.apache.hive.beeline.Commands.sql(Commands.java:1094)

at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1180)

at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1013)

at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:922)

at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:518)

at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:226)

at org.apache.hadoop.util.RunJar.main(RunJar.java:141)

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:187)

at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:271)

at org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:439)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:416)

at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:282)

at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:501)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)

at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:747)

at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.getSparkMemoryAndCores(SetSparkReducerParallelism.java:157)

at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:117)

at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)

at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runJoinOptimizations(SparkCompiler.java:313)

at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:124)

at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:101)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10316)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10109)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:560)

at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)

at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1345)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:185)




Thanks

1 REPLY 1

Re: Error With Hive Query Running on Spark

New Contributor

I Resolved the issue by modifying the hive on spark configurations. It wasnt a query error but the hive on spark settings which was causing the issue. Got to be very careful with those.