Support Questions

Find answers, ask questions, and share your expertise

Error With Hive Query Running on Spark

avatar

I am trying to run a hive query using spark engine the query works when using the map reduce engine but I prefer SPark.


Here is a link to the query.


https://paste.ofcode.org/33QE3uXDGWkdQsQbtsthEn5

Error message below.


I was thinking this message is coming either from a small typo or some misconfiguration with spark and hive.

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:241)

at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:227)

at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:255)

at org.apache.hive.beeline.Commands.executeInternal(Commands.java:989)

at org.apache.hive.beeline.Commands.execute(Commands.java:1180)

at org.apache.hive.beeline.Commands.sql(Commands.java:1094)

at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1180)

at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1013)

at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:922)

at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:518)

at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:226)

at org.apache.hadoop.util.RunJar.main(RunJar.java:141)

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:187)

at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:271)

at org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:439)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:416)

at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:282)

at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:501)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)

at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)

at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:747)

at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.

at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.getSparkMemoryAndCores(SetSparkReducerParallelism.java:157)

at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:117)

at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)

at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)

at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runJoinOptimizations(SparkCompiler.java:313)

at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:124)

at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:101)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10316)

at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10109)

at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:560)

at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)

at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1345)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:185)




Thanks

2 REPLIES 2

avatar

I Resolved the issue by modifying the hive on spark configurations. It wasnt a query error but the hive on spark settings which was causing the issue. Got to be very careful with those.

avatar
Explorer

Facing similar issue.

What are all the parameters did you modified...