Spark on hive fails with java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job

[#1]

Hi,

All my hive jobs with spark as execution engine ( in fact all services that use spark for execution) are failing with below error. Hive version is 1.2.1000 and spark version is 1.6.2 i'm using hdp 2.5.

2017-06-02 11:48:10,090 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/02 11:48:10 INFO Client: Application report for application_1496301296700_0022 (state: RUNNING) 2017-06-02 11:48:11,092 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/02 11:48:11 INFO Client: Application report for application_1496301296700_0022 (state: RUNNING) 2017-06-02 11:48:11,395 WARN [RPC-Handler-3]: rpc.RpcDispatcher (RpcDispatcher.java:handleError(142)) - Received error message:io.netty.handler.codec.DecoderException: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:358) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230) at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at org.apache.hive.spark.client.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:96) at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:327) ... 15 more Caused by: java.lang.ClassNotFoundException: org.apache.hive.spark.client.Job at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 39 more . 2017-06-02 11:48:11,398 WARN [RPC-Handler-3]: client.SparkClientImpl (SparkClientImpl.java:rpcClosed(116)) - Client RPC channel closed unexpectedly.

[#2]

@Rajesh Reddy, As per stack trace , it looks like spark job does not find proper hive Jars. You can try couple of solutions to resolve this issue.

1) Copy hive-site.xml to $SPARK_HOME/conf dir ( or /etc/spark/conf)

2) set spark.driver.extraClassPath and point to hive-exec-*.jar

[#3]

@yvora i have made the mentioned changes now the error has gone but no spark job is launched. There is no spark job in spark UI.

hive> set hive.execution.engine=spark;

hive> set spark.driver.extraClassPath=/usr/hdp/current/hive-client/lib/hive-exec.jar ;

hive> select count(*) from atlas.employee;

Query ID = root_20170607172759_1ebd1bdc-cac5-4e20-a012-2242ddf4e753 Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Spark Job = caa2546e-7379-48ed-b2b8-b0055f7e49ab

Status: Failed FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

hive> exit;

Hive.log

2017-06-07 17:28:48,769 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:48 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:49,771 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:49 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:50,773 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:50 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:51,762 INFO [RPC-Handler-3]: client.SparkClientImpl (SparkClientImpl.java:handle(522)) - Received result for caa2546e-7379-48ed-b2b8-b0055f7e49ab 2017-06-07 17:28:51,776 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:51 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:52,691 ERROR [main]: status.SparkJobMonitor (SessionState.java:printError(989)) - Status: Failed 2017-06-07 17:28:52,691 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=SparkRunJob start=1496836708670 end=1496836732691 duration=24021 from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor> 2017-06-07 17:28:52,751 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-06-07 17:28:52,751 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,752 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1496836732751 end=1496836732752 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,752 ERROR [main]: ql.Driver (SessionState.java:printError(989)) - FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask 2017-06-07 17:28:52,753 INFO [main]: ql.Driver (Driver.java:execute(1635)) - Resetting the caller context to 2017-06-07 17:28:52,753 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=Driver.execute start=1496836681184 end=1496836732753 duration=51569 from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,753 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,753 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1496836732753 end=1496836732753 duration=0 from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,778 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:52 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING)

[#4]

@Rajesh Reddy, can you please enable debug level and retry? As stated in below link, the issue may be related to jar conflict.

https://stackoverflow.com/questions/37135532/hive-on-spark-cdh5-7-execution-error

[#5]

Hi @yvora, I have enabled debug. The yarn application log shows below error.

17/06/08 09:51:49 WARN Rpc: Invalid log level null, reverting to default.

17/06/08 09:51:50 ERROR ApplicationMaster: User class threw exception: java.util.concurrent.ExecutionException: javax.security.sasl.SaslExceptio n: Client closed before SASL negotiation finished. java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: Client closed before SASL negotiation finished. at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:156) at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$anon$2.run(ApplicationMaster.scala:559) Caused by: javax.security.sasl.SaslException: Client closed before SASL negotiation finished. at org.apache.hive.spark.client.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:449) at org.apache.hive.spark.client.rpc.SaslHandler.channelInactive(SaslHandler.java:90) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.hive.spark.client.rpc.KryoMessageCodec.channelInactive(KryoMessageCodec.java:127) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828) at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 17/06/08 09:51:50 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.util.concurrent.Exec utionException: javax.security.sasl.SaslException: Client closed before SASL negotiation finished.) 17/06/08 09:51:59 ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.

[#6]

@Rajesh Reddy , Looks like you are hitting https://issues.apache.org/jira/browse/HIVE-12649.

[#7]

Hi @yvora thanks for the details. Is there any possibility of specifying hive to use spark2 rather than spark??

[#8]

@Rajesh Reddy, you can follow below document to use spark2 with hive.

https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

[#9]

@Rajesh Reddy, Can you please accept this answer if above conservation helped you ?

[#10]

@yvora I get the same error, and tryed to do your recomadtion, but get the same error described by @Rajesh Reddy.

The firts option you sugested doesn't work, because when you restart the spark a new conf is put with only with connection of hive.metastore.uris. Any one get a solution of integration Hive on Spark?

Thanks.