Hi, We are running Spark Thrift Server on HDP .2.6.3.0-235. Sometimes it goes down w/o obvious reason and I would like to find out why. I see that yarn app is killed by someone (which is fair, someone could kill it.), but the whole service goes down if YARN app goes down? Is it by deesign? 18/04/16 12:37:20 INFO SessionState: Created HDFS directory: /tmp/hive/hive/8fa3df5f-73ef-4c31-9a27-d9e22334579f/_tmp_space.db 18/04/16 12:37:20 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/home/hive/spark-warehouse 18/04/16 12:37:49 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state KILLED! 18/04/16 12:37:49 INFO HiveServer2: Shutting down HiveServer2 18/04/16 12:37:49 INFO ThriftCLIService: Thrift server has stopped 18/04/16 12:37:49 INFO AbstractService: Service:ThriftBinaryCLIService is stopped. 18/04/16 12:37:49 INFO AbstractService: Service:OperationManager is stopped. 18/04/16 12:37:49 INFO AbstractService: Service:SessionManager is stopped. 18/04/16 12:37:49 INFO AbstractService: Service:CLIService is stopped. 18/04/16 12:37:49 INFO AbstractService: Service:HiveServer2 is stopped. 18/04/16 12:37:49 INFO AbstractConnector: Stopped Spark@47457a81{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 18/04/16 12:37:49 INFO SparkUI: Stopped Spark web UI at http://185.204.3.180:4040 18/04/16 12:37:49 ERROR TransportClient: Failed to send RPC 5517988194331422796 to /185.204.3.100:50030: java.nio.channels.ClosedChannelException java.nio.channels.ClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) 18/04/16 12:37:49 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful java.io.IOException: Failed to send RPC 5517988194331422796 to /185.204.3.100:50030: java.nio.channels.ClosedChannelException at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34) at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.channels.ClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) 18/04/16 12:37:49 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 18/04/16 12:37:49 ERROR Utils: Uncaught exception in thread Yarn application state monitor org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:551) at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:94) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:151) at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:517) at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1670) at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1928) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1317) at org.apache.spark.SparkContext.stop(SparkContext.scala:1927) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:108) Caused by: java.io.IOException: Failed to send RPC 5517988194331422796 to /185.204.3.100:50030: java.nio.channels.ClosedChannelException at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34) at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.channels.ClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) 18/04/16 12:37:49 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/04/16 12:37:49 INFO MemoryStore: MemoryStore cleared 18/04/16 12:37:49 INFO BlockManager: BlockManager stopped 18/04/16 12:37:49 INFO BlockManagerMaster: BlockManagerMaster stopped 18/04/16 12:37:49 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/04/16 12:37:49 INFO SparkContext: Successfully stopped SparkContext ... View more

Hi, trying to run simple python script using spark2 and spark-submit in yarn-client mode, Getting Heart beat 2018-03-31 15:38:26 INFO AbstractService:272 - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state STARTED; cause: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227) at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161) at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94) at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:152) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164) at org.apache.spark.SparkContext.<init>(SparkContext.scala:500) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:238) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219) ... 21 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) ... 22 more 2018-03-31 15:38:26 ERROR SparkContext:91 - Error initializing SparkContext. java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found What do I do wrong? ... View more

Hi, I need to run spark job from oozie java action as regular java application with main class. I can't use oozie spark or shell actino for many reasons. I've built jar with spark-yarn 2.2 hdp 2.6.3.0 dependecy. I can easilly run it using spark-submit. I try to run same jar using same main class as oozie java action and get exception: Caused by: java.lang.IllegalStateException: Library directory '/hadoop/yarn/local/usercache/root/appcache/application_1521375636129_0022/container_e06_1521375636129_0022_01_000002/assembly/target/scala-2.11/jars' does not exist; make sure Spark is built. at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:260) at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:359) at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38) I took a look at the source code, there are some weird checks.... What is the right way to disable them? ... View more

Hi, I'm trying to create hive table and query it using %sql. I tried both: %spark and %sql, nothing helps. Here is my DDL: %sql create external table MY_TABLE row format serde 'com.my.MyAvroSerde' with serdeproperties ('serialization.class'='com.my.ContainerProto') stored as inputformat 'com.my.ProtoAvroFileFormat' LOCATION 'hdfs://my/data' Thrown exception: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException java.lang.ClassNotFoundException: Class com.my.ContainerProto not found) It's confusing since spark paragraph works well with code %spark import com.my.ContainerProto // bla-bla val rdd = sc.newAPIHadoopFile[AvroKey[ByteBuffer], NullWritable, AvroKeyInputFormat[ByteBuffer]]("hdfs://my/data") rdd.map{bytes => ContainerProto.fromBytes(bytes)} Code executed and result produced. Why sql or spark paragraph doesn't see my 3rd party jars when I try to create hive table? Spark interpreter has proper configuration for required third party jars. ... View more