Created 07-31-2019 03:16 PM
Hi,
I am getting the following exception on one node when submitting a spark job, it is working fine if submitted from any other node. Any help will be highly appreciated.
19/07/31 15:56:19 INFO Executor: Starting executor ID driver on host localhost 19/07/31 15:56:19 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35492. 19/07/31 15:56:19 INFO NettyBlockTransferService: Server created on hdata5.dom.local:35492 19/07/31 15:56:19 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 19/07/31 15:56:19 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hdata5.dom.local, 35492, None) 19/07/31 15:56:19 INFO BlockManagerMasterEndpoint: Registering block manager hdata5.dom.local:35492 with 366.3 MB RAM, BlockManagerId(driver, hdata5.dom.local, 35492, None) 19/07/31 15:56:19 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hdata5.dom.local, 35492, None) 19/07/31 15:56:19 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, hdata5.dom.local, 35492, None) Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/Class;)Lorg/apache/hadoop/io/retry/RetryPolicy; at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:318) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:235) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:468) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1897) at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:74) at org.apache.spark.SparkContext.<init>(SparkContext.scala:520) at com.dom.pipeline.spark.MergeDom$.main(MergeDom.scala:100) at com.dom.pipeline.spark.MergeDom.main(MergeDom.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 19/07/31 15:56:20 INFO DiskBlockManager: Shutdown hook called 19/07/31 15:56:20 INFO ShutdownHookManager: Shutdown hook called 19/07/31 15:56:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-9dc84539-36a1-40b3-b991-ccf28473cea1