HDP 2.6.3.0 spark + python java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found

[#1]

Hi, trying to run simple python script using spark2 and spark-submit in yarn-client mode,

Getting

Heart beat
2018-03-31 15:38:26 INFO  AbstractService:272 - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state STARTED; cause: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
	at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)
	at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)
	at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:152)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.GatewayConnection.run(GatewayConnection.java:214)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
	... 21 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
	... 22 more
2018-03-31 15:38:26 ERROR SparkContext:91 - Error initializing SparkContext.
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found

What do I do wrong?

[#2]

@Sergey Sheypak

Your issue looks very similar to the one mentioned in the SupportKB article: https://community.hortonworks.com/content/supportkb/178800/errorclass-orgapachehadoopyarnclientreque...

So can you please try changing the following and then see if it works:

On the Client Side code updating the yarn-site.xml to use the "org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider" class instead of "org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider"

:

<property>
      <name>yarn.client.failover-proxy-provider</name>
      <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property><br>

.

Or the other option will be to explicitly download the jar which contains class "org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider" and add it to the client classpath.

[#3]

Hi Jay Kumar,

thanks for your reply. I plan to change it during my spark app submission yarn-client mode using

-- conf spark.hadoop.yarn.client.failover-proxy-provider=org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider

what do you think, is it a good idea?

[#4]

@Sergey Sheypak

Good idea, Worth to try that.