Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

FranciscoPaco — Fri, 16 Sep 2022 11:27:04 GMT

Hi,

I recently deployed spark on YARN in CDH 5.10.0

When i launch my script, i get this kind of Warning :

17/04/12 16:22:25 WARN net.ScriptBasedMapping: Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py <IP_NodeManager> 
java.io.IOException: Cannot run program "/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py" (in directory "/yarn/nm/usercache/root/appcache/application_1492005737104_0001/container_1492005737104_0001_01_000001"): error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:548)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
	at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
	at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
	at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
	at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
	at org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:38)
	at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:310)
	at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:299)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:299)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:207)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receive$1.applyOrElse(CoarseGrainedSchedulerBackend.scala:126)
	at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:217)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 25 more

Did someone already have this issue please?

Thanks!

Francis

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

FranciscoPaco — Tue, 25 Apr 2017 11:35:54 GMT

Help please...

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

FranciscoPaco — Tue, 25 Apr 2017 15:02:07 GMT

Solved!

I added a HDFS Gateway for Host that execute Spark-shell 🙂

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

FranciscoPaco — Thu, 27 Apr 2017 12:56:24 GMT

MEA CULPA!!!

Not HDFS Gateway but Spark Gateway on node where i wanted to execute spark shell.

By adding this role instance, the node will receive the client configurations 🙂

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Fawze — Fri, 28 Apr 2017 06:28:44 GMT

This is a bug that fixed at 5.10.1

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

FranciscoPaco — Tue, 02 May 2017 09:26:22 GMT

Thanks Fawze!

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

mageru9 — Thu, 11 May 2017 19:48:53 GMT

What is the bug? I don't see it here when searching under Spark.

https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed_in_510.html#fixed_issues_5101

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Fawze — Thu, 11 May 2017 20:20:31 GMT

@mageru9

https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_rn_known_issues.html

question Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py in Archives of Support Questions (Read Only)

Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py

Re: Spark Exception running /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py