Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Trying to execute Spark in yarn-cluster

Trying to execute Spark in yarn-cluster

New Contributor

I'm trying to execute Spark with yarn-cluster mode. It works in yarn-client but I get this error with yarn-cluster and I don't know why. 

I guess that it's a bad configuration in Cloudera Manager, but, I have no idea. 

 

I execute spark with this:

 

spark-submit --name "xxx" --class spark.CentralLog --master yarn-cluster --jars $SPARK_CLASSPATH --executor-memory 1g --num-executors 6 --executor-cores 2 /opt/centralLogs/libProject/paas.jar

 

 

2015-12-14 14:38:52,779 [main] INFO org.apache.spark.deploy.yarn.Client - Uploading resource file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar -> hdfs://ose10kafkaelk.novalocal:8020/user/cloud-user/.sparkStaging/application_1450092077590_0017/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar 2015-12-14 14:38:53,911 [main] INFO org.apache.spark.deploy.yarn.Client - Uploading resource file:/opt/centralLogs/libProject/paas.jar -> hdfs://ose10kafkaelk.novalocal:8020/user/cloud-user/.sparkStaging/application_1450092077590_0017/paas.jar 2015-12-14 14:38:54,002 [main] INFO org.apache.spark.deploy.yarn.Client - Deleting staging directory .sparkStaging/application_1450092077590_0017 Exception in thread "main" java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127) at org.apache.hadoop.fs.Path.(Path.java:135) at org.apache.hadoop.fs.Path.(Path.java:94) at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:241) at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:332) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$2.apply(Client.scala:396) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$2.apply(Client.scala:395) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:395) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:393) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:393) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:635) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:124) at org.apache.spark.deploy.yarn.Client.run(Client.scala:914) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:973) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

1 REPLY 1
Highlighted

Re: Trying to execute Spark in yarn-cluster

Cloudera Employee

Hi Guillermo,

 

I've seen similar errors when an extra "/" is found at the end of a path specification.  It seems like the error is happening when your job files are being transferred to HDFS.

 

Start with a simple "pi" job to isolate whether you have a problem with job-specific configuration or cluster-wide configuration:

 

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster /opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 10

 

Nick