Support Questions

Find answers, ask questions, and share your expertise

Spark benchmark with spark-bench tool

While running the spark-bench command ./examples/multi-submit-sparkpi/ from spark bench distribution file, getting this error

Exception in thread "main" java.lang.Exception: spark-submit failed to complete properly given these arguments: --class --master yarn-client /home/dialdev/spark-bench_2.1.1_0.2.2-RELEASE/lib/spark-bench-2.1.1_0.2.2-RELEASE.jar /tmp/spark-bench-3184271760086441425.conf at$.launch(SparkLaunch.scala:47) at$anonfun$run$2.apply(SparkLaunch.scala:39) at$anonfun$run$2.apply(SparkLaunch.scala:39) at scala.collection.immutable.List.foreach(List.scala:381) at$.run(SparkLaunch.scala:39) at$.main(SparkLaunch.scala:16) at

In, the environment variables are set as follows:

export SPARK_HOME=/usr/hdp/

export SPARK_MASTER_HOST=yarn-client (also tried with yarn)

Thanks in advance


@vshukla Please help in this regard.

New Contributor

Are you running Spark 1.6 or Spark 2.1?

If 2.1, change SPARK_HOME=/usr/hdp/current/spark2-client, if 1.6, use ../spark-client instead.

Also, make sure the minimal-example.conf works first before trying the multi-submit-example to make sure your spark-bench environment is fine.