Spark benchmark with spark-bench tool

[#1]

While running the spark-bench command ./examples/multi-submit-sparkpi/multi-submit-example.sh from spark bench distribution file, getting this error

Exception in thread "main" java.lang.Exception: spark-submit failed to complete properly given these arguments: --class com.ibm.sparktc.sparkbench.cli.CLIKickoff --master yarn-client /home/dialdev/spark-bench_2.1.1_0.2.2-RELEASE/lib/spark-bench-2.1.1_0.2.2-RELEASE.jar /tmp/spark-bench-3184271760086441425.conf at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.launch(SparkLaunch.scala:47) at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$anonfun$run$2.apply(SparkLaunch.scala:39) at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$anonfun$run$2.apply(SparkLaunch.scala:39) at scala.collection.immutable.List.foreach(List.scala:381) at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.run(SparkLaunch.scala:39) at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch$.main(SparkLaunch.scala:16) at com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch.main(SparkLaunch.scala)

In spark-bench-env.sh, the environment variables are set as follows:

export SPARK_HOME=/usr/hdp/2.5.3.0-37/spark

export SPARK_MASTER_HOST=yarn-client (also tried with yarn)

Thanks in advance

[#2]

@vshukla Please help in this regard.

[#3]

Are you running Spark 1.6 or Spark 2.1?

If 2.1, change SPARK_HOME=/usr/hdp/current/spark2-client, if 1.6, use ../spark-client instead.

Also, make sure the minimal-example.conf works first before trying the multi-submit-example to make sure your spark-bench environment is fine.