I'm trying to run this command ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 512m --executor-memory 1g --executor-cores 1 --queue default examples/jars/spark-examples_2.11-220.127.116.11.6.2.0-205.jar 10 in the directory /usr/hdp/current/spark2-client with spark user.
I got the error log mentioned in "log.png". From log application through resource_manger:8088 i got the errors in my ambari-agent1 and my ambari-agent2 as shown respectively in "log1.png" and "log2.png".
How can i submit spark jobs correctly?
P.S: Sometimes, when i run spark submit, the job is correctly finished but it seems like it submitted to one node. I conclude this fact from application log through resource_manger:8088 as shown in result.png and details.png.
How can i fix this issue?