Not able to run the sample PI program in YARN cluster mode

[#1]

Hi folks,

I am trying to run the spark PI example on the Hortonworks cluster. I could run this successfully in the local mode and in the yarn client mode . When I try to run it using YARN cluster mode I do not see any out put or error. I am not sure if it is running or is there a bug. Can you please help me understand this behavior?

Here are the commands I am trying to use.

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster ../lib/spark-examples*.jar 10

Also tried with

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ../lib/spark-examples*.jar 10

(One more question are above both correct ?)

Both the times I see the following console log but not the PI value. ( When I run in local mode or yarn-client mode I see the value of PI printed on the console)

16/02/11 11:06:05 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
16/02/11 11:06:05 INFO SecurityManager: Changing view acls to: username
16/02/11 11:06:05 INFO SecurityManager: Changing modify acls to: username
16/02/11 11:06:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(username); users with modify permissions: Set(username)
16/02/11 11:06:06 INFO Client: Submitting application 432 to ResourceManager
16/02/11 11:06:06 INFO YarnClientImpl: Submitted application application_1454617624671_0432
16/02/11 11:06:07 INFO Client: Application report for application_1454617624671_0432 (state: ACCEPTED)
16/02/11 11:06:07 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1455188766121
         final status: UNDEFINED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
..............
16/02/11 11:06:23 INFO Client: Application report for application_1454617624671_0432 (state: FINISHED)
16/02/11 11:06:23 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: APP_Master_host_IP
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1455188766121
         final status: SUCCEEDED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
16/02/11 11:06:23 INFO ShutdownHookManager: Shutdown hook called
16/02/11 11:06:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-54dc94ab-cf66-4d17-9940-1c31ba7e9850
[username@remoteIP bin]$

[#2]

@Darpan Patel

The output will be in log file 😉

[#3]

See this http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_spark-quickstart/content/run_spark_pi.htm...

Job output should list the estimated value of pi. In the following example, output was directed to stdout:

Log Type: stdout
Log Upload Time: 10-Jun-2015 17:13:33
Log Length: 23 

Pi is roughly 3.142532

[#4]

Thanks @Neeraj Sabharwal

[#5 · Accepted solution]

@Darpan Patel go to the job history, drill down to the logs of the task attempt and you should see the result.

[#6]

Thanks @Artem Ervits