Not able to run the sample PI program in YARN cluster mode

darpan_be — Thu, 11 Feb 2016 19:30:33 GMT

Hi folks,

I am trying to run the spark PI example on the Hortonworks cluster. I could run this successfully in the local mode and in the yarn client mode . When I try to run it using YARN cluster mode I do not see any out put or error. I am not sure if it is running or is there a bug. Can you please help me understand this behavior?

Here are the commands I am trying to use.

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster ../lib/spark-examples*.jar 10

Also tried with

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ../lib/spark-examples*.jar 10

(One more question are above both correct ?)

Both the times I see the following console log but not the PI value. ( When I run in local mode or yarn-client mode I see the value of PI printed on the console)

16/02/11 11:06:05 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
16/02/11 11:06:05 INFO SecurityManager: Changing view acls to: username
16/02/11 11:06:05 INFO SecurityManager: Changing modify acls to: username
16/02/11 11:06:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(username); users with modify permissions: Set(username)
16/02/11 11:06:06 INFO Client: Submitting application 432 to ResourceManager
16/02/11 11:06:06 INFO YarnClientImpl: Submitted application application_1454617624671_0432
16/02/11 11:06:07 INFO Client: Application report for application_1454617624671_0432 (state: ACCEPTED)
16/02/11 11:06:07 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1455188766121
         final status: UNDEFINED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
..............
16/02/11 11:06:23 INFO Client: Application report for application_1454617624671_0432 (state: FINISHED)
16/02/11 11:06:23 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: APP_Master_host_IP
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1455188766121
         final status: SUCCEEDED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
16/02/11 11:06:23 INFO ShutdownHookManager: Shutdown hook called
16/02/11 11:06:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-54dc94ab-cf66-4d17-9940-1c31ba7e9850
[username@remoteIP bin]$

Re: Not able to run the sample PI program in YARN cluster mode

nsabharwal — Thu, 11 Feb 2016 19:34:26 GMT

@Darpan Patel

The output will be in log file 😉

Re: Not able to run the sample PI program in YARN cluster mode

nsabharwal — Thu, 11 Feb 2016 19:38:29 GMT

See this http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_spark-quickstart/content/run_spark_pi.html

Job output should list the estimated value of pi. In the following example, output was directed to stdout:

Log Type: stdout
Log Upload Time: 10-Jun-2015 17:13:33
Log Length: 23

Pi is roughly 3.142532

Re: Not able to run the sample PI program in YARN cluster mode

aervits — Thu, 11 Feb 2016 19:40:38 GMT

@Darpan Patel go to the job history, drill down to the logs of the task attempt and you should see the result.

Re: Not able to run the sample PI program in YARN cluster mode

darpan_be — Thu, 11 Feb 2016 20:03:24 GMT

Thanks @Neeraj Sabharwal

Re: Not able to run the sample PI program in YARN cluster mode

darpan_be — Thu, 11 Feb 2016 20:03:44 GMT

Thanks @Artem Ervits

question Re: Not able to run the sample PI program in YARN cluster mode in Archives of Support Questions (Read Only)

Not able to run the sample PI program in YARN cluster mode

Re: Not able to run the sample PI program in YARN cluster mode

Re: Not able to run the sample PI program in YARN cluster mode

Re: Not able to run the sample PI program in YARN cluster mode

Re: Not able to run the sample PI program in YARN cluster mode

Re: Not able to run the sample PI program in YARN cluster mode