Support Questions
Find answers, ask questions, and share your expertise

Not able to run the sample PI program in YARN cluster mode

Solved Go to solution

Not able to run the sample PI program in YARN cluster mode

Explorer

Hi folks,

I am trying to run the spark PI example on the Hortonworks cluster. I could run this successfully in the local mode and in the yarn client mode . When I try to run it using YARN cluster mode I do not see any out put or error. I am not sure if it is running or is there a bug. Can you please help me understand this behavior?

Here are the commands I am trying to use.

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster ../lib/spark-examples*.jar 10

Also tried with

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ../lib/spark-examples*.jar 10

(One more question are above both correct ?)

Both the times I see the following console log but not the PI value. ( When I run in local mode or yarn-client mode I see the value of PI printed on the console)

16/02/11 11:06:05 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
16/02/11 11:06:05 INFO SecurityManager: Changing view acls to: username
16/02/11 11:06:05 INFO SecurityManager: Changing modify acls to: username
16/02/11 11:06:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(username); users with modify permissions: Set(username)
16/02/11 11:06:06 INFO Client: Submitting application 432 to ResourceManager
16/02/11 11:06:06 INFO YarnClientImpl: Submitted application application_1454617624671_0432
16/02/11 11:06:07 INFO Client: Application report for application_1454617624671_0432 (state: ACCEPTED)
16/02/11 11:06:07 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1455188766121
         final status: UNDEFINED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
..............
16/02/11 11:06:23 INFO Client: Application report for application_1454617624671_0432 (state: FINISHED)
16/02/11 11:06:23 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: APP_Master_host_IP
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1455188766121
         final status: SUCCEEDED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
16/02/11 11:06:23 INFO ShutdownHookManager: Shutdown hook called
16/02/11 11:06:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-54dc94ab-cf66-4d17-9940-1c31ba7e9850
[username@remoteIP bin]$


1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Not able to run the sample PI program in YARN cluster mode

Mentor

@Darpan Patel go to the job history, drill down to the logs of the task attempt and you should see the result.

View solution in original post

5 REPLIES 5
Highlighted

Re: Not able to run the sample PI program in YARN cluster mode

@Darpan Patel

The output will be in log file ;)

Highlighted

Re: Not able to run the sample PI program in YARN cluster mode

See this http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_spark-quickstart/content/run_spark_pi.htm...

Job output should list the estimated value of pi. In the following example, output was directed to stdout:

Log Type: stdout
Log Upload Time: 10-Jun-2015 17:13:33
Log Length: 23 

Pi is roughly 3.142532

Highlighted

Re: Not able to run the sample PI program in YARN cluster mode

Explorer
Highlighted

Re: Not able to run the sample PI program in YARN cluster mode

Mentor

@Darpan Patel go to the job history, drill down to the logs of the task attempt and you should see the result.

View solution in original post

Highlighted

Re: Not able to run the sample PI program in YARN cluster mode

Explorer

Thanks @Artem Ervits