Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Not able to run the sample PI program in YARN cluster mode

avatar
Rising Star

Hi folks,

I am trying to run the spark PI example on the Hortonworks cluster. I could run this successfully in the local mode and in the yarn client mode . When I try to run it using YARN cluster mode I do not see any out put or error. I am not sure if it is running or is there a bug. Can you please help me understand this behavior?

Here are the commands I am trying to use.

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster ../lib/spark-examples*.jar 10

Also tried with

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster ../lib/spark-examples*.jar 10

(One more question are above both correct ?)

Both the times I see the following console log but not the PI value. ( When I run in local mode or yarn-client mode I see the value of PI printed on the console)

16/02/11 11:06:05 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
16/02/11 11:06:05 INFO SecurityManager: Changing view acls to: username
16/02/11 11:06:05 INFO SecurityManager: Changing modify acls to: username
16/02/11 11:06:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(username); users with modify permissions: Set(username)
16/02/11 11:06:06 INFO Client: Submitting application 432 to ResourceManager
16/02/11 11:06:06 INFO YarnClientImpl: Submitted application application_1454617624671_0432
16/02/11 11:06:07 INFO Client: Application report for application_1454617624671_0432 (state: ACCEPTED)
16/02/11 11:06:07 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1455188766121
         final status: UNDEFINED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
..............
16/02/11 11:06:23 INFO Client: Application report for application_1454617624671_0432 (state: FINISHED)
16/02/11 11:06:23 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: APP_Master_host_IP
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1455188766121
         final status: SUCCEEDED
         tracking URL: http://SomeIP:8088/proxy/application_1454617624671_0432/
         user: username
16/02/11 11:06:23 INFO ShutdownHookManager: Shutdown hook called
16/02/11 11:06:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-54dc94ab-cf66-4d17-9940-1c31ba7e9850
[username@remoteIP bin]$


1 ACCEPTED SOLUTION

avatar
Master Mentor

@Darpan Patel go to the job history, drill down to the logs of the task attempt and you should see the result.

View solution in original post

5 REPLIES 5

avatar
Master Mentor
@Darpan Patel

The output will be in log file 😉

avatar
Master Mentor

See this http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_spark-quickstart/content/run_spark_pi.htm...

Job output should list the estimated value of pi. In the following example, output was directed to stdout:

Log Type: stdout
Log Upload Time: 10-Jun-2015 17:13:33
Log Length: 23 

Pi is roughly 3.142532

avatar
Rising Star

avatar
Master Mentor

@Darpan Patel go to the job history, drill down to the logs of the task attempt and you should see the result.

avatar
Rising Star

Thanks @Artem Ervits