Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark action always submit to 0.0.0.0:8032

avatar

I'm trying to submit a spark action using Oozie with master=yarn-cluster and mode=cluster but the job hangs trying to submit to RM 0.0.0.0:8032

In the YARN logs of the application there are the the following 2 lines:

org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at hdp24node00/192.168.178.30:8050
org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
...
...
org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

It first try to connect to the right resource manager, but then defaults to 0.0.0.0:8032 until it fails.

yarn.resourcemanager.address is set to hdp24node00:8050 and job-tracker property is also set to hdp24node00:8050.

Where is the problem?

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Nicola Marangoni

@David Tam @Breandán Mac Parland

Oozie Spark action is not supported in HDP 2.4 or earlier. There are workarounds on this site you may find but they're not supported by Hortonworks. This means you can get it to work but you can't open tickets in regards to them with our support. Can you set jobTracker to port 8032 instead of 8050 and try again? Look at this question as it's similar https://community.hortonworks.com/questions/11599/how-to-change-resourcemanager-port-on-oozie.html#a...

Hortonworks uses the source code from Apache Oozie and internally Apache Oozie version expects port 8032 for yarn.resourcemanager.address not 8050. Look at the unit test example https://github.com/apache/oozie/search?utf8=%E2%9C%93&q=8032

in our private repos, this is an identical unit test. You guys all have similar questions, I will raise an internal discussion on how to proceed with this.

Here's a note on Spark action in HDP http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_spark-guide/content/ch_introduction-spark...

View solution in original post

11 REPLIES 11

avatar
New Contributor

I am running HDP 2.4.2 and here is the stack trace from yarn node manager logs

2016-06-30 12:17:23,376 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_e11_1467282483822_0010_02_000001

2016-06-30 12:17:23,378 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(866)) - Failed to download resource { { file:/hadoop/yarn/local/filecache/117/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar, 1467269880000, FILE, null },pending,[(container_e11_1467282483822_0010_02_000001)],22592100725522,DOWNLOADING}

java.io.FileNotFoundException: File file:/hadoop/yarn/local/filecache/117/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar does not exist

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)

at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

avatar
New Contributor

We've three nodes cluster of HDP 2.4 stack. We are trying to run spark jobs using oozie workflow, did all the workaround mentioned in this post as well as posted in other Hortonworks community. But still we are getting error:

"Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Call From ip-172-63-10-15.eu-west-1.compute.internal/172.63.10.15 to 0.0.0.0:8032 failed on co nnection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused java.net.ConnectException: Call From ip-172-63-10-15.eu-west-1.compute.internal/172.63.10.15 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused"

Any suggestion, what we are missing and what settings needs to done?