Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark action always submit to 0.0.0.0:8032

Solved Go to solution
Highlighted

Re: Spark action always submit to 0.0.0.0:8032

New Contributor

I am running HDP 2.4.2 and here is the stack trace from yarn node manager logs

2016-06-30 12:17:23,376 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_e11_1467282483822_0010_02_000001

2016-06-30 12:17:23,378 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(866)) - Failed to download resource { { file:/hadoop/yarn/local/filecache/117/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar, 1467269880000, FILE, null },pending,[(container_e11_1467282483822_0010_02_000001)],22592100725522,DOWNLOADING}

java.io.FileNotFoundException: File file:/hadoop/yarn/local/filecache/117/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar does not exist

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)

at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Highlighted

Re: Spark action always submit to 0.0.0.0:8032

New Contributor

We've three nodes cluster of HDP 2.4 stack. We are trying to run spark jobs using oozie workflow, did all the workaround mentioned in this post as well as posted in other Hortonworks community. But still we are getting error:

"Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Call From ip-172-63-10-15.eu-west-1.compute.internal/172.63.10.15 to 0.0.0.0:8032 failed on co nnection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused java.net.ConnectException: Call From ip-172-63-10-15.eu-west-1.compute.internal/172.63.10.15 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused"

Any suggestion, what we are missing and what settings needs to done?

Don't have an account?
Coming from Hortonworks? Activate your account here