Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

couldn't able to execute spark jar in cluster mode

Solved Go to solution
Highlighted

couldn't able to execute spark jar in cluster mode

New Contributor

I am working on Spark with scala , I have created a jar and tried to execute in the cluster mode as below spark-submit --class com.test.sales.LoadSalesData \ --master yarn \ --deploy-mode cluster \ --driver-memory 4g \

--executor-memory 2g \

--executor-cores 1 \

test_salesdata_2.10-1.0.jar

but I am getting below error ,

17/01/18 05:15:16 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:17 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:18 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:19 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:20 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:21 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:22 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:23 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED)

client token: N/A diagnostics: Application application_1483728394968_0141 failed 2 times due to AM Container for appattempt_1483728394968_0141_000002 exited with exitCode: 1 3

i could able to execute the same code in spark-shell with out any issues.. could you please let me know ,if I am missing anything to run the spark job in cluster mode. thank you !

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: couldn't able to execute spark jar in cluster mode

@Satya

yarn application log could give the actual error.

Can you also check if you have set master inside the code?

View solution in original post

5 REPLIES 5
Highlighted

Re: couldn't able to execute spark jar in cluster mode

Expert Contributor

Looks like your job was submitted to Yarn. Can you log information from spark history UI or from Yarn

Regards

Pranay Vyas

Highlighted

Re: couldn't able to execute spark jar in cluster mode

Expert Contributor

Please check from the Yarn RM UI why the application master failed? The AM logs would give you indication depending on exceptions as to what caused the failures . Once that is identified, you can proceed to fix it.

Highlighted

Re: couldn't able to execute spark jar in cluster mode

@Satya

yarn application log could give the actual error.

Can you also check if you have set master inside the code?

View solution in original post

Highlighted

Re: couldn't able to execute spark jar in cluster mode

New Contributor

Thank you.. It has been resolved now... i forgot to remove .setMaster("local") in the spark jar.. after removing that job went fine..

Re: couldn't able to execute spark jar in cluster mode

Expert Contributor

Please accept the answer by Sandeep so that this question can be marked as addressed.

Don't have an account?
Coming from Hortonworks? Activate your account here