Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark load and Join is so slow and failing after long wait

Highlighted

Spark load and Join is so slow and failing after long wait

Explorer

HI,

I need to create feature dataset from huge koggle data. There are 6 to 7 huge files and highest of the size is 7GB. I am just using standard operations, reading the data from the parquet files, and then creating DF and performing joins as per the requirement.

I have 6 Nodes cluster and it is having sufficient memory. However, when i run the job using spark stanalone clustermanager with maximum memory and cores, the job is failing. The same steps that i have done are working fine in spark-shell however.

I have searched enough to find what configurations might have gone wrong and i have used same configurations that i used with Spark-shell, and used cache and persist both types and still, ended up receiving "No space Left on Disk" or "failing in the middle".. What could have been wrong.? and why the same operations running and completing without any fuss in spark-shell.? I have not changed a bit of code from shell to my app.

3 REPLIES 3

Re: Spark load and Join is so slow and failing after long wait

Expert Contributor

Srini,

Can you paste the error message you are getting when you do the spark-submit?

Re: Spark load and Join is so slow and failing after long wait

Mentor

@Srinivasarao Daruna can you post your solution to close out the thread?

Re: Spark load and Join is so slow and failing after long wait

Expert Contributor

Have you tried using YARN instead of Spark's cluster manager? Try "--master yarn". If you post your command-line arguments it might help give us some more clues.

Don't have an account?
Coming from Hortonworks? Activate your account here