I have a Oozie job which starts a Oozie shell action,the shell action starts a spark application (spark2-submit). I am mostly doing spark sql. The jobs runs for a while and suddenly hangs. It starts the spark application all over again.
I ran the same spark application in CDSW and it ran fine without issues.
The same is happening with other Oozie job . The only common thing between these two jobs is that they run longer, around 2hrs.
Any help will be helpful.
The oozie mapper was running out of 4GB memory. I changed that to 8GB. Now the job ran fine without restarts.
Congratulations on resolving your issue @Sunil. Please don't forget to mark the reply that helped resolve the issue as the answer. That way when others have a similar issue they will be more likely to find it.
I get the following error
line 2: spark-submit: command not found Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code