Spark Streaming - out of memory when submit using Oozie

[#1]

Dear Colleages,

I submitted a Spark Streaming job via Oozie and get the following error messages:

Warning: Skip remote jar hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20160405235854/oozie/oozie-sharelib-oozie.jar.
Halting due to Out Of Memory Error...
Halting due to Out Of Memory Error...

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "LeaseRenewer:[email protected]:8020"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Executor task launch worker-2"

Do you have an idea or a solution to prevent these error messages?

Thanks in advance and best regards,

 butkiz

[#2]

Hi,

Have you fixed the issue..?

[#3 · Accepted solution]

Dear butkiz,

Please add the following property to the configuration block of the oozie spark action to give this more memory.

<property>
    <name>oozie.launcher.mapreduce.map.memory.mb</name>
   <value>4096</value>
</property>
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>4096</value>
</property>
<property>
    <name>oozie.launcher.mapred.child.java.opts</name>
    <value>-Xmx4096m</value>
</property>

[#4]

Hi,

it works applying above configuration.

But now i have a NullPointerException in my spark code (rdd.foreach):
...

kafkaStream.foreachRDD(new VoidFunction<JavaPairRDD<String, byte[]>>() {
			
			public void call(JavaPairRDD<String, byte[]> rdd) throws Exception {
			    rdd.foreach(new VoidFunction<Tuple2<String, byte[]>>() {
					
			    	public void call(Tuple2<String, byte[]> avroRecord) throws Exception {

In local mode it works but not in yarn-cluster.

Do you have any ideas in order to get it running?

Best Regards,

 Butkiz