Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

submittin spark jobs in oozie

submittin spark jobs in oozie

New Contributor

I am submitting the spark jobs in Ooze but getting the below error? Please look in to this.

2016-09-09 00:11:40,999 WARN SparkActionExecutor:523 - SERVER[MSSUbuntuServer.corp.microsoft.com] USER[root] GROUP[-] TOKEN[] APP[SparkFileCopy] JOB[0000011-160902183749193-oozie-oozi-W] ACTION[0000011-160902183749193-oozie-oozi-W@spark-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode; 2016-09-09 00:11:41,001 WARN SparkActionExecutor:523 - SERVER[MSSUbuntuServer.corp.microsoft.com] USER[root] GROUP[-] TOKEN[] APP[SparkFileCopy] JOB[0000011-160902183749193-oozie-oozi-W] ACTION[0000011-160902183749193-oozie-oozi-W@spark-node] Launcher exception: com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode; java.lang.NoSuchMethodError: com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode; at org.apache.spark.util.collection.OpenHashSet.org$apache$spark$util$collection$OpenHashSet$$hashcode(OpenHashSet.scala:261) at org.apache.spark.util.collection.OpenHashSet$mcI$sp.getPos$mcI$sp(OpenHashSet.scala:165) at org.apache.spark.util.collection.OpenHashSet$mcI$sp.contains$mcI$sp(OpenHashSet.scala:102) at org.apache.spark.util.SizeEstimator$$anonfun$visitArray$2.apply$mcVI$sp(SizeEstimator.scala:214) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.SizeEstimator$.visitArray(SizeEstimator.scala:210) at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:169) at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:161) at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:155) at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78) at org.apache.spark.util.collection.SizeTracker$class.afterUpdate(SizeTracker.scala:70) at org.apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31) at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:236) at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:126) at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:104) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:743) at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:594) at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:865) at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:79) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:68) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:36) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62) at org.apache.spark.SparkContext.broadcast(SparkContext.scala:809) at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:559) at org.apache.spark.SparkContext.textFile(SparkContext.scala:471) at org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:174) at org.apache.oozie.example.SparkFileCopy.main(SparkFileCopy.java:36) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:104) at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:95) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

4 REPLIES 4

Re: submittin spark jobs in oozie

Mentor

What version of HDP is this? Spark with Oozie is only supported in HDP 2.5

Re: submittin spark jobs in oozie

New Contributor

- HDP Version 2.4

Re: submittin spark jobs in oozie

Mentor

@narender pasunooti to use Spark with Oozie you need to upgrade to HDP 2.5, there is a workaround but it is not supported. Please consider going to 2.5

Re: submittin spark jobs in oozie

Expert Contributor

@narender pasunooti

The above exception occurred because same jar file with different versions might have set in classpath when Spark application launched from Oozie. These issues has been handled in HDP 2.5 and Spark application works successfully with Oozie.

But to make the Spark application work with Oozie for HDP 2.4, please perform following prerequiste steps first.

https://community.hortonworks.com/articles/48920/how-to-run-spark-action-in-oozie-of-hdp-230.html

Hope this helps you.

Don't have an account?
Coming from Hortonworks? Activate your account here