Hi Tejas, Thanks for your response. Yes, agreed that it will be available when they will release pig 0.17 version, but as of now for development environment they have created github repo for pig 0.17 which we can use and implement spark as execution engine. I have already implemented spark as local mode but facing some issue with yarn-client mode. Error for yarn-client mode: sshuser@hn0-dfspar:~/pig/bin$ ./pig -x spark
Using Spark Home: /usr/hdp/current/spark-client
17/06/02 05:41:54 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
17/06/02 05:41:54 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
17/06/02 05:41:54 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL
17/06/02 05:41:54 INFO pig.ExecTypeProvider: Trying ExecType : TEZ
17/06/02 05:41:54 INFO pig.ExecTypeProvider: Trying ExecType : SPARK
17/06/02 05:41:54 INFO pig.ExecTypeProvider: Picked SPARK as the ExecType
17/06/02 05:41:54 ERROR pig.Main: ERROR 2998: Unhandled internal error. org/apache/spark/scheduler/SparkListener
17/06/02 05:41:54 WARN pig.Main: There is no log file to write to.
17/06/02 05:41:54 ERROR pig.Main: java.lang.NoClassDefFoundError: org/apache/spark/scheduler/SparkListener
at org.apache.pig.backend.hadoop.executionengine.spark.SparkExecutionEngine.<init>(SparkExecutionEngine.java:35)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkExecType.getExecutionEngine(SparkExecType.java:42)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:269)
at org.apache.pig.impl.PigContext.<init>(PigContext.java:256)
at org.apache.pig.Main.run(Main.java:389)
at org.apache.pig.Main.main(Main.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.scheduler.SparkListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more Regards, Mitesh
... View more
