Support Questions
Find answers, ask questions, and share your expertise

Pig script running on MapReduce but not on Tez

Super Collaborator

Hi Guys,

I am using latest version of Pig(0.16.0) and Tez version is 0.8.5. The pig script is running fine on MapReduce, but not with Tez. Tez with MapReduce and Hive(2.1.1) is working fine. Can this be a version mismatch problem? Please have a look at the logs:

Pig Stack Trace --------------- ERROR 2017: Internal error creating job configuration. org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias joined at org.apache.pig.PigServer.openIterator(PigServer.java:1019) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) at org.apache.pig.Main.run(Main.java:564) at org.apache.pig.Main.main(Main.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias joined at org.apache.pig.PigServer.storeEx(PigServer.java:1122) at org.apache.pig.PigServer.store(PigServer.java:1081) at org.apache.pig.PigServer.openIterator(PigServer.java:994) ... 13 more Caused by: org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR 2017: Internal error creating job configuration. at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:137) at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.compile(TezJobCompiler.java:78) at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:198) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308) at org.apache.pig.PigServer.launchPlan(PigServer.java:1474) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459) at org.apache.pig.PigServer.storeEx(PigServer.java:1118) ... 15 more Caused by: java.lang.NoSuchMethodException: org.apache.tez.dag.api.DAG.setCallerContext(org.apache.tez.client.CallerContext) at java.lang.Class.getMethod(Class.java:1786) at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:128) ... 21 more ================================================================================

1 REPLY 1

Guru

The latest two versions of Hortonworks Data Platform (HDP), v 2.5 and 2.6, support pig 0.16 and tez 0.7. These versions have been thoroughly cross-tested.

In terms of Apache releases, pig 0.16 was released on June 08 2016. Tez 0.83, 0.84, and 0.85 were released Apr 15 2016, Jul 08 2016, Mar 13 2017. According to Tez version compatibility statement, all 0.8x are likely to be compatible with each other. Because 0.85 is a very recent release, it is likely there is a bug.

I suggest that you try pig 0.16 against Tez 0.83 or 0.84. If it works for these, you will know there is a bug in 0.85 (for which you should file a jira). If it does not work for these, try it against Tez 0.7.

; ;