Support Questions

Find answers, ask questions, and share your expertise

Oozie work flow - PIG action failed

avatar
Expert Contributor

Hi I am trying to run a PIG action in oozie workflow and getting this error:

Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. org/apache/hadoop/hive/shims/ShimLoader

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader
	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
	at org.apache.hive.hcatalog.pig.PigHCatUtil.getHCatServerUri(PigHCatUtil.java:134)
	at org.apache.hive.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:217)
	at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
	at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
	at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901)
	at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
	at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
	at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
	at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
	at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
	at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
	at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735)
	at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443)
	at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387)
	at org.apache.pig.PigServer.executeBatch(PigServer.java:412)
	at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
	at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
	at org.apache.pig.Main.run(Main.java:502)
	at org.apache.pig.PigRunner.run(PigRunner.java:49)
	at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:288)
	at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:231)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
	at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.shims.ShimLoader
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 40 more
================================================================================
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
1 ACCEPTED SOLUTION

avatar

Sushil Saxena: Regarding issue in Falcon there is FALCON-1787 jira created.

If you are running Falcon process, updating "oozie.action.sharelib.for.pig" in Oozie-site.xml will not help. Falcon generates the pig action at the run time & generates the workflow, it uses pig-action.xml defined in Falcon codebase. pig-action.xml that Falcon uses does not have hive in the share lib config "oozie.action.sharelib.for.pig".

Workflow action configuration generated by Falcon is overriding the one defined in ooze-site.xml.

Few work arounds:

1> In the process entity definition add new custom property. Resubmit the process.

<properties> <property name="oozie.action.sharelib.for.pig" value="hive,pig,hcatalog"/> </properties>

2> Update pig-action.xml to have hive in the share lib config and repackage falcon-oozie-adaptor-<version>.jar and replace jar at "/usr/hdp/current/falcon-server/webapp/falcon/WEB-INF/lib” and restart Falcon

3> If you have Falcon code downloaded then update pig-action.xml at oozie/src/main/resources/action/process/pig-action.xml and then build falcon and reinstall it

View solution in original post

6 REPLIES 6

avatar
Master Mentor
@Sushil Saxena

I have a fix documented in this thread.

avatar
Expert Contributor

Please let me know which one I need to set:

oozie.action.sharelib.for.pig=pig,hcatalog

oozie.action.sharelib.for.pig=hive,pig,hcatalog

oozie.action.sharelib.for.hive=hive,hcatalog,sqoop

Also where I need to set: In Job Config or Hive-site.xml as custom property.

avatar
Master Mentor

@Sushil Saxena

I'll make it easy for you, just copy this workflow from my github page it has everything you need to run on sandbox, on your own environment, copy hive-site.xml from your cluster to lib folder. To answer your question, you choose the packages depending on what you're using, if you're using hcatalog you need pig,hcatalog and hive, if you use sqoop, add sqoop, etc.

avatar
Expert Contributor

It works fine when we made changes in workflow.xml for oozie 4.2. But when we are running Falcon process and then system will create the oozie workflow. At PIG action, it got failed with ShimsLoader error. Actually falcon is not picking changes made in oozie-site.xml:

oozie.action.sharelib.for.pig=pig,hcatalog,hive

I also tried restarting Falcon, OOzie, Hive bit still the not picking up the above property.

avatar
Master Mentor
@Sushil Saxena

right click pig and refresh client configs. As far as Falcon not picking up Oozie configs, have you confirmed that there are no stale services in existence?

avatar

Sushil Saxena: Regarding issue in Falcon there is FALCON-1787 jira created.

If you are running Falcon process, updating "oozie.action.sharelib.for.pig" in Oozie-site.xml will not help. Falcon generates the pig action at the run time & generates the workflow, it uses pig-action.xml defined in Falcon codebase. pig-action.xml that Falcon uses does not have hive in the share lib config "oozie.action.sharelib.for.pig".

Workflow action configuration generated by Falcon is overriding the one defined in ooze-site.xml.

Few work arounds:

1> In the process entity definition add new custom property. Resubmit the process.

<properties> <property name="oozie.action.sharelib.for.pig" value="hive,pig,hcatalog"/> </properties>

2> Update pig-action.xml to have hive in the share lib config and repackage falcon-oozie-adaptor-<version>.jar and replace jar at "/usr/hdp/current/falcon-server/webapp/falcon/WEB-INF/lib” and restart Falcon

3> If you have Falcon code downloaded then update pig-action.xml at oozie/src/main/resources/action/process/pig-action.xml and then build falcon and reinstall it