JA018 java.lang.NoSuchMethodError: org.apache.oozie.action.hadoop.Hive2Main.setYarnTag()

[#1]

Have a 2 node cluster, things seem to work, HOWEVER, workflows occasionally (like 70% of the time) fail with the following error:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], main() threw exception, org.apache.oozie.action.hadoop.Hive2Main.setYarnTag(Lorg/apache/hadoop/conf/Configuration;)V
java.lang.NoSuchMethodError: org.apache.oozie.action.hadoop.Hive2Main.setYarnTag(Lorg/apache/hadoop/conf/Configuration;)V

!!! 

I would understand if she was failing consistently, which would mean misconfiguration.  But sometimes it passes, and workflows complete successfully.  However, most of the time she fails with that same error.

Please advise how to troubleshoot  or where to look for more hints/logs that would point out the problem!

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], main() threw exception, org.apache.oozie.action.hadoop.Hive2Main.setYarnTag(Lorg/apache/hadoop/conf/Configuration;)V
java.lang.NoSuchMethodError: org.apache.oozie.action.hadoop.Hive2Main.setYarnTag(Lorg/apache/hadoop/conf/Configuration;)V
at org.apache.oozie.action.hadoop.Hive2Main.initActionConf(Hive2Main.java:82)
at org.apache.oozie.action.hadoop.Hive2Main.run(Hive2Main.java:109)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
at org.apache.oozie.action.hadoop.Hive2Main.main(Hive2Main.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Oozie Launcher failed, finishing Hadoop job gracefully

[#2]

The error quotes a missing function that has been present in Oozie since CDH 5.5.0. It therefore appears that somehow your environment is keeping or passing around an older jar of 'oozie-sharelib-oozie' artifact that is without this added function.

If its your sharelib that's carrying a bad file, you can inspect it via:

# hadoop fs -ls -R /user/oozie/ | grep sharelib-oozie

The above should return only a single jar file size and the version of the filename should match what you are running. If you get 3 or more files in the output, consider redeploying your ShareLib via https://www.cloudera.com/documentation/enterprise/latest/topics/admin_oozie_sharelib.html#concept_i2...

If you just get one version of the jar instead, then perhaps some application jar of your project(s) is assembling a fat jar that includes Oozie Sharelib dependencies in it, albeit from a non-CDH version, or a very old CDH version (< 5.5.0).

You can inspect suspect jars by running:

# jar tf filename.jar | grep LauncherMain

Repack all such Oozie-including jars to exclude Oozie dependencies in them, as the system classpath will already provide the dependencies and of the right version.