OOZIE co-ordinator jobs failing frequently with "org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy" and "java.lang.AbstractMethodError: org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus"

[#1]

Hi ,

HDP 2.6.3

Ambari :2.6

we are facing one strange issue in oozie co-ordinator job, the same job is successful in one run and failing in next run.[failing frequently ]

"org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy" and "java.lang.AbstractMethodError: org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus"

THese two exceptions are trhrowing , strange thing is only one exception is encountered in failure.below is the log

java.lang.AbstractMethodError: org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus" Exception:

2018-09-28 13:01:00,154 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@:start:] Start action [0001906-180924020054520-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-09-28 13:01:00,154 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@:start:] [***0001906-180924020054520-oozie-oozi-W@:start:***]Action status=DONE 2018-09-28 13:01:00,154 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@:start:] [***0001906-180924020054520-oozie-oozi-W@:start:***]Action updated in DB! 2018-09-28 13:01:00,298 INFO WorkflowNotificationXCommand:520 - SERVER[<hostname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0001906-180924020054520-oozie-oozi-W@:start: 2018-09-28 13:01:00,299 INFO WorkflowNotificationXCommand:520 - SERVER[<hostname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0001906-180924020054520-oozie-oozi-W 2018-09-28 13:01:00,337 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] Start action [0001906-180924020054520-oozie-oozi-W@<user>] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-09-28 13:01:01,913 INFO HiveActionExecutor:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[${applicationName}] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] Trying to get job [job_1535720217078_27385], attempt [1] 2018-09-28 13:01:01,929 INFO HiveActionExecutor:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[${applicationName}] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] checking action, hadoop job ID [job_1535720217078_27385] status [RUNNING] 2018-09-28 13:01:01,931 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[${applicationName}] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] [***0001906-180924020054520-oozie-oozi-W@<user>***]Action status=RUNNING 2018-09-28 13:01:01,931 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[${applicationName}] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] [***0001906-180924020054520-oozie-oozi-W@<user>***]Action updated in DB! 2018-09-28 13:01:01,940 INFO WorkflowNotificationXCommand:520 - SERVER[<hostname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] No Notification URL is defined. Therefore nothing to notify for job 0001906-180924020054520-oozie-oozi-W@<user> 2018-09-28 13:03:18,448 INFO CallbackServlet:520 - SERVER[<hostname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] callback for action [0001906-180924020054520-oozie-oozi-W@<user>] 2018-09-28 13:03:18,485 INFO HiveActionExecutor:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] Trying to get job [job_1535720217078_27385], attempt [1] 2018-09-28 13:03:18,595 INFO HiveActionExecutor:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] Hadoop Jobs launched : [job_1535720217078_27388] 2018-09-28 13:03:18,596 INFO HiveActionExecutor:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] action completed, external ID [job_1535720217078_27385] 2018-09-28 13:03:18,598 WARN HiveActionExecutor:523 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/shims/HadoopShims$HdfsFileStatus;Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;)V 2018-09-28 13:03:18,598 WARN HiveActionExecutor:523 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] Launcher exception: org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/shims/HadoopShims$HdfsFileStatus;Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;)V java.lang.AbstractMethodError: org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/shims/HadoopShims$HdfsFileStatus;Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;)V at org.apache.hadoop.hive.common.FileUtils.mkdir(FileUtils.java:527) at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:236) at org.apache.hadoop.hive.ql.Context.getExtTmpPathRelTo(Context.java:426) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6284) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9020) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8911) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9756) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9649) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10122) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:325) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10133) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:315) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:429) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:718) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:336) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:313) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:58) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:240) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) 2018-09-28 13:03:18,633 INFO ActionEndXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] ERROR is considered as FAILED for SLA 2018-09-28 13:03:18,678 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@kill] Start action [0001906-180924020054520-oozie-oozi-W@kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-09-28 13:03:18,679 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@kill] [***0001906-180924020054520-oozie-oozi-W@kill***]Action status=DONE 2018-09-28 13:03:18,679 INFO ActionStartXCommand:520 - SERVER[<hostname>] USER[d<username>] GROUP[-] TOKEN[] APP[mdghive] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@kill] [***0001906-180924020054520-oozie-oozi-W@kill***]Action updated in DB! 2018-09-28 13:03:18,805 INFO WorkflowNotificationXCommand:520 - SERVER[<hostname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@<user>] No Notification URL is defined. Therefore nothing to notify for job 0001906-180924020054520-oozie-oozi-W@<user> 2018-09-28 13:03:18,805 INFO WorkflowNotificationXCommand:520 - SERVER[<hostname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[0001906-180924020054520-oozie-oozi-W@kill] No Notification URL is defined. Therefore nothing to notify for job 0001906-180924020054520-oozie-oozi-W@kill 2018-09-28 13:03:18,806 INFO WorkflowNotificationXCommand:520 - SERVER[<hostname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001906-180924020054520-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0001906-180924020054520-oozie-oozi-W

org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy:

2018-09-30 13:01:00,125 INFO ActionStartXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@:start:] Start action [0002306-180924020054520-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-09-30 13:01:00,126 INFO ActionStartXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@:start:] [***0002306-180924020054520-oozie-oozi-W@:start:***]Action status=DONE 2018-09-30 13:01:00,126 INFO ActionStartXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@:start:] [***0002306-180924020054520-oozie-oozi-W@:start:***]Action updated in DB! 2018-09-30 13:01:00,197 INFO WorkflowNotificationXCommand:520 - SERVER[<hostnname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0002306-180924020054520-oozie-oozi-W@:start: 2018-09-30 13:01:00,198 INFO WorkflowNotificationXCommand:520 - SERVER[<hostnname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0002306-180924020054520-oozie-oozi-W 2018-09-30 13:01:00,227 INFO ActionStartXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] Start action [0002306-180924020054520-oozie-oozi-W@<username>] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-09-30 13:01:01,819 INFO HiveActionExecutor:520 - SERVER[<hostname>-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] Trying to get job [job_1535720217078_28637], attempt [1] 2018-09-30 13:01:01,838 INFO HiveActionExecutor:520 - SERVER[<hostname>-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] checking action, hadoop job ID [job_1535720217078_28637] status [RUNNING] 2018-09-30 13:01:01,840 INFO ActionStartXCommand:520 - SERVER[<hostname>-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] [***0002306-180924020054520-oozie-oozi-W@<username>***]Action status=RUNNING 2018-09-30 13:01:01,840 INFO ActionStartXCommand:520 - SERVER[<hostname>-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] [***0002306-180924020054520-oozie-oozi-W@<username>***]Action updated in DB! 2018-09-30 13:01:01,847 INFO WorkflowNotificationXCommand:520 - SERVER[<hostnname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] No Notification URL is defined. Therefore nothing to notify for job 0002306-180924020054520-oozie-oozi-W@<username> 2018-09-30 13:03:00,032 INFO CallbackServlet:520 - SERVER[<hostnname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] callback for action [0002306-180924020054520-oozie-oozi-W@<username>] 2018-09-30 13:03:00,068 INFO HiveActionExecutor:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] Trying to get job [job_1535720217078_28637], attempt [1] 2018-09-30 13:03:00,156 INFO HiveActionExecutor:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] action completed, external ID [job_1535720217078_28637] 2018-09-30 13:03:00,158 WARN HiveActionExecutor:523 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Lorg/apache/hadoop/hive/conf/HiveConf;Lorg/apache/hadoop/hive/metastore/HiveMetaHookLoader;Ljava/util/Map;Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient; 2018-09-30 13:03:00,158 WARN HiveActionExecutor:523 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] Launcher exception: org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Lorg/apache/hadoop/hive/conf/HiveConf;Lorg/apache/hadoop/hive/metastore/HiveMetaHookLoader;Ljava/util/Map;Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient; java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Lorg/apache/hadoop/hive/conf/HiveConf;Lorg/apache/hadoop/hive/metastore/HiveMetaHookLoader;Ljava/util/Map;Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient; at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:336) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:313) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:58) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:240) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) 2018-09-30 13:03:00,218 INFO ActionEndXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] ERROR is considered as FAILED for SLA 2018-09-30 13:03:00,259 INFO ActionStartXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@kill] Start action [0002306-180924020054520-oozie-oozi-W@kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-09-30 13:03:00,259 INFO ActionStartXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@kill] [***0002306-180924020054520-oozie-oozi-W@kill***]Action status=DONE 2018-09-30 13:03:00,259 INFO ActionStartXCommand:520 - SERVER[<hostnname>] USER[<groupname>] GROUP[-] TOKEN[] APP[mdghive] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@kill] [***0002306-180924020054520-oozie-oozi-W@kill***]Action updated in DB! 2018-09-30 13:03:00,350 INFO WorkflowNotificationXCommand:520 - SERVER[<hostnname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@kill] No Notification URL is defined. Therefore nothing to notify for job 0002306-180924020054520-oozie-oozi-W@kill 2018-09-30 13:03:00,350 INFO WorkflowNotificationXCommand:520 - SERVER[<hostnname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0002306-180924020054520-oozie-oozi-W 2018-09-30 13:03:00,350 INFO WorkflowNotificationXCommand:520 - SERVER[<hostnname>] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0002306-180924020054520-oozie-oozi-W] ACTION[0002306-180924020054520-oozie-oozi-W@<username>] No Notification URL is defined. Therefore nothing to notify for job 0002306-180924020054520-oozie-oozi-W@<username>

Thanks,

Anji

[#2]

@Artem Ervits could you please help me in this.

[#3]

@Kuldeep Kulkarni need your support 🙂