Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

How to fix JA006 in Oozie

Labels:
avatar
author-rank hsfelix
Rising Star

Created ‎05-26-2017 09:08 AM

I'm still trying to get oozie working... when I submit a job it hangs for 41minutes and then crashes with a JA006 exception.

I think the problem is in the jason serde. Has anyone has this issue?

This is the cloudera twitter pipeline example...

2017-05-25 17:48:15,895  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@:start:] Start action [0000000-170525174610830-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-05-25 17:48:15,898  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@:start:] [***0000000-170525174610830-oozie-oozi-W@:start:***]Action status=DONE
2017-05-25 17:48:15,898  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@:start:] [***0000000-170525174610830-oozie-oozi-W@:start:***]Action updated in DB!
2017-05-25 17:48:16,050  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000000-170525174610830-oozie-oozi-W@:start:
2017-05-25 17:48:16,050  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000000-170525174610830-oozie-oozi-W
2017-05-25 17:48:16,064  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Start action [0000000-170525174610830-oozie-oozi-W@hive-add-partition] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-05-25 17:48:16,121  INFO HadoopAccessorService:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Processing configuration files under [/usr/hdp/current/oozie-server/conf/action-conf/hive] for action [hive] and hostPort [*]
2017-05-25 17:48:16,134  INFO HadoopAccessorService:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Processing configuration file [hive-site.xml]
2017-05-25 17:48:16,136  INFO HadoopAccessorService:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Processing configuration file [hive-site_bak.xml]
2017-05-25 17:48:16,138  INFO HadoopAccessorService:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Processing configuration file [tez-site.xml]
2017-05-25 18:29:10,096  WARN ActionStartXCommand:523 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Error starting action [hive-add-partition]. ErrorType [TRANSIENT], ErrorCode [  JA006], Message [  JA006: Call From sandbox.hortonworks.com/192.168.0.73 to sandbox.hortonworks.com:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused]
org.apache.oozie.action.ActionExecutorException:   JA006: Call From sandbox.hortonworks.com/192.168.0.73 to sandbox.hortonworks.com:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1139)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1293)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Call From sandbox.hortonworks.com/192.168.0.73 to sandbox.hortonworks.com:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor89.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
	at org.apache.hadoop.ipc.Client.call(Client.java:1431)
	at org.apache.hadoop.ipc.Client.call(Client.java:1358)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy39.getDelegationToken(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getDelegationToken(ApplicationClientProtocolPBClientImpl.java:310)
	at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy40.getDelegationToken(Unknown Source)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getRMDelegationToken(YarnClientImpl.java:526)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.getDelegationToken(ResourceMgrDelegate.java:175)
	at org.apache.hadoop.mapred.YARNRunner.getDelegationToken(YARNRunner.java:221)
	at org.apache.hadoop.mapreduce.Cluster.getDelegationToken(Cluster.java:401)
	at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1240)
	at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1237)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:1236)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:465)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1336)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1087)
	... 10 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:612)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:710)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
	at org.apache.hadoop.ipc.Client.call(Client.java:1397)
	... 33 more
2017-05-25 18:29:10,098  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Next Retry, Attempt Number [1] in [10,000] milliseconds
2017-05-25 18:29:10,144  WARN ActionStartXCommand:523 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] ELException in ActionStartXCommand 
javax.servlet.jsp.el.ELException: No function is mapped to the name "coord:dataIn"
	at org.apache.commons.el.Logger.logError(Logger.java:481)
	at org.apache.commons.el.Logger.logError(Logger.java:498)
	at org.apache.commons.el.Logger.logError(Logger.java:525)
	at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:150)
	at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
	at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
	at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
	at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:204)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:195)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2017-05-25 18:29:10,144  WARN ActionStartXCommand:523 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Failing Job due to failed action [hive-add-partition]
2017-05-25 18:29:10,145  WARN LiteWorkflowInstance:523 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] Workflow Failed. Failing node [hive-add-partition]
2017-05-25 18:29:10,186  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] No Notification URL is defined. Therefore nothing to notify for job 0000000-170525174610830-oozie-oozi-W@hive-add-partition
2017-05-25 18:29:10,203  INFO KillXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[] STARTED WorkflowKillXCommand for jobId=0000000-170525174610830-oozie-oozi-W
2017-05-25 18:29:10,215  INFO KillXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[] ENDED WorkflowKillXCommand for jobId=0000000-170525174610830-oozie-oozi-W
2017-05-25 18:29:10,216  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000000-170525174610830-oozie-oozi-W
2017-05-25 18:29:20,138  WARN ActionStartXCommand:523 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000000-170525174610830-oozie-oozi-W] ACTION[0000000-170525174610830-oozie-oozi-W@hive-add-partition] E0816: Action pending=[false], status=[FAILED]. Skipping ActionStart Execution, Error Code: E0816
3,532 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank nshelke author-rank
Guru

Created ‎05-26-2017 12:28 PM

@Hugo Felix

Can you put value of following properties in job.properties and try again,

Current values in job.properties,

  1. job-tracker=hdfs://192.168.0.73:8021
  2. jobTracker=hdfs://192.168.0.73:8021

Replace with

  1. job-tracker=192.168.0.73:<PORT>
  2. jobTracker=192.168.0.73:<PORT>

where <PORT>= Value from Ambari UI -> YARN ->yarn.resourcemanager.address

And then try again.

View solution in original post

2,704 Views
4 REPLIES 4

avatar
author-rank nshelke author-rank
Guru

Created ‎05-26-2017 09:36 AM

@Hugo Felix

Is your jobhistory server running in ambari or not?

Please check and confirm. Also share your job.properties file contents.

2,704 Views

avatar
author-rank hsfelix
Rising Star

Created ‎05-26-2017 10:32 AM

@nshelke

Yes it's running in ambari.

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
nameNode=hdfs://192.168.0.73:8020
job-tracker=hdfs://192.168.0.73:8021
jobTracker=hdfs://192.168.0.73:8021
workflowRoot=${nameNode}/user/admin/oozie-workflows
# jobStart and jobEnd must be in UTC, because Oozie does not yet support
# custom timezones
jobStart=2017-05-22T17:00Z
jobEnd=2018-12-31T23:00Z
# This should be set to an hour boundary. In this case, it is set to 8 hours
# before the jobStart, since PST is GMT-8
initialDataset=2017-05-21T14:00Z
# Timezone offset between UTC and the server timezone
tzOffset=1
oozie.use.system.libpath=true
oozie.coord.application.path=${nameNode}/user/admin/oozie-workflows/coord-app.xml

Coord-app.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<coordinator-app name="add-partition-coord" frequency="${coord:hours(1)}"
                 start="${jobStart}" end="${jobEnd}"
                 timezone="UTC"
                 xmlns="uri:oozie:coordinator:0.1">
  <datasets>
    <dataset name="tweets" frequency="${coord:hours(1)}"
             initial-instance="${initialDataset}" timezone="America/Los_Angeles">
      <uri-template>hdfs://hadoop1:8020/user/flume/tweets/${YEAR}/${MONTH}/${DAY}/${HOUR}</uri-template>
      <done-flag></done-flag>
    </dataset>
  </datasets>
  <input-events>
    <data-in name="input" dataset="tweets">
      <instance>${coord:current(coord:tzOffset() / 60)}</instance>
    </data-in>
    <data-in name="readyIndicator" dataset="tweets">
      <!-- I've done something here that is a little bit of a hack. Since Flume
           doesn't have a good mechanism for notifying an application of when it has
           rolled to a new directory, we can just use the next directory as an
           input event, which instructs Oozie not to kick off a coordinator
           action until the next dataset starts being available. -->
      <instance>${coord:current(1 + (coord:tzOffset() / 60))}</instance>
    </data-in>
  </input-events>
  <action>
    <workflow>
      <app-path>${workflowRoot}/hive-action.xml</app-path>
      <configuration>
        <property>
          <name>wfInput</name>
          <value>${coord:dataIn('input')}</value>
        </property>
        <property>
          <name>dateHour</name>
          <value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), tzOffset, 'HOUR'), 'yyyyMMddHH')}</value>
        </property>
      </configuration>
    </workflow>
  </action>
</coordinator-app>

hive-action.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:0.2" name="hive-add-partition-wf">
  <start to="hive-add-partition"/>

  <action name="hive-add-partition">
    <hive xmlns="uri:oozie:hive-action:0.2">
      <job-tracker>${jobTracker}</job-tracker>
      <name-node>${nameNode}</name-node>
      <prepare>
        <delete path="${workflowRoot}/output-data/hive"/>
        <mkdir path="${workflowRoot}/output-data"/>
      </prepare>
      <job-xml>${workflowRoot}/hive-site.xml</job-xml>
      <configuration>
        <property>
          <name>oozie.hive.defaults</name>
          <value>${workflowRoot}/hive-site.xml</value>
        </property>
      </configuration>
      <script>add_partition.q</script>
      <param>JSON_SERDE=${workflowRoot}/lib/json-serde-1.3.6-SNAPSHOT-jar-with-dependencies.jar</param>
      <param>WFINPUT=${wfInput}</param>
      <param>DATEHOUR=${dateHour}</param>
    </hive>
    <ok to="end"/>
    <error to="fail"/>
  </action>

  <kill name="fail">
    <message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  </kill>
  <end name="end"/>
</workflow-app>

add_partition.q

--
--  Licensed to the Apache Software Foundation (ASF) under one
--  or more contributor license agreements.  See the NOTICE file
--  distributed with this work for additional information
--  regarding copyright ownership.  The ASF licenses this file
--  to you under the Apache License, Version 2.0 (the
--  "License"); you may not use this file except in compliance
--  with the License.  You may obtain a copy of the License at
--
--       http://www.apache.org/licenses/LICENSE-2.0
--
--  Unless required by applicable law or agreed to in writing, software
--  distributed under the License is distributed on an "AS IS" BASIS,
--  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
--  See the License for the specific language governing permissions and
--  limitations under the License.
--

ADD JAR ${JSON_SERDE};
ALTER TABLE tweets ADD IF NOT EXISTS PARTITION (datehour = ${DATEHOUR}) LOCATION '${WFINPUT}';
2,704 Views

avatar
author-rank nshelke author-rank
Guru

Created ‎05-26-2017 12:28 PM

@Hugo Felix

Can you put value of following properties in job.properties and try again,

Current values in job.properties,

  1. job-tracker=hdfs://192.168.0.73:8021
  2. jobTracker=hdfs://192.168.0.73:8021

Replace with

  1. job-tracker=192.168.0.73:<PORT>
  2. jobTracker=192.168.0.73:<PORT>

where <PORT>= Value from Ambari UI -> YARN ->yarn.resourcemanager.address

And then try again.

2,705 Views

avatar
author-rank hsfelix
Rising Star

Created ‎05-26-2017 03:13 PM

@nshelke Many thanks for your quick answer. Changed the port to 8050... get almost an instant kill.:

2017-05-26 15:05:40,929  WARN HiveActionExecutor:523 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]
2017-05-26 15:05:19,223  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@:start:] Start action [0000010-170525174610830-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-05-26 15:05:19,225  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@:start:] [***0000010-170525174610830-oozie-oozi-W@:start:***]Action status=DONE
2017-05-26 15:05:19,225  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@:start:] [***0000010-170525174610830-oozie-oozi-W@:start:***]Action updated in DB!
2017-05-26 15:05:19,254  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000010-170525174610830-oozie-oozi-W
2017-05-26 15:05:19,254  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000010-170525174610830-oozie-oozi-W@:start:
2017-05-26 15:05:19,261  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] Start action [0000010-170525174610830-oozie-oozi-W@hive-add-partition] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-05-26 15:05:23,376  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] Trying to get job [job_1495727958373_0003], attempt [1]
2017-05-26 15:05:23,392  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] checking action, hadoop job ID [job_1495727958373_0003] status [RUNNING]
2017-05-26 15:05:23,393  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] [***0000010-170525174610830-oozie-oozi-W@hive-add-partition***]Action status=RUNNING
2017-05-26 15:05:23,393  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] [***0000010-170525174610830-oozie-oozi-W@hive-add-partition***]Action updated in DB!
2017-05-26 15:05:23,399  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] No Notification URL is defined. Therefore nothing to notify for job 0000010-170525174610830-oozie-oozi-W@hive-add-partition
2017-05-26 15:05:40,648  INFO CallbackServlet:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] callback for action [0000010-170525174610830-oozie-oozi-W@hive-add-partition]
2017-05-26 15:05:40,776  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] Trying to get job [job_1495727958373_0003], attempt [1]
2017-05-26 15:05:40,883  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] action completed, external ID [job_1495727958373_0003]
2017-05-26 15:05:40,929  WARN HiveActionExecutor:523 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]
2017-05-26 15:05:40,982  INFO ActionEndXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] ERROR is considered as FAILED for SLA
2017-05-26 15:05:41,010  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@fail] Start action [0000010-170525174610830-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-05-26 15:05:41,011  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@fail] [***0000010-170525174610830-oozie-oozi-W@fail***]Action status=DONE
2017-05-26 15:05:41,011  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@fail] [***0000010-170525174610830-oozie-oozi-W@fail***]Action updated in DB!
2017-05-26 15:05:41,070  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@fail] No Notification URL is defined. Therefore nothing to notify for job 0000010-170525174610830-oozie-oozi-W@fail
2017-05-26 15:05:41,070  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000010-170525174610830-oozie-oozi-W
2017-05-26 15:05:41,070  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-170525174610830-oozie-oozi-W] ACTION[0000010-170525174610830-oozie-oozi-W@hive-add-partition] No Notification URL is defined. Therefore nothing to notify for job 0000010-170525174610830-oozie-oozi-W@hive-add-partition
2,704 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements