Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Job is Running on Hadoop But Suspended in oozie with JA006 error

Job is Running on Hadoop But Suspended in oozie with JA006 error

New Contributor

How to fix JA006 in Oozie

Oozie Log :

-----------------------------

2018-09-21 17:28:18,537 INFO ActionStartXCommand:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@:start:] Start action [0000012-180920191643837-oozie-hado-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-21 17:28:18,545 INFO ActionStartXCommand:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@:start:] [***0000012-180920191643837-oozie-hado-W@:start:***]Action status=DONE
2018-09-21 17:28:18,545 INFO ActionStartXCommand:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@:start:] [***0000012-180920191643837-oozie-hado-W@:start:***]Action updated in DB!
2018-09-21 17:28:18,684 INFO WorkflowNotificationXCommand:520 - SERVER[scipnode4] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000012-180920191643837-oozie-hado-W@:start:
2018-09-21 17:28:18,684 INFO WorkflowNotificationXCommand:520 - SERVER[scipnode4] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000012-180920191643837-oozie-hado-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000012-180920191643837-oozie-hado-W
2018-09-21 17:28:18,759 INFO ActionStartXCommand:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] Start action [0000012-180920191643837-oozie-hado-W@Shell_Action] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-21 17:28:19,919 INFO ShellActionExecutor:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] checking action, hadoop job ID [job_1537252793437_0046] status [RUNNING]
2018-09-21 17:28:19,922 INFO ActionStartXCommand:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] [***0000012-180920191643837-oozie-hado-W@Shell_Action***]Action status=RUNNING
2018-09-21 17:28:19,922 INFO ActionStartXCommand:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] [***0000012-180920191643837-oozie-hado-W@Shell_Action***]Action updated in DB!
2018-09-21 17:28:19,952 INFO WorkflowNotificationXCommand:520 - SERVER[scipnode4] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] No Notification URL is defined. Therefore nothing to notify for job 0000012-180920191643837-oozie-hado-W@Shell_Action
2018-09-21 17:28:58,905 INFO CallbackServlet:520 - SERVER[scipnode4] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] callback for action [0000012-180920191643837-oozie-hado-W@Shell_Action]
2018-09-21 17:29:29,346 WARN ShellActionExecutor:523 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] Exception in check(). Message[java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort]
java.io.IOException: java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:344)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:429)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:804)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:214)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:602)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:600)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:600)
at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:610)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:640)
at org.apache.oozie.action.hadoop.JavaActionExecutor.getRunningJob(JavaActionExecutor.java:1436)
at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1460)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:182)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort
at sun.reflect.GeneratedConstructorAccessor119.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1439)
at org.apache.hadoop.ipc.Client.call(Client.java:1349)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy33.getJobReport(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:325)
... 21 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:411)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1554)
at org.apache.hadoop.ipc.Client.call(Client.java:1385)
... 30 more
2018-09-21 17:29:29,369 WARN ActionCheckXCommand:523 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] Exception while executing check(). Error Code [ JA006], Message[ JA006: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort]
org.apache.oozie.action.ActionExecutorException: JA006: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:457)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)
at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1571)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:182)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort
at sun.reflect.GeneratedConstructorAccessor119.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1439)
at org.apache.hadoop.ipc.Client.call(Client.java:1349)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy33.getJobReport(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:325)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:429)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:804)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:214)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:602)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:600)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:600)
at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:610)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:640)
at org.apache.oozie.action.hadoop.JavaActionExecutor.getRunningJob(JavaActionExecutor.java:1436)
at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1460)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:411)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1554)
at org.apache.hadoop.ipc.Client.call(Client.java:1385)
... 30 more
2018-09-21 17:29:29,373 INFO ActionCheckXCommand:520 - SERVER[scipnode4] USER[hadoop] GROUP[-] TOKEN[] APP[Shell_Action] JOB[0000012-180920191643837-oozie-hado-W] ACTION[0000012-180920191643837-oozie-hado-W@Shell_Action] Next Retry, Attempt Number [1] in [10,000] milliseconds

Hadoop Log

----------------------

yarn/staging/history/done_intermediate/hadoop/job_1537252793437_0044_conf.xml_tmp to hdfs://STERLITE:8020/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1537252793437_0044_conf.xml
2018-09-21 16:07:03,409 INFO [Thread-67] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://STERLITE:8020/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1537252793437_0044-1537526214153-hadoop-oozie%3Alauncher%3AT%3Dshell%3AW%3DShell_Action%3A-1537526223028-1-0-SUCCEEDED-default-1537526217114.jhist_tmp to hdfs://STERLITE:8020/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1537252793437_0044-1537526214153-hadoop-oozie%3Alauncher%3AT%3Dshell%3AW%3DShell_Action%3A-1537526223028-1-0-SUCCEEDED-default-1537526217114.jhist
2018-09-21 16:07:03,409 INFO [Thread-67] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2018-09-21 16:07:03,411 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1537252793437_0044_m_000000_0
2018-09-21 16:07:03,433 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1537252793437_0044_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2018-09-21 16:07:03,450 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to 
2018-09-21 16:07:03,450 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://scipnode25:19888/jobhistory/job/job_1537252793437_0044
2018-09-21 16:07:03,457 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered.
2018-09-21 16:07:04,462 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2018-09-21 16:07:04,463 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://STERLITE:8020 /tmp/hadoop-yarn/staging/hadoop/.staging/job_1537252793437_0044
2018-09-21 16:07:04,471 INFO [Thread-67] org.apache.hadoop.ipc.Server: Stopping server on 34492
2018-09-21 16:07:04,475 INFO [IPC Server listener on 34492] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34492
2018-09-21 16:07:04,475 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2018-09-21 16:07:04,475 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2018-09-21 16:07:04,479 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2018-09-21 16:07:04,493 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job end notification started for jobID : job_1537252793437_0044
2018-09-21 16:07:04,493 INFO [Thread-67] org.mortbay.log: Job end notification attempts left 0
2018-09-21 16:07:04,494 INFO [Thread-67] org.mortbay.log: Job end notification trying http://scipnode4:11000/oozie/callback?id=0000009-180920191643837-oozie-hado-W@Shell_Action&status=SU...
2018-09-21 16:07:04,510 INFO [Thread-67] org.mortbay.log: Job end notification to http://scipnode4:11000/oozie/callback?id=0000009-180920191643837-oozie-hado-W@Shell_Action&status=SU... succeeded
2018-09-21 16:07:04,510 INFO [Thread-67] org.mortbay.log: Job end notification succeeded for job_1537252793437_0044
2018-09-21 16:07:09,512 INFO [Thread-67] org.apache.hadoop.ipc.Server: Stopping server on 42034
2018-09-21 16:07:09,513 INFO [IPC Server listener on 42034] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 42034
2018-09-21 16:07:09,513 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2018-09-21 16:07:09,518 INFO [Thread-67] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0

core-site.xml

<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>hadoop</value>
</property>
<property>

coordinator.prperties

startTime=2018-09-20T09:20Z
endTime=2018-09-21T15:20Z
timezone=GMT+0530
concurrency_level=1
execution_order=FIFO
nameNode=hdfs://STERLITE:8020
jobTracker=scipnode1:8032,scipnode2:8032
queueName=default
user.name=hadoop
oozie.use.system.libpath=true
oozie.libpath=/user/hadoop/share/lib
wf_app_path=${nameNode}/user/${user.name}/oozie/workflow/shell
myscript=test.sh
inputDir=${nameNode}/input
outputDir=${nameNode}/output
oozie.coord.application.path=${nameNode}/user/${user.name}/oozie/coordinator/shell
workflowAppUri=${wf_app_path}
myscriptPath=${wf_app_path}

datanode process

[hadoop@node4 ~]$ netstat -nalp|grep 50020
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 20790/java

Don't have an account?
Coming from Hortonworks? Activate your account here