Cloudera Community

Welcome to the Cloudera Community

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

Oozie Workflow with a Spark action in a kerberized cluster - Error

avatar
author-rank VGIT
New Contributor

Created ‎08-03-2017 09:09 AM

I'm using a kerberized CDH 5.8.3 for some test.

I have a spark program that works well when launched from a ssh session using a specific user (i.e. user01).

Then I created an Oozie Workflow containing a single spark action that is configured with several options and spark properties which are needed by the spark application.

I have added also some extra configuration I have read around the blogs about similar problem; but nothing works out.

The log I have is the following:

2017-08-03 17:14:58,281 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@:start:] Start action [0000035-170731101034824-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-08-03 17:14:58,281 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@:start:] [***0000035-170731101034824-oozie-oozi-W@:start:***]Action status=DONE
2017-08-03 17:14:58,281 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@:start:] [***0000035-170731101034824-oozie-oozi-W@:start:***]Action updated in DB!
2017-08-03 17:14:58,315 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[nyhost.myrealm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000035-170731101034824-oozie-oozi-W
2017-08-03 17:14:58,315 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[nyhost.myrealm] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000035-170731101034824-oozie-oozi-W@:start:
2017-08-03 17:14:58,330 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@spark-f02e] Start action [0000035-170731101034824-oozie-oozi-W@spark-f02e] with user-retry state : userRetryCount [0], userRetryMax [1], userRetryInterval [10]
2017-08-03 17:24:08,862 WARN org.apache.oozie.action.hadoop.HbaseCredentials: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@spark-f02e] Exception in receiving hbase credentials
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
Thu Aug 03 17:14:58 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=0, waitTime=2
Thu Aug 03 17:14:58 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=1, waitTime=2
Thu Aug 03 17:14:59 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=2, waitTime=2
Thu Aug 03 17:15:00 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=3, waitTime=2
Thu Aug 03 17:15:02 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=4, waitTime=1
Thu Aug 03 17:15:06 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=5, waitTime=2
Thu Aug 03 17:15:16 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=6, waitTime=3
Thu Aug 03 17:15:26 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=7, waitTime=1
Thu Aug 03 17:15:36 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=8, waitTime=2
Thu Aug 03 17:15:46 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=9, waitTime=2
Thu Aug 03 17:16:06 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=10, waitTime=2
Thu Aug 03 17:16:26 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=11, waitTime=2
Thu Aug 03 17:16:46 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=12, waitTime=2
Thu Aug 03 17:17:06 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=13, waitTime=2
Thu Aug 03 17:17:26 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=14, waitTime=2
Thu Aug 03 17:17:47 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=15, waitTime=2
Thu Aug 03 17:18:07 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=16, waitTime=1
Thu Aug 03 17:18:27 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=17, waitTime=2
Thu Aug 03 17:18:47 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=18, waitTime=1
Thu Aug 03 17:19:07 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=19, waitTime=2
Thu Aug 03 17:19:27 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=20, waitTime=2
Thu Aug 03 17:19:47 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=21, waitTime=2
Thu Aug 03 17:20:07 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=22, waitTime=2
Thu Aug 03 17:20:27 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=23, waitTime=1
Thu Aug 03 17:20:47 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=24, waitTime=2
Thu Aug 03 17:21:07 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=25, waitTime=2
Thu Aug 03 17:21:28 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=26, waitTime=2
Thu Aug 03 17:21:48 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=27, waitTime=2
Thu Aug 03 17:22:08 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=28, waitTime=1
Thu Aug 03 17:22:28 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=29, waitTime=2
Thu Aug 03 17:22:48 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=30, waitTime=2
Thu Aug 03 17:23:08 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=31, waitTime=2
Thu Aug 03 17:23:28 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=32, waitTime=2
Thu Aug 03 17:23:48 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=33, waitTime=1
Thu Aug 03 17:24:08 CEST 2017, RpcRetryingCaller{globalStartTime=1501773298487, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2

	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
	at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:69)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:85)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:83)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
	at org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:82)
	at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1211)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1061)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1291)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	... 26 more
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1084)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)
2017-08-03 17:24:08,863 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@spark-f02e] Error starting action [spark-f02e]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2]
org.apache.oozie.action.ActionExecutorException: JA009: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1126)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1291)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
	at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:69)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:85)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:83)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
	at org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:82)
	at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1211)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1061)
	... 11 more
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1084)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)
2017-08-03 17:24:08,863 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@spark-f02e] Next Retry, Attempt Number [1] in [10,000] milliseconds
2017-08-03 17:24:08,884 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@spark-f02e] Start action [0000035-170731101034824-oozie-oozi-W@spark-f02e] with user-retry state : userRetryCount [0], userRetryMax [1], userRetryInterval [10]
2017-08-03 17:27:07,053 INFO org.apache.oozie.DagEngine: SERVER[nyhost.myrealm] USER[hue doAs user01] GROUP[-] TOKEN[-] APP[-] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[] User user01 killed the WF job 0000035-170731101034824-oozie-oozi-W
2017-08-03 17:27:20,788 INFO org.apache.oozie.DagEngine: SERVER[nyhost.myrealm] USER[hue doAs user01] GROUP[-] TOKEN[-] APP[-] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[] User user01 killed the WF job 0000035-170731101034824-oozie-oozi-W
2017-08-03 17:33:19,557 WARN org.apache.oozie.action.hadoop.HbaseCredentials: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@spark-f02e] Exception in receiving hbase credentials
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
Thu Aug 03 17:24:09 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=0, waitTime=2
Thu Aug 03 17:24:09 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=1, waitTime=2
Thu Aug 03 17:24:09 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=2, waitTime=2
Thu Aug 03 17:24:10 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=3, waitTime=1
Thu Aug 03 17:24:12 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=4, waitTime=2
Thu Aug 03 17:24:16 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=5, waitTime=2
Thu Aug 03 17:24:26 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=6, waitTime=2
Thu Aug 03 17:24:36 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=7, waitTime=2
Thu Aug 03 17:24:47 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=8, waitTime=2
Thu Aug 03 17:24:57 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=9, waitTime=2
Thu Aug 03 17:25:17 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=10, waitTime=2
Thu Aug 03 17:25:37 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=11, waitTime=1
Thu Aug 03 17:25:57 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=12, waitTime=2
Thu Aug 03 17:26:17 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=13, waitTime=2
Thu Aug 03 17:26:37 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=14, waitTime=2
Thu Aug 03 17:26:57 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=15, waitTime=2
Thu Aug 03 17:27:17 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=16, waitTime=2
Thu Aug 03 17:27:38 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=17, waitTime=2
Thu Aug 03 17:27:58 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=18, waitTime=1
Thu Aug 03 17:28:18 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=19, waitTime=2
Thu Aug 03 17:28:38 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=20, waitTime=2
Thu Aug 03 17:28:58 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=21, waitTime=2
Thu Aug 03 17:29:18 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=22, waitTime=2
Thu Aug 03 17:29:38 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=23, waitTime=2
Thu Aug 03 17:29:58 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=24, waitTime=2
Thu Aug 03 17:30:18 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=25, waitTime=2
Thu Aug 03 17:30:39 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=26, waitTime=2
Thu Aug 03 17:30:59 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=27, waitTime=2
Thu Aug 03 17:31:19 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=28, waitTime=2
Thu Aug 03 17:31:39 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=29, waitTime=2
Thu Aug 03 17:31:59 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=30, waitTime=2
Thu Aug 03 17:32:19 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=31, waitTime=1
Thu Aug 03 17:32:39 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=32, waitTime=2
Thu Aug 03 17:32:59 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=33, waitTime=2
Thu Aug 03 17:33:19 CEST 2017, RpcRetryingCaller{globalStartTime=1501773849045, pause=100, retries=35}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2

	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
	at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:69)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:85)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:83)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
	at org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:82)
	at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1211)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1061)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1291)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	... 24 more
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1084)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)
2017-08-03 17:33:19,558 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[nyhost.myrealm] USER[user01] GROUP[-] TOKEN[] APP[VG_Test-Spark-Kerberos] JOB[0000035-170731101034824-oozie-oozi-W] ACTION[0000035-170731101034824-oozie-oozi-W@spark-f02e] Error starting action [spark-f02e]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2]
org.apache.oozie.action.ActionExecutorException: JA009: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1126)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1291)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to myhost01.myrealm/10.xxx.xx.xx:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
	at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
	at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:69)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:85)
	at org.apache.oozie.action.hadoop.HbaseCredentials$1.run(HbaseCredentials.java:83)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
	at org.apache.oozie.action.hadoop.HbaseCredentials.obtainToken(HbaseCredentials.java:82)
	at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:56)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1211)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1061)
	... 9 more
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to myhost01.myrealm/10.xxx.xx.xx:60020 is closing. Call id=34, waitTime=2
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1084)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)

I have checked my configuration several times, but the problem still exists.

 

Any help will be appreciated.

 

 

1,333
0 Kudos
2
Who agreed with this topic