Support Questions
Find answers, ask questions, and share your expertise

Authorization Issue in Namenode and mapreduce

Highlighted

Authorization Issue in Namenode and mapreduce

Explorer

Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException)
 
For Namenode ---> User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
For Mapreduce---> User:
$RO8000-M3N7MDQMGTVO@XXXX.COM is not allowed to impersonate jhs/xxxx-080060093.xxxx.com@XXXX.COM

Anyone know the solution.

Thanks in Advance

3 REPLIES 3
Highlighted

Re: Authorization Issue in Namenode and mapreduce

Master Collaborator

@ManjunathK We might help if you can attach full error log file. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Highlighted

Re: Authorization Issue in Namenode and mapreduce

Explorer

@GangWar Please find the attached logs, 

2020-02-27 12:47:24,571 WARN ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(394)) - Unable to trigger a roll of the active NN
java.util.concurrent.ExecutionException: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
Caused by: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy29.rollEditLog(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
... 12 more
2020-02-27 12:47:24,590 WARN namenode.FSEditLog (JournalSet.java:selectInputStreams(272)) - Unable to determine input streams from QJM to [10.80.60.93:8485, 10.80.60.88:8485, 10.80.60.91:8485]. Skipping.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
10.80.60.91:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.93:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.88:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:485)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:269)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1672)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1705)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:299)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
2020-02-27 12:47:25,394 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34535 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:26,396 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34453 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:27,399 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:28,401 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:29,403 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34805 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:29,634 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:47:30,406 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:31,408 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40194 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:32,411 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39393 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:33,413 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35952 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:34,416 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:35,418 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38921 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:36,421 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39596 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:37,423 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36465 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:38,426 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40212 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:39,428 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32804 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:40,431 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:41,433 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33185 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:42,436 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40629 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:43,438 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35358 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:44,440 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35313 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:45,443 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:46,445 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38930 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:47,447 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40397 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:48,450 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33575 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:49,452 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:50,454 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:51,457 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35674 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:52,459 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38558 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:53,462 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35255 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:54,464 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40883 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:55,466 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37031 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:56,469 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:57,471 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:58,474 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37358 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:59,476 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36990 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:59,654 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:47:59,738 INFO provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(312)) - Audit Status Log: name=hdfs.async.multi_dest.batch.solr, interval=01:00.011 minutes, events=1, failedCount=1, totalEvents=4497, totalFailedCount=4497
2020-02-27 12:47:59,740 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 0
2020-02-27 12:47:59,742 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 1
2020-02-27 12:47:59,743 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 2
2020-02-27 12:47:59,745 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 3
2020-02-27 12:47:59,746 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 4
2020-02-27 12:47:59,747 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 5
2020-02-27 12:47:59,747 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(374)) - failed to log audit event: {"repoType":1,"repo":"dev-33_hadoop","reqUser":"mapred","evtTime":"2020-02-13 08:31:15.892","access":"READ_EXECUTE","resource":"/mr-history/tmp","resType":"path","action":"execute","result":1,"policy":-1,"reason":"/mr-history/tmp","enforcer":"hadoop-acl","cliIP":"10.80.60.93","agentHost":"xxxx-080060093.xxxx.com","logType":"RangerAudit","id":"f796585a-9fe3-4f9f-af5d-863470dbd766-2","seq_num":3,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"dev-33"}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:821)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1263)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:35)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523)
at org.apache.ranger.audit.utils.SolrAppUtil.addDocsToSolr(SolrAppUtil.java:32)
at org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:225)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:172)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:426)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:796)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:406)
... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 10 more
2020-02-27 12:47:59,748 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(383)) - Log failure count: 1 in past 01:00.010 minutes; 4498 during process lifetime
2020-02-27 12:47:59,748 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(709)) - Error sending logs to consumer. provider=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:47:59,748 INFO queue.AuditFileSpool (AuditFileSpool.java:runLogAudit(769)) - Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:48:00,478 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:01,482 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:02,484 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35750 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:03,486 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39553 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:04,489 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33346 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:05,491 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:06,494 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39028 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:07,496 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:08,499 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39962 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:09,501 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40086 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:10,503 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:11,505 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34705 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:12,508 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36411 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:13,510 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32770 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:14,513 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:15,515 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34307 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:16,517 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:17,520 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:18,522 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:19,524 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:20,527 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37471 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:21,529 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37323 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:22,531 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39352 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:23,534 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34100 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:24,536 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:24,655 INFO ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(378)) - Triggering log roll on remote NameNode
2020-02-27 12:48:25,538 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34796 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:25,656 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:26,541 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40910 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:26,657 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:27,543 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:27,657 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:28,546 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:28,658 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:29,548 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39941 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:29,658 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:29,678 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:48:30,550 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36315 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:30,659 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:31,553 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:31,660 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:32,555 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:32,660 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:33,558 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36277 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:33,661 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:34,560 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:34,661 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:35,562 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34999 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:35,662 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 10 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:36,565 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38030 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:36,663 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 11 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:37,567 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40625 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:37,663 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 12 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:38,570 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38451 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:38,664 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 13 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:39,572 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40695 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:39,665 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 14 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:40,575 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:40,665 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 15 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:41,577 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:41,666 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 16 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:42,579 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39076 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:42,666 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 17 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:43,582 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36581 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:43,667 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 18 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:44,584 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:44,667 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 19 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:45,587 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34352 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:45,668 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 20 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:46,589 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38307 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:46,669 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 21 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:47,591 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:47,669 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 22 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:48,594 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40947 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:48,670 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 23 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:49,596 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40287 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:49,670 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 24 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:50,599 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:50,671 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 25 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:51,601 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:51,671 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 26 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:52,604 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39461 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:52,672 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 27 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:53,606 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34234 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:53,672 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 28 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:54,609 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38251 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:54,673 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 29 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:55,611 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:55,674 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 30 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:56,614 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34565 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:56,674 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 31 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:57,616 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:57,675 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 32 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:58,619 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33986 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:58,675 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 33 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:59,621 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:59,676 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 34 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:59,698 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:48:59,748 INFO provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(312)) - Audit Status Log: name=hdfs.async.multi_dest.batch.solr, interval=01:00.010 minutes, events=1, failedCount=1, totalEvents=4498, totalFailedCount=4498
2020-02-27 12:48:59,751 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 0
2020-02-27 12:48:59,752 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 1
2020-02-27 12:48:59,754 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 2
2020-02-27 12:48:59,756 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 3
2020-02-27 12:48:59,757 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 4
2020-02-27 12:48:59,758 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 5
2020-02-27 12:48:59,758 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(374)) - failed to log audit event: {"repoType":1,"repo":"dev-33_hadoop","reqUser":"mapred","evtTime":"2020-02-13 08:31:15.892","access":"READ_EXECUTE","resource":"/mr-history/tmp","resType":"path","action":"execute","result":1,"policy":-1,"reason":"/mr-history/tmp","enforcer":"hadoop-acl","cliIP":"10.80.60.93","agentHost":"xxxx-080060093.xxxx.com","logType":"RangerAudit","id":"f796585a-9fe3-4f9f-af5d-863470dbd766-2","seq_num":3,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"dev-33"}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:821)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1263)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:35)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523)
at org.apache.ranger.audit.utils.SolrAppUtil.addDocsToSolr(SolrAppUtil.java:32)
at org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:225)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:172)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:426)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:796)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:406)
... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 10 more
2020-02-27 12:48:59,758 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(383)) - Log failure count: 1 in past 01:00.011 minutes; 4499 during process lifetime
2020-02-27 12:48:59,758 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(709)) - Error sending logs to consumer. provider=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:48:59,759 INFO queue.AuditFileSpool (AuditFileSpool.java:runLogAudit(769)) - Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:49:00,624 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35278 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:00,677 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 35 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:01,627 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36596 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:01,677 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 36 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:02,630 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:02,678 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 37 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:03,632 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33301 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:03,679 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 38 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:04,635 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35221 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:04,679 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 39 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:05,637 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:05,680 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 40 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:06,640 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:06,681 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 41 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:07,642 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35195 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:07,681 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 42 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:08,645 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37055 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:08,682 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 43 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:09,649 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:09,682 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 44 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:10,651 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37476 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:10,683 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 45 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:11,654 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:11,683 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 46 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:12,656 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39556 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:12,684 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 47 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:13,659 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38001 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:13,684 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 48 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:14,661 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:14,685 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 49 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:14,686 WARN ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(394)) - Unable to trigger a roll of the active NN
java.util.concurrent.ExecutionException: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
Caused by: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy29.rollEditLog(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
... 12 more
2020-02-27 12:49:14,704 WARN namenode.FSEditLog (JournalSet.java:selectInputStreams(272)) - Unable to determine input streams from QJM to [10.80.60.93:8485, 10.80.60.88:8485, 10.80.60.91:8485]. Skipping.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
10.80.60.91:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.93:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.88:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:485)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:269)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1672)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1705)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:299)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)

Re: Authorization Issue in Namenode and mapreduce

Contributor

Hi @ManjunathK .

 

I think HDFS level authentication is required for current user XXX.

 

 

Please check HDFS files level access/Permission  for current user .(used file in map process)

 

 

Thanks

HadoopHelp

Don't have an account?