Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Authorization Issue in Namenode and mapreduce

Highlighted

Authorization Issue in Namenode and mapreduce

Explorer

Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException)
 
For Namenode ---> User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
For Mapreduce---> User:
$RO8000-M3N7MDQMGTVO@XXXX.COM is not allowed to impersonate jhs/xxxx-080060093.xxxx.com@XXXX.COM

Anyone know the solution.

Thanks in Advance

3 REPLIES 3

Re: Authorization Issue in Namenode and mapreduce

Expert Contributor

@ManjunathK We might help if you can attach full error log file. 

Highlighted

Re: Authorization Issue in Namenode and mapreduce

Explorer

@GangWar Please find the attached logs, 

2020-02-27 12:47:24,571 WARN ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(394)) - Unable to trigger a roll of the active NN
java.util.concurrent.ExecutionException: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
Caused by: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy29.rollEditLog(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
... 12 more
2020-02-27 12:47:24,590 WARN namenode.FSEditLog (JournalSet.java:selectInputStreams(272)) - Unable to determine input streams from QJM to [10.80.60.93:8485, 10.80.60.88:8485, 10.80.60.91:8485]. Skipping.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
10.80.60.91:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.93:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.88:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:485)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:269)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1672)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1705)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:299)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
2020-02-27 12:47:25,394 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34535 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:26,396 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34453 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:27,399 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:28,401 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:29,403 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34805 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:29,634 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:47:30,406 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:31,408 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40194 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:32,411 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39393 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:33,413 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35952 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:34,416 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:35,418 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38921 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:36,421 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39596 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:37,423 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36465 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:38,426 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40212 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:39,428 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32804 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:40,431 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:41,433 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33185 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:42,436 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40629 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:43,438 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35358 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:44,440 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35313 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:45,443 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:46,445 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38930 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:47,447 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40397 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:48,450 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33575 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:49,452 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:50,454 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:51,457 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35674 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:52,459 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38558 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:53,462 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35255 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:54,464 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40883 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:55,466 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37031 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:56,469 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:57,471 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:58,474 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37358 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:59,476 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36990 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:47:59,654 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:47:59,738 INFO provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(312)) - Audit Status Log: name=hdfs.async.multi_dest.batch.solr, interval=01:00.011 minutes, events=1, failedCount=1, totalEvents=4497, totalFailedCount=4497
2020-02-27 12:47:59,740 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 0
2020-02-27 12:47:59,742 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 1
2020-02-27 12:47:59,743 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 2
2020-02-27 12:47:59,745 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 3
2020-02-27 12:47:59,746 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 4
2020-02-27 12:47:59,747 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 5
2020-02-27 12:47:59,747 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(374)) - failed to log audit event: {"repoType":1,"repo":"dev-33_hadoop","reqUser":"mapred","evtTime":"2020-02-13 08:31:15.892","access":"READ_EXECUTE","resource":"/mr-history/tmp","resType":"path","action":"execute","result":1,"policy":-1,"reason":"/mr-history/tmp","enforcer":"hadoop-acl","cliIP":"10.80.60.93","agentHost":"xxxx-080060093.xxxx.com","logType":"RangerAudit","id":"f796585a-9fe3-4f9f-af5d-863470dbd766-2","seq_num":3,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"dev-33"}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:821)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1263)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:35)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523)
at org.apache.ranger.audit.utils.SolrAppUtil.addDocsToSolr(SolrAppUtil.java:32)
at org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:225)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:172)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:426)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:796)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:406)
... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 10 more
2020-02-27 12:47:59,748 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(383)) - Log failure count: 1 in past 01:00.010 minutes; 4498 during process lifetime
2020-02-27 12:47:59,748 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(709)) - Error sending logs to consumer. provider=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:47:59,748 INFO queue.AuditFileSpool (AuditFileSpool.java:runLogAudit(769)) - Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:48:00,478 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:01,482 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:02,484 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35750 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:03,486 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39553 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:04,489 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33346 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:05,491 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:06,494 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39028 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:07,496 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:08,499 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39962 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:09,501 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40086 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:10,503 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:11,505 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34705 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:12,508 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36411 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:13,510 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32770 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:14,513 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:15,515 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34307 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:16,517 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:17,520 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:18,522 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:19,524 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:20,527 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37471 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:21,529 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37323 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:22,531 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39352 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:23,534 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34100 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:24,536 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:24,655 INFO ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(378)) - Triggering log roll on remote NameNode
2020-02-27 12:48:25,538 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34796 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:25,656 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:26,541 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40910 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:26,657 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:27,543 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:27,657 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:28,546 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:28,658 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:29,548 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39941 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:29,658 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:29,678 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:48:30,550 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36315 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:30,659 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:31,553 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:31,660 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:32,555 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:32,660 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:33,558 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36277 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:33,661 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:34,560 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:34,661 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:35,562 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34999 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:35,662 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 10 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:36,565 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38030 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:36,663 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 11 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:37,567 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40625 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:37,663 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 12 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:38,570 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38451 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:38,664 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 13 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:39,572 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40695 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:39,665 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 14 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:40,575 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:40,665 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 15 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:41,577 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:41,666 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 16 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:42,579 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39076 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:42,666 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 17 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:43,582 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36581 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:43,667 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 18 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:44,584 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:44,667 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 19 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:45,587 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34352 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:45,668 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 20 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:46,589 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38307 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:46,669 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 21 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:47,591 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:47,669 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 22 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:48,594 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40947 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:48,670 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 23 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:49,596 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:40287 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:49,670 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 24 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:50,599 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:50,671 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 25 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:51,601 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:51,671 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 26 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:52,604 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39461 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:52,672 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 27 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:53,606 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34234 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:53,672 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 28 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:54,609 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38251 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:54,673 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 29 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:55,611 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:55,674 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 30 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:56,614 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:34565 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:56,674 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 31 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:57,616 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:57,675 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 32 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:58,619 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33986 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:58,675 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 33 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:59,621 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:48:59,676 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 34 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:48:59,698 ERROR client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(167)) - Error getting policies for serviceName=dev-33_hadoopresponse=null
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3097)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
... 13 more
Caused by: java.lang.NullPointerException
2020-02-27 12:48:59,748 INFO provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(312)) - Audit Status Log: name=hdfs.async.multi_dest.batch.solr, interval=01:00.010 minutes, events=1, failedCount=1, totalEvents=4498, totalFailedCount=4498
2020-02-27 12:48:59,751 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 0
2020-02-27 12:48:59,752 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 1
2020-02-27 12:48:59,754 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 2
2020-02-27 12:48:59,756 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 3
2020-02-27 12:48:59,757 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 4
2020-02-27 12:48:59,758 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(1167)) - Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 5
2020-02-27 12:48:59,758 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(374)) - failed to log audit event: {"repoType":1,"repo":"dev-33_hadoop","reqUser":"mapred","evtTime":"2020-02-13 08:31:15.892","access":"READ_EXECUTE","resource":"/mr-history/tmp","resType":"path","action":"execute","result":1,"policy":-1,"reason":"/mr-history/tmp","enforcer":"hadoop-acl","cliIP":"10.80.60.93","agentHost":"xxxx-080060093.xxxx.com","logType":"RangerAudit","id":"f796585a-9fe3-4f9f-af5d-863470dbd766-2","seq_num":3,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"dev-33"}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:821)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1263)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1192)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:35)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523)
at org.apache.ranger.audit.utils.SolrAppUtil.addDocsToSolr(SolrAppUtil.java:32)
at org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:225)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:172)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:426)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:796)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://xxxx-080060093.xxxx.com:8886/solr/ranger_audits_shard1_replica_n1
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:406)
... 6 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 10 more
2020-02-27 12:48:59,758 WARN provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(383)) - Log failure count: 1 in past 01:00.011 minutes; 4499 during process lifetime
2020-02-27 12:48:59,758 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(709)) - Error sending logs to consumer. provider=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:48:59,759 INFO queue.AuditFileSpool (AuditFileSpool.java:runLogAudit(769)) - Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=hdfs.async.multi_dest.batch, consumer=hdfs.async.multi_dest.batch.solr
2020-02-27 12:49:00,624 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35278 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:00,677 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 35 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:01,627 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:36596 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:01,677 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 36 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:02,630 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:02,678 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 37 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:03,632 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:33301 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:03,679 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 38 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:04,635 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35221 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:04,679 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 39 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:05,637 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:05,680 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 40 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:06,640 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:06,681 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 41 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:07,642 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:35195 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:07,681 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 42 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:08,645 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37055 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:08,682 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 43 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:09,649 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:09,682 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 44 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:10,651 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:37476 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:10,683 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 45 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:11,654 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:11,683 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 46 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:12,656 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:39556 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:12,684 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 47 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:13,659 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:38001 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:13,684 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 48 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:14,661 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 10.80.60.93:32769 for protocol org.apache.hadoop.ha.HAServiceProtocol is unauthorized for user nn/xxxx-080060093.xxxx.com@XXXX.COM (auth:PROXY) via $6H4000-5GC6FEEDKVFM@XXXX.COM (auth:KERBEROS)
2020-02-27 12:49:14,685 INFO ipc.Client (Client.java:handleConnectionFailure(942)) - Retrying connect to server: xxxx-080060088.xxxx.com/10.80.60.88:8020. Already tried 49 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2020-02-27 12:49:14,686 WARN ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(394)) - Unable to trigger a roll of the active NN
java.util.concurrent.ExecutionException: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:382)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:441)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
Caused by: java.net.ConnectException: Call From xxxx-080060093.xxxx.com/10.80.60.93 to xxxx-080060088.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy29.rollEditLog(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:150)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:367)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:364)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:514)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
... 12 more
2020-02-27 12:49:14,704 WARN namenode.FSEditLog (JournalSet.java:selectInputStreams(272)) - Unable to determine input streams from QJM to [10.80.60.93:8485, 10.80.60.88:8485, 10.80.60.91:8485]. Skipping.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
10.80.60.91:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.93:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
10.80.60.88:8485: User: $6H4000-5GC6FEEDKVFM@XXXX.COM is not allowed to impersonate nn/xxxx-080060093.xxxx.com@XXXX.COM
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:485)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:269)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1672)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1705)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:299)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:480)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)

Highlighted

Re: Authorization Issue in Namenode and mapreduce

Contributor

Hi @ManjunathK .

 

I think HDFS level authentication is required for current user XXX.

 

 

Please check HDFS files level access/Permission  for current user .(used file in map process)

 

 

Thanks

HadoopHelp

Don't have an account?
Coming from Hortonworks? Activate your account here