Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Atlas: Metadata Server Web UI: Connection failed to http://xx-xxx-xx-xxxx.xxxxx.xx:21000/

avatar
author-rank Koffi
Contributor

Created ‎10-26-2021 02:15 PM

Hi

i have the following Atlas error on ambari UI for some days :

Koffi_0-1635281327844.png

and in the atlas logs i can see the following lines:

 

Hi

i have the following Atlas error on ambari UI for some days :

 

2021-10-26 16:47:45,958 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ Initiating re-login for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (KerberosLogin:364)
2021-10-26 16:47:54,277 WARN  - [Thread-5:] ~ Not attempting to re-login since the last re-login was attempted less than 60 seconds before. (Login:330)
2021-10-26 16:47:54,277 WARN  - [Thread-5:] ~ No TGT found: will try again at Tue Oct 26 16:48:54 EDT 2021 (Login:136)
2021-10-26 16:47:54,277 INFO  - [Thread-5:] ~ TGT refresh sleeping until: Tue Oct 26 16:48:54 EDT 2021 (Login:181)
2021-10-26 16:47:55,958 WARN  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. (KerberosLogin:332)
2021-10-26 16:47:55,958 WARN  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: No TGT found: will try again at Tue Oct 26 16:48:55 EDT 2021 (KerberosLogin:143)
2021-10-26 16:47:55,958 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: TGT refresh sleeping until: Tue Oct 26 16:48:55 EDT 2021 (KerberosLogin:188)
2021-10-26 16:48:02,170 WARN  - [Thread-11:] ~ Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1635281252168 (UserGroupInformation:1221)
2021-10-26 16:48:02,170 ERROR - [Thread-11:] ~ Error getting policies for serviceName=xxxxxxxxx_xxxxxresponse=null (RangerAdminRESTClient:167)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Too many open files
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
        at com.sun.jersey.api.client.Client.handle(Client.java:652)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.connect(Socket.java:587)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
        ... 13 more
2021-10-26 16:48:32,171 ERROR - [Thread-11:] ~ Error renewing TGT and relogin. Ignoring Exception, and continuing with the old TGT (MiscUtil:510)
org.apache.hadoop.security.KerberosAuthException: Login failure for user: atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX javax.security.auth.login.LoginException: Cannot locate KDC
        at org.apache.hadoop.security.UserGroupInformation.unprotectedRelogin(UserGroupInformation.java:1191)
        at org.apache.hadoop.security.UserGroupInformation.relogin(UserGroupInformation.java:1157)
        at org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1126)
        at org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1058)
        at org.apache.ranger.audit.provider.MiscUtil.getUGILoginUser(MiscUtil.java:508)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:106)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
        at org.apache.hadoop.security.UserGroupInformation.unprotectedRelogin(UserGroupInformation.java:1185)
        ... 8 more
Caused by: KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Config.java:1084)
        at sun.security.krb5.KdcComm.send(KdcComm.java:218)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 21 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm XXXX.XXXXX.XX
        at sun.security.krb5.Config.getKDCFromDNS(Config.java:1181)
        at sun.security.krb5.Config.getKDCList(Config.java:1057)
        ... 26 more
2021-10-26 16:48:32,173 ERROR - [Thread-11:] ~ Error getting policies for serviceName=xxxxxxxxx_xxxxxresponse=null (RangerAdminRESTClient:167)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Too many open files
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
        at com.sun.jersey.api.client.Client.handle(Client.java:652)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.connect(Socket.java:587)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
        ... 13 more
2021-10-26 16:48:54,277 INFO  - [Thread-5:] ~ Initiating logout for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (Login:389)
2021-10-26 16:48:54,277 INFO  - [Thread-5:] ~ Initiating re-login for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (Login:398)
2021-10-26 16:48:54,279 WARN  - [Thread-5:] ~ Could not refresh TGT for principal: atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX. Will retry. (Login:235)
javax.security.auth.login.LoginException: Cannot locate KDC
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.zookeeper.Login.reLogin(Login.java:399)
        at org.apache.zookeeper.Login.access$400(Login.java:43)
        at org.apache.zookeeper.Login$1.run(Login.java:230)
        at java.lang.Thread.run(Thread.java:748)
Caused by: KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Config.java:1084)
        at sun.security.krb5.KdcComm.send(KdcComm.java:218)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 15 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm XXXX.XXXXX.XX
        at sun.security.krb5.Config.getKDCFromDNS(Config.java:1181)
        at sun.security.krb5.Config.getKDCList(Config.java:1057)
        ... 20 more
2021-10-26 16:48:55,958 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ Initiating logout for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (KerberosLogin:354)
2021-10-26 16:48:55,958 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ Initiating re-login for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (KerberosLogin:364)
2021-10-26 16:49:02,174 WARN  - [Thread-11:] ~ Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1635281312170 (UserGroupInformation:1221)
2021-10-26 16:49:02,174 ERROR - [Thread-11:] ~ Error getting policies for serviceName=xxxxxxxxx_xxxxxresponse=null (RangerAdminRESTClient:167)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Too many open files
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
        at com.sun.jersey.api.client.Client.handle(Client.java:652)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.connect(Socket.java:587)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
        ... 13 more
2021-10-26 16:49:04,279 WARN  - [Thread-5:] ~ Not attempting to re-login since the last re-login was attempted less than 60 seconds before. (Login:330)
2021-10-26 16:49:04,279 WARN  - [Thread-5:] ~ No TGT found: will try again at Tue Oct 26 16:50:04 EDT 2021 (Login:136)
2021-10-26 16:49:04,279 INFO  - [Thread-5:] ~ TGT refresh sleeping until: Tue Oct 26 16:50:04 EDT 2021 (Login:181)
2021-10-26 16:49:05,959 WARN  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. (KerberosLogin:332)
2021-10-26 16:49:05,959 WARN  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: No TGT found: will try again at Tue Oct 26 16:50:05 EDT 2021 (KerberosLogin:143)
2021-10-26 16:49:05,959 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: TGT refresh sleeping until: Tue Oct 26 16:50:05 EDT 2021 (KerberosLogin:188)
2021-10-26 16:49:32,175 ERROR - [Thread-11:] ~ Error renewing TGT and relogin. Ignoring Exception, and continuing with the old TGT (MiscUtil:510)
org.apache.hadoop.security.KerberosAuthException: Login failure for user: atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX javax.security.auth.login.LoginException: Cannot locate KDC
        at org.apache.hadoop.security.UserGroupInformation.unprotectedRelogin(UserGroupInformation.java:1191)
        at org.apache.hadoop.security.UserGroupInformation.relogin(UserGroupInformation.java:1157)
        at org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1126)
        at org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1058)
        at org.apache.ranger.audit.provider.MiscUtil.getUGILoginUser(MiscUtil.java:508)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:106)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
        at org.apache.hadoop.security.UserGroupInformation.unprotectedRelogin(UserGroupInformation.java:1185)
        ... 8 more
Caused by: KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Config.java:1084)
        at sun.security.krb5.KdcComm.send(KdcComm.java:218)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 21 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm XXXX.XXXXX.XX
        at sun.security.krb5.Config.getKDCFromDNS(Config.java:1181)
        at sun.security.krb5.Config.getKDCList(Config.java:1057)
        ... 26 more
2021-10-26 16:49:32,176 ERROR - [Thread-11:] ~ Error getting policies for serviceName=xxxxxxxxx_xxxxxresponse=null (RangerAdminRESTClient:167)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Too many open files
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
        at com.sun.jersey.api.client.Client.handle(Client.java:652)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.connect(Socket.java:587)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
        ... 13 more
2021-10-26 16:50:02,176 WARN  - [Thread-11:] ~ Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1635281372174 (UserGroupInformation:1221)
2021-10-26 16:50:02,177 ERROR - [Thread-11:] ~ Error getting policies for serviceName=xxxxxxxxx_xxxxxresponse=null (RangerAdminRESTClient:167)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Too many open files
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
        at com.sun.jersey.api.client.Client.handle(Client.java:652)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.connect(Socket.java:587)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
        ... 13 more
2021-10-26 16:50:04,279 INFO  - [Thread-5:] ~ Initiating logout for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (Login:389)
2021-10-26 16:50:04,280 INFO  - [Thread-5:] ~ Initiating re-login for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (Login:398)
2021-10-26 16:50:04,281 WARN  - [Thread-5:] ~ Could not refresh TGT for principal: atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX. Will retry. (Login:235)
javax.security.auth.login.LoginException: Cannot locate KDC
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.zookeeper.Login.reLogin(Login.java:399)
        at org.apache.zookeeper.Login.access$400(Login.java:43)
        at org.apache.zookeeper.Login$1.run(Login.java:230)
        at java.lang.Thread.run(Thread.java:748)
Caused by: KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Config.java:1084)
        at sun.security.krb5.KdcComm.send(KdcComm.java:218)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 15 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm XXXX.XXXXX.XX
        at sun.security.krb5.Config.getKDCFromDNS(Config.java:1181)
        at sun.security.krb5.Config.getKDCList(Config.java:1057)
        ... 20 more
2021-10-26 16:50:05,959 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ Initiating logout for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (KerberosLogin:354)
2021-10-26 16:50:05,959 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ Initiating re-login for atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX (KerberosLogin:364)
^_2021-10-26 16:50:14,282 WARN  - [Thread-5:] ~ Not attempting to re-login since the last re-login was attempted less than 60 seconds before. (Login:330)
2021-10-26 16:50:14,282 WARN  - [Thread-5:] ~ No TGT found: will try again at Tue Oct 26 16:51:14 EDT 2021 (Login:136)
2021-10-26 16:50:14,282 INFO  - [Thread-5:] ~ TGT refresh sleeping until: Tue Oct 26 16:51:14 EDT 2021 (Login:181)
2021-10-26 16:50:15,960 WARN  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. (KerberosLogin:332)
2021-10-26 16:50:15,960 WARN  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: No TGT found: will try again at Tue Oct 26 16:51:15 EDT 2021 (KerberosLogin:143)
2021-10-26 16:50:15,961 INFO  - [kafka-kerberos-refresh-thread-atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX:] ~ [Principal=atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX]: TGT refresh sleeping until: Tue Oct 26 16:51:15 EDT 2021 (KerberosLogin:188)
2021-10-26 16:50:32,177 ERROR - [Thread-11:] ~ Error renewing TGT and relogin. Ignoring Exception, and continuing with the old TGT (MiscUtil:510)
org.apache.hadoop.security.KerberosAuthException: Login failure for user: atlas/xx-xxx-xx-xxxx.xxxxx.xx@XXXX.XXXXX.XX javax.security.auth.login.LoginException: Cannot locate KDC
        at org.apache.hadoop.security.UserGroupInformation.unprotectedRelogin(UserGroupInformation.java:1191)
        at org.apache.hadoop.security.UserGroupInformation.relogin(UserGroupInformation.java:1157)
        at org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1126)
        at org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1058)
        at org.apache.ranger.audit.provider.MiscUtil.getUGILoginUser(MiscUtil.java:508)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:106)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
        at org.apache.hadoop.security.UserGroupInformation.unprotectedRelogin(UserGroupInformation.java:1185)
        ... 8 more
Caused by: KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Config.java:1084)
        at sun.security.krb5.KdcComm.send(KdcComm.java:218)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 21 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm XXXX.XXXXX.XX
        at sun.security.krb5.Config.getKDCFromDNS(Config.java:1181)
        at sun.security.krb5.Config.getKDCList(Config.java:1057)
        ... 26 more
2021-10-26 16:50:32,178 ERROR - [Thread-11:] ~ Error getting policies for serviceName=xxxxxxxxx_xxxxxresponse=null (RangerAdminRESTClient:167)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Too many open files
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
        at com.sun.jersey.api.client.Client.handle(Client.java:652)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:120)
        at org.apache.ranger.admin.client.RangerAdminRESTClient$3.run(RangerAdminRESTClient.java:113)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:123)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
        at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.connect(Socket.java:587)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
        ... 13 more

 

Please advise.  Thanks! Koffi

 

Environnement infos:

HDP-3.0.1.0

HDFS 3.1.0
YARN 3.1.0
MapReduce2 3.0.0.3.0
Hive 3.0.0.3.0
HBase 2.0.0.3.0
ZooKeeper 3.4.9.3.0
Ambari Metrics 0.1.0
Atlas 0.7.0.3.0
Kafka 1.0.0.3.0
Knox 0.5.0.3.0
Ranger 1.0.0.3.0
Kerberos 1.10.3-30



1,224 Views
0 Kudos
1 REPLY 1

avatar
author-rank Shelton
Master Mentor

Created ‎10-27-2021 11:16 AM

@Koffi 

There are a couple of things here you first need to resolve too many open files issue by checking the ulimit

$ ulimit -n

To increase for the current session depending on the above output

ulimit -n 102400

Edit /etc/security/limits.conf to make the change permanent.

Then restart the kdc and kadmin depending on your Linux version systemctl  

# /etc/rc.d/init.d/krb5kdc start
# /etc/rc.d/init.d/kadmin start

 

Then restart Atlas from the Ambari UI 

 

Please revert after these actions

 

Geoffrey

 

 

 

 

 

 

 

 

 

1,191 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements