Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Manager cannot find default kerberos realm

Solved Go to solution

Cloudera Manager cannot find default kerberos realm

Explorer

I've configured kerberos using cloudera manager 5.13 with open ldap as its backend and sssd for the groups name mapping. I'm able to successfully kinit and klist as well as run jobs on the cluster.

 

However, when I try to open the Snapshots section or the File browser section, I get the following exception:

 

 

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)

         at com.google.common.cache.LocalCache.get(LocalCache.java:4000)

         at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)

         at com.cloudera.cmf.service.GenericServiceCdhClient.<init>(GenericServiceCdhClient.java:148)

         at com.cloudera.cmf.service.GenericServiceCdhClient.<init>(GenericServiceCdhClient.java:102)

         at com.cloudera.cmf.service.hdfs.HdfsClient.<init>(HdfsClient.java:61)

         at com.cloudera.api.dao.impl.SnapshotManagerDaoImpl.getSnapshottableDirListing(SnapshotManagerDaoImpl.java:539)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:498)

         at com.cloudera.api.dao.impl.ManagerDaoBase.invokeMethodInExistingTransaction(ManagerDaoBase.java:327)

         at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:274)

         at com.sun.proxy.$Proxy178.getSnapshottableDirListing(Unknown Source)

         at com.cloudera.server.web.cmf.bdr2.BDR2SnapshotPoliciesDTO.<init>(BDR2SnapshotPoliciesDTO.java:170)

         at com.cloudera.server.web.cmf.bdr2.BDR2SnapshotPoliciesDTO.<init>(BDR2SnapshotPoliciesDTO.java:134)

         at com.cloudera.server.web.cmf.bdr2.BDR2Controller.snapshotPoliciesJson(BDR2Controller.java:142)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:498)

         at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)

         at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)

         at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)

         at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)

         at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

         at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)

         at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)

         at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)

         at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)

         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)

         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)

         at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

         at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

         at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)

         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)

         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

         at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)

         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

         at org.mortbay.jetty.Server.handle(Server.java:326)

         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)

         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)

         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at com.google.common.base.Throwables.propagate(Throwables.java:160)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:294)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:417)

         at com.cloudera.cmf.service.GenericServiceCdhClient.newClient(GenericServiceCdhClient.java:289)

         at com.cloudera.cmf.service.GenericServiceCdhClient.access$100(GenericServiceCdhClient.java:56)

         at com.cloudera.cmf.service.GenericServiceCdhClient$2.call(GenericServiceCdhClient.java:144)

         at com.cloudera.cmf.service.GenericServiceCdhClient$2.call(GenericServiceCdhClient.java:135)

         at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)

         at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)

         at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)

         at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)

         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)

         ... 87 more

Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at java.util.concurrent.FutureTask.report(FutureTask.java:122)

         at java.util.concurrent.FutureTask.get(FutureTask.java:192)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:288)

         ... 97 more

Caused by: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)

         at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:275)

         at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)

         at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:337)

         at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:331)

         at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:263)

         at com.cloudera.cmf.cdh5client.CDH5ObjectFactoryImpl.login(CDH5ObjectFactoryImpl.java:191)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:579)

         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

         at java.util.concurrent.FutureTask.run(FutureTask.java:266)

         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

         at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.reflect.InvocationTargetException

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:498)

         at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84)

         at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)

         ... 12 more

Caused by: KrbException: Cannot locate default realm

         at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)

         ... 18 more

and the following is my effective krb.conf file generated by the cloduera manager:

 

[libdefaults]
default_realm = CLIENT.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
udp_preference_limit = 1
kdc_timeout = 10000
default_realm = CLIENT.COM
dns_lookup_realm = false
dns_lookup_kdc = false
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
 
[realms]
CLIENT.COM = {
kdc = d1master03-nn.client
admin_server = d1master03-nn.client
default_domain = .client
database_module = openldap_ldapconf
kdc=p1master03-nn.client
admin_server=p1master03-nn.client
}
[domain_realm]
.client = CLIENT.COM
client = CLIENT.COM
[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}
 
[dbmodules]
openldap_ldapconf = {
db_library = kldap
ldap_kerberos_container_dn = cn=kerberos,dc=client,dc=com,dc=sa
ldap_kdc_dn = cn=Manager,dc=client,dc=com,dc=sa
# this object needs to have read rights on
# the realm container, principal container and realm sub-trees
ldap_kadmind_dn = cn=Manager,dc=client,dc=com,dc=sa
# this object needs to have read and write rights on
# the realm container, principal container and realm sub-trees
ldap_service_password_file = /etc/krb5.d/stash.keyfile
ldap_servers = ldapi:/// ldap:///d1master03-nn.client:389
ldap_conns_per_server = 5
}
 

Whats the issue here?

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Cloudera Manager cannot find default kerberos realm

Explorer

I needed to restart the cloudera scm server by running the following command on the cluster where cloudera manager is installed:

 

systemctl restart cloudera-scm-server

systemctl restart cloudera-scm-agent

 

1 REPLY 1
Highlighted

Re: Cloudera Manager cannot find default kerberos realm

Explorer

I needed to restart the cloudera scm server by running the following command on the cluster where cloudera manager is installed:

 

systemctl restart cloudera-scm-server

systemctl restart cloudera-scm-agent