Reply
Explorer
Posts: 9
Registered: ‎11-12-2018
Accepted Solution

Cloudera Manager cannot find default kerberos realm

I've configured kerberos using cloudera manager 5.13 with open ldap as its backend and sssd for the groups name mapping. I'm able to successfully kinit and klist as well as run jobs on the cluster.

 

However, when I try to open the Snapshots section or the File browser section, I get the following exception:

 

 

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)

         at com.google.common.cache.LocalCache.get(LocalCache.java:4000)

         at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)

         at com.cloudera.cmf.service.GenericServiceCdhClient.<init>(GenericServiceCdhClient.java:148)

         at com.cloudera.cmf.service.GenericServiceCdhClient.<init>(GenericServiceCdhClient.java:102)

         at com.cloudera.cmf.service.hdfs.HdfsClient.<init>(HdfsClient.java:61)

         at com.cloudera.api.dao.impl.SnapshotManagerDaoImpl.getSnapshottableDirListing(SnapshotManagerDaoImpl.java:539)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:498)

         at com.cloudera.api.dao.impl.ManagerDaoBase.invokeMethodInExistingTransaction(ManagerDaoBase.java:327)

         at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:274)

         at com.sun.proxy.$Proxy178.getSnapshottableDirListing(Unknown Source)

         at com.cloudera.server.web.cmf.bdr2.BDR2SnapshotPoliciesDTO.<init>(BDR2SnapshotPoliciesDTO.java:170)

         at com.cloudera.server.web.cmf.bdr2.BDR2SnapshotPoliciesDTO.<init>(BDR2SnapshotPoliciesDTO.java:134)

         at com.cloudera.server.web.cmf.bdr2.BDR2Controller.snapshotPoliciesJson(BDR2Controller.java:142)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:498)

         at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)

         at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)

         at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)

         at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)

         at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

         at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)

         at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)

         at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)

         at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)

         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)

         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)

         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

         at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)

         at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

         at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

         at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)

         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)

         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

         at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)

         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

         at org.mortbay.jetty.Server.handle(Server.java:326)

         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)

         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)

         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at com.google.common.base.Throwables.propagate(Throwables.java:160)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:294)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:417)

         at com.cloudera.cmf.service.GenericServiceCdhClient.newClient(GenericServiceCdhClient.java:289)

         at com.cloudera.cmf.service.GenericServiceCdhClient.access$100(GenericServiceCdhClient.java:56)

         at com.cloudera.cmf.service.GenericServiceCdhClient$2.call(GenericServiceCdhClient.java:144)

         at com.cloudera.cmf.service.GenericServiceCdhClient$2.call(GenericServiceCdhClient.java:135)

         at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)

         at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)

         at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)

         at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)

         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)

         ... 87 more

Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at java.util.concurrent.FutureTask.report(FutureTask.java:122)

         at java.util.concurrent.FutureTask.get(FutureTask.java:192)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:288)

         ... 97 more

Caused by: java.lang.IllegalArgumentException: Can't get Kerberos realm

         at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)

         at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:275)

         at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)

         at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:337)

         at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:331)

         at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:263)

         at com.cloudera.cmf.cdh5client.CDH5ObjectFactoryImpl.login(CDH5ObjectFactoryImpl.java:191)

         at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:579)

         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

         at java.util.concurrent.FutureTask.run(FutureTask.java:266)

         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

         at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.reflect.InvocationTargetException

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:498)

         at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84)

         at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)

         ... 12 more

Caused by: KrbException: Cannot locate default realm

         at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)

         ... 18 more

and the following is my effective krb.conf file generated by the cloduera manager:

 

[libdefaults]
default_realm = CLIENT.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
udp_preference_limit = 1
kdc_timeout = 10000
default_realm = CLIENT.COM
dns_lookup_realm = false
dns_lookup_kdc = false
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
 
[realms]
CLIENT.COM = {
kdc = d1master03-nn.client
admin_server = d1master03-nn.client
default_domain = .client
database_module = openldap_ldapconf
kdc=p1master03-nn.client
admin_server=p1master03-nn.client
}
[domain_realm]
.client = CLIENT.COM
client = CLIENT.COM
[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}
 
[dbmodules]
openldap_ldapconf = {
db_library = kldap
ldap_kerberos_container_dn = cn=kerberos,dc=client,dc=com,dc=sa
ldap_kdc_dn = cn=Manager,dc=client,dc=com,dc=sa
# this object needs to have read rights on
# the realm container, principal container and realm sub-trees
ldap_kadmind_dn = cn=Manager,dc=client,dc=com,dc=sa
# this object needs to have read and write rights on
# the realm container, principal container and realm sub-trees
ldap_service_password_file = /etc/krb5.d/stash.keyfile
ldap_servers = ldapi:/// ldap:///d1master03-nn.client:389
ldap_conns_per_server = 5
}
 

Whats the issue here?

 

Explorer
Posts: 9
Registered: ‎11-12-2018

Re: Cloudera Manager cannot find default kerberos realm

I needed to restart the cloudera scm server by running the following command on the cluster where cloudera manager is installed:

 

systemctl restart cloudera-scm-server

systemctl restart cloudera-scm-agent