I've configured kerberos using cloudera manager 5.13 with open ldap as its backend and sssd for the groups name mapping. I'm able to successfully kinit and klist as well as run jobs on the cluster.   However, when I try to open the Snapshots section or the File browser section, I get the following exception:     com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263) at com.google.common.cache.LocalCache.get(LocalCache.java:4000) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789) at com.cloudera.cmf.service.GenericServiceCdhClient.<init>(GenericServiceCdhClient.java:148) at com.cloudera.cmf.service.GenericServiceCdhClient.<init>(GenericServiceCdhClient.java:102) at com.cloudera.cmf.service.hdfs.HdfsClient.<init>(HdfsClient.java:61) at com.cloudera.api.dao.impl.SnapshotManagerDaoImpl.getSnapshottableDirListing(SnapshotManagerDaoImpl.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudera.api.dao.impl.ManagerDaoBase.invokeMethodInExistingTransaction(ManagerDaoBase.java:327) at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:274) at com.sun.proxy.$Proxy178.getSnapshottableDirListing(Unknown Source) at com.cloudera.server.web.cmf.bdr2.BDR2SnapshotPoliciesDTO.<init>(BDR2SnapshotPoliciesDTO.java:170) at com.cloudera.server.web.cmf.bdr2.BDR2SnapshotPoliciesDTO.<init>(BDR2SnapshotPoliciesDTO.java:134) at com.cloudera.server.web.cmf.bdr2.BDR2Controller.snapshotPoliciesJson(BDR2Controller.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78) at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm at com.google.common.base.Throwables.propagate(Throwables.java:160) at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:294) at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:417) at com.cloudera.cmf.service.GenericServiceCdhClient.newClient(GenericServiceCdhClient.java:289) at com.cloudera.cmf.service.GenericServiceCdhClient.access$100(GenericServiceCdhClient.java:56) at com.cloudera.cmf.service.GenericServiceCdhClient$2.call(GenericServiceCdhClient.java:144) at com.cloudera.cmf.service.GenericServiceCdhClient$2.call(GenericServiceCdhClient.java:135) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257) ... 87 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.cloudera.cmf.cdhclient.CdhExecutorFactory.createExecutor(CdhExecutorFactory.java:288) ... 97 more Caused by: java.lang.IllegalArgumentException: Can't get Kerberos realm at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:275) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:337) at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:331) at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:263) at com.cloudera.cmf.cdh5client.CDH5ObjectFactoryImpl.login(CDH5ObjectFactoryImpl.java:191) at com.cloudera.cmf.cdhclient.CdhExecutorFactory$SecureClassLoaderSetupTask.run(CdhExecutorFactory.java:579) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84) at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63) ... 12 more Caused by: KrbException: Cannot locate default realm at sun.security.krb5.Config.getDefaultRealm(Config.java:1029) ... 18 more and the following is my effective krb.conf file generated by the cloduera manager:   [libdefaults] default_realm = CLIENT.COM dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 udp_preference_limit = 1 kdc_timeout = 10000 default_realm = CLIENT.COM dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = FILE:/tmp/krb5cc_%{uid} [realms] CLIENT.COM = { kdc = d1master03-nn.client admin_server = d1master03-nn.client default_domain = .client database_module = openldap_ldapconf kdc=p1master03-nn.client admin_server=p1master03-nn.client } [domain_realm] .client = CLIENT.COM client = CLIENT.COM [appdefaults] pam = { debug = false ticket_lifetime = 36000 renew_lifetime = 36000 forwardable = true krb4_convert = false } [dbmodules] openldap_ldapconf = { db_library = kldap ldap_kerberos_container_dn = cn=kerberos,dc=client,dc=com,dc=sa ldap_kdc_dn = cn=Manager,dc=client,dc=com,dc=sa # this object needs to have read rights on # the realm container, principal container and realm sub-trees ldap_kadmind_dn = cn=Manager,dc=client,dc=com,dc=sa # this object needs to have read and write rights on # the realm container, principal container and realm sub-trees ldap_service_password_file = /etc/krb5.d/stash.keyfile ldap_servers = ldapi:/// ldap:///d1master03-nn.client:389 ldap_conns_per_server = 5 } Whats the issue here?   ... View more

I've a very weired issue, where my hadoop cluster has run out of space. Upon investagtion I found out that one of the database was consuming about 77 TB of space. However when I  go inside the directory the total space consumed by all tables is about 5TB. So what is consuming the rest of the space or where did it go?   I'm finding space using the following command:   hadoop fs -du -h /user/hive/warehouse   My cloudera manager is 5.13 ... View more

I've cloudera manager managed 5.13 clusters. (Prod and DR). I had tested the Backup and Disaster Recovery (BDR) and it was working fine. Now I've kerberized both servers and installed the sentry service. I've a superuser configured which is in the supergroup with all the permissions (same user on both clusters). I can kinit and list as well.   Both my clusters are on the same realm, with KDC installed on two servers in master slave configuration.    I've created a new BDR schedule to replicate hive tables from prod to DR. When I dry run it, it fails on "Transfer Metadata Files" with the following error: Hdfs Copy File Command Failed because of java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Can't get Kerberos realm.        What can be the cause of this problem? ... View more

I'm trying to understand and secure my cloudera cluster managed by CM 5.13. I was going with the Local MIT KDC with Active Directory Integration. I have setup a local MIT KDC (server is not kerberized yet) and also a local openLdap server since I dont have a Windows server so using openLDAP instead which will provide the directory services and can be replaced with AD later on.     My question is that in order to establish a cross realm trust between MIT KDC and openldap, do I need to configure a KDC within openLDAP as well (since AD comes with a KDC)? How do I establish a trust relationship with the openLDAP? Any tutorial? ... View more