Server Error when attempting to setup Kerberos

[#1]

I'm in the process of trying to enable Kerberos on the following version of HDP 2.3 (HDP-2.3.4.0-3485). I have the following components selected/installed:

• HDFS
• MapReduce2
• YARN
• Tez
• Hive
• HBase
• Pig
• ZooKeeper
• Ambari Metrics
• Kerberos

I encountered a error message similar to this one when trying to enable kerberos.

NOTE: This dialog comes up when I attempt to regenerate my Kerberos keys. I also see the following exception in the ambari-server.log file:

19 Feb 2016 11:45:15,118  INFO [qtp-client-3081] AmbariManagementControllerImpl:1324 - Received a updateCluster request, clusterId=2, clusterName=dev09_ost_hivetest_h, securityType=KERBEROS, request={ clusterName=dev09_ost_hivetest_h, clusterId=2, provisioningState=null, securityType=KERBEROS, stackVersion=HDP-2.3, desired_scv=null, hosts=[] }
19 Feb 2016 11:45:15,157  WARN [qtp-client-3081] ServletHandler:563 - /api/v1/clusters/dev09_ost_hivetest_h
java.lang.NullPointerException
 at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.persistActions(ActionDBAccessorImpl.java:300)
 at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)
 at org.apache.ambari.server.actionmanager.ActionManager.sendActions(ActionManager.java:99)
 at org.apache.ambari.server.controller.internal.RequestStageContainer.persist(RequestStageContainer.java:216)
 at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1567)
 at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1308)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:241)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:238)
 at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:330)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider.updateResources(ClusterResourceProvider.java:238)
 at org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:310)
 at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:104)
 at org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:42)
 at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
 at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
 at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:105)
 at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:74)
 at org.apache.ambari.server.api.services.ClusterService.updateCluster(ClusterService.java:151)
 at sun.reflect.GeneratedMethodAccessor192.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
 at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
 at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
 at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:182)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
 at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
 at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
 at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
 at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:209)
 at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:198)
 at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:132)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
 at org.eclipse.jetty.server.Server.handle(Server.java:370)
 at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
 at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
 at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
 at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
 at java.lang.Thread.run(Thread.java:745)

Searching for this issue I've come across others who are encountering this exact issue: https://mail-archives.apache.org/mod_mbox/ambari-user/201602.mbox/%3C56B0CE2C.9050902@roo.ee%3E. No resolution that I've seen as of yet though.

[#2]

This is most likely a hostname issue. Check to make sure that the host thinks it is the correct name when running

hostname -f

It is possible that the ambari-agent running on the ambari-server host is using a different hostname than the one reported by hostname -f. This includes differences in character case. Ideally all hostnames are in lowercase.

[#3]

The output of `hostname -f` and hostnamectl match so I don't think this is the issue.

[root@dev09-ost-hivetest-h-hb02 ~]# hostname -f
dev09-ost-hivetest-h-hb02.td.local

[root@dev09-ost-hivetest-h-hb02 ~]# hostnamectl
   Static hostname: dev09-ost-hivetest-h-hb02.td.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 61aaddd051a8fb40b29e47fd1b6c7084
           Boot ID: af96bd95fae147b8abb044cc7a95f78d
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.10.1.el7.x86_64
      Architecture: x86-64

[root@dev09-ost-hivetest-h-hb02 ~]# hostname
dev09-ost-hivetest-h-hb02.td.local

[#4]

@Sam Mingolelli, what version of Ambari is this?

[#5]

@Robert Levas - Version2.1.2.1

[#6]

@Sam Mingolelli

Is there an Ambari agent on the same host as the Ambari server? If not, try to add the Ambari server host to the cluster using Ambari's add host facility. Also, check to make sure the hostname returned by hostname -f exists in the list of hosts presented on the host screen in the Ambari UI.

[#7]

@Robert Levas

Yes there's a ambari server & agent on the same host. When doing the installation the hostname was found w/o issue when you do the search.

[#8 · Accepted solution]

I was able to work around my issue by explicitly setting my hostname in /etc/hosts in addition to hostnamectl. I think when Ambari constructs the Kerberos principals it is using the hostname that would resolve for the IP address that's assigned to my box.

Using the output from hostname -A lead me to a solution in addition to this snippet in Ambari Agent's log file:

java.io.IOException: Login failure for dn/host-192-168-114-49.td.local@<REDACTED KERBEROS REALM> from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Notice the hostname is thought to be host-192-168-114-49.td.local however in hostnamectl it's set to dev09-ost-hivetest-h-hb02.td.local. These being out of sync was ultimately my issue.

I created this Jira in the Ambari project about this as well: https://issues.apache.org/jira/browse/AMBARI-15165