Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Server Error when attempting to setup Kerberos

avatar
Rising Star

I'm in the process of trying to enable Kerberos on the following version of HDP 2.3 (HDP-2.3.4.0-3485). I have the following components selected/installed:

  • HDFS
  • MapReduce2
  • YARN
  • Tez
  • Hive
  • HBase
  • Pig
  • ZooKeeper
  • Ambari Metrics
  • Kerberos

I encountered a error message similar to this one when trying to enable kerberos.

2267-unknown.png

NOTE: This dialog comes up when I attempt to regenerate my Kerberos keys. I also see the following exception in the ambari-server.log file:

19 Feb 2016 11:45:15,118  INFO [qtp-client-3081] AmbariManagementControllerImpl:1324 - Received a updateCluster request, clusterId=2, clusterName=dev09_ost_hivetest_h, securityType=KERBEROS, request={ clusterName=dev09_ost_hivetest_h, clusterId=2, provisioningState=null, securityType=KERBEROS, stackVersion=HDP-2.3, desired_scv=null, hosts=[] }
19 Feb 2016 11:45:15,157  WARN [qtp-client-3081] ServletHandler:563 - /api/v1/clusters/dev09_ost_hivetest_h
java.lang.NullPointerException
 at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.persistActions(ActionDBAccessorImpl.java:300)
 at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)
 at org.apache.ambari.server.actionmanager.ActionManager.sendActions(ActionManager.java:99)
 at org.apache.ambari.server.controller.internal.RequestStageContainer.persist(RequestStageContainer.java:216)
 at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1567)
 at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1308)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:241)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:238)
 at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:330)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider.updateResources(ClusterResourceProvider.java:238)
 at org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:310)
 at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:104)
 at org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:42)
 at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
 at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
 at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:105)
 at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:74)
 at org.apache.ambari.server.api.services.ClusterService.updateCluster(ClusterService.java:151)
 at sun.reflect.GeneratedMethodAccessor192.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
 at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
 at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
 at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:182)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
 at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
 at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
 at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
 at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:209)
 at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:198)
 at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:132)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
 at org.eclipse.jetty.server.Server.handle(Server.java:370)
 at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
 at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
 at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
 at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
 at java.lang.Thread.run(Thread.java:745)

Searching for this issue I've come across others who are encountering this exact issue: https://mail-archives.apache.org/mod_mbox/ambari-user/201602.mbox/%3C56B0CE2C.9050902@roo.ee%3E. No resolution that I've seen as of yet though.

1 ACCEPTED SOLUTION

avatar
Rising Star

I was able to work around my issue by explicitly setting my hostname in /etc/hosts in addition to hostnamectl. I think when Ambari constructs the Kerberos principals it is using the hostname that would resolve for the IP address that's assigned to my box.

Using the output from hostname -A lead me to a solution in addition to this snippet in Ambari Agent's log file:

java.io.IOException: Login failure for dn/host-192-168-114-49.td.local@<REDACTED KERBEROS REALM> from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Notice the hostname is thought to be host-192-168-114-49.td.local however in hostnamectl it's set to dev09-ost-hivetest-h-hb02.td.local. These being out of sync was ultimately my issue.

I created this Jira in the Ambari project about this as well: https://issues.apache.org/jira/browse/AMBARI-15165

View solution in original post

7 REPLIES 7

avatar

This is most likely a hostname issue. Check to make sure that the host thinks it is the correct name when running

hostname -f

It is possible that the ambari-agent running on the ambari-server host is using a different hostname than the one reported by hostname -f. This includes differences in character case. Ideally all hostnames are in lowercase.

avatar
Rising Star

The output of `hostname -f` and hostnamectl match so I don't think this is the issue.

[root@dev09-ost-hivetest-h-hb02 ~]# hostname -f
dev09-ost-hivetest-h-hb02.td.local

[root@dev09-ost-hivetest-h-hb02 ~]# hostnamectl
   Static hostname: dev09-ost-hivetest-h-hb02.td.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 61aaddd051a8fb40b29e47fd1b6c7084
           Boot ID: af96bd95fae147b8abb044cc7a95f78d
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.10.1.el7.x86_64
      Architecture: x86-64

[root@dev09-ost-hivetest-h-hb02 ~]# hostname
dev09-ost-hivetest-h-hb02.td.local

avatar

@Sam Mingolelli, what version of Ambari is this?

avatar
Rising Star

@Robert Levas - Version2.1.2.1

avatar
@Sam Mingolelli

Is there an Ambari agent on the same host as the Ambari server? If not, try to add the Ambari server host to the cluster using Ambari's add host facility. Also, check to make sure the hostname returned by hostname -f exists in the list of hosts presented on the host screen in the Ambari UI.

avatar
Rising Star
@Robert Levas

Yes there's a ambari server & agent on the same host. When doing the installation the hostname was found w/o issue when you do the search.

avatar
Rising Star

I was able to work around my issue by explicitly setting my hostname in /etc/hosts in addition to hostnamectl. I think when Ambari constructs the Kerberos principals it is using the hostname that would resolve for the IP address that's assigned to my box.

Using the output from hostname -A lead me to a solution in addition to this snippet in Ambari Agent's log file:

java.io.IOException: Login failure for dn/host-192-168-114-49.td.local@<REDACTED KERBEROS REALM> from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Notice the hostname is thought to be host-192-168-114-49.td.local however in hostnamectl it's set to dev09-ost-hivetest-h-hb02.td.local. These being out of sync was ultimately my issue.

I created this Jira in the Ambari project about this as well: https://issues.apache.org/jira/browse/AMBARI-15165