Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Server Error when attempting to setup Kerberos

Solved Go to solution

Server Error when attempting to setup Kerberos

Contributor

I'm in the process of trying to enable Kerberos on the following version of HDP 2.3 (HDP-2.3.4.0-3485). I have the following components selected/installed:

  • HDFS
  • MapReduce2
  • YARN
  • Tez
  • Hive
  • HBase
  • Pig
  • ZooKeeper
  • Ambari Metrics
  • Kerberos

I encountered a error message similar to this one when trying to enable kerberos.

2267-unknown.png

NOTE: This dialog comes up when I attempt to regenerate my Kerberos keys. I also see the following exception in the ambari-server.log file:

19 Feb 2016 11:45:15,118  INFO [qtp-client-3081] AmbariManagementControllerImpl:1324 - Received a updateCluster request, clusterId=2, clusterName=dev09_ost_hivetest_h, securityType=KERBEROS, request={ clusterName=dev09_ost_hivetest_h, clusterId=2, provisioningState=null, securityType=KERBEROS, stackVersion=HDP-2.3, desired_scv=null, hosts=[] }
19 Feb 2016 11:45:15,157  WARN [qtp-client-3081] ServletHandler:563 - /api/v1/clusters/dev09_ost_hivetest_h
java.lang.NullPointerException
 at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.persistActions(ActionDBAccessorImpl.java:300)
 at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)
 at org.apache.ambari.server.actionmanager.ActionManager.sendActions(ActionManager.java:99)
 at org.apache.ambari.server.controller.internal.RequestStageContainer.persist(RequestStageContainer.java:216)
 at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1567)
 at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1308)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:241)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:238)
 at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:330)
 at org.apache.ambari.server.controller.internal.ClusterResourceProvider.updateResources(ClusterResourceProvider.java:238)
 at org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:310)
 at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:104)
 at org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:42)
 at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
 at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
 at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:105)
 at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:74)
 at org.apache.ambari.server.api.services.ClusterService.updateCluster(ClusterService.java:151)
 at sun.reflect.GeneratedMethodAccessor192.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
 at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
 at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
 at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
 at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
 at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
 at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:182)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
 at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
 at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
 at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
 at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:209)
 at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:198)
 at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:132)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
 at org.eclipse.jetty.server.Server.handle(Server.java:370)
 at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
 at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
 at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
 at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
 at java.lang.Thread.run(Thread.java:745)

Searching for this issue I've come across others who are encountering this exact issue: https://mail-archives.apache.org/mod_mbox/ambari-user/201602.mbox/%3C56B0CE2C.9050902@roo.ee%3E. No resolution that I've seen as of yet though.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Server Error when attempting to setup Kerberos

Contributor

I was able to work around my issue by explicitly setting my hostname in /etc/hosts in addition to hostnamectl. I think when Ambari constructs the Kerberos principals it is using the hostname that would resolve for the IP address that's assigned to my box.

Using the output from hostname -A lead me to a solution in addition to this snippet in Ambari Agent's log file:

java.io.IOException: Login failure for dn/host-192-168-114-49.td.local@<REDACTED KERBEROS REALM> from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Notice the hostname is thought to be host-192-168-114-49.td.local however in hostnamectl it's set to dev09-ost-hivetest-h-hb02.td.local. These being out of sync was ultimately my issue.

I created this Jira in the Ambari project about this as well: https://issues.apache.org/jira/browse/AMBARI-15165

7 REPLIES 7

Re: Server Error when attempting to setup Kerberos

This is most likely a hostname issue. Check to make sure that the host thinks it is the correct name when running

hostname -f

It is possible that the ambari-agent running on the ambari-server host is using a different hostname than the one reported by hostname -f. This includes differences in character case. Ideally all hostnames are in lowercase.

Re: Server Error when attempting to setup Kerberos

Contributor

The output of `hostname -f` and hostnamectl match so I don't think this is the issue.

[root@dev09-ost-hivetest-h-hb02 ~]# hostname -f
dev09-ost-hivetest-h-hb02.td.local

[root@dev09-ost-hivetest-h-hb02 ~]# hostnamectl
   Static hostname: dev09-ost-hivetest-h-hb02.td.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 61aaddd051a8fb40b29e47fd1b6c7084
           Boot ID: af96bd95fae147b8abb044cc7a95f78d
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.10.1.el7.x86_64
      Architecture: x86-64

[root@dev09-ost-hivetest-h-hb02 ~]# hostname
dev09-ost-hivetest-h-hb02.td.local

Re: Server Error when attempting to setup Kerberos

@Sam Mingolelli, what version of Ambari is this?

Re: Server Error when attempting to setup Kerberos

Contributor

@Robert Levas - Version2.1.2.1

Re: Server Error when attempting to setup Kerberos

@Sam Mingolelli

Is there an Ambari agent on the same host as the Ambari server? If not, try to add the Ambari server host to the cluster using Ambari's add host facility. Also, check to make sure the hostname returned by hostname -f exists in the list of hosts presented on the host screen in the Ambari UI.

Re: Server Error when attempting to setup Kerberos

Contributor
@Robert Levas

Yes there's a ambari server & agent on the same host. When doing the installation the hostname was found w/o issue when you do the search.

Highlighted

Re: Server Error when attempting to setup Kerberos

Contributor

I was able to work around my issue by explicitly setting my hostname in /etc/hosts in addition to hostnamectl. I think when Ambari constructs the Kerberos principals it is using the hostname that would resolve for the IP address that's assigned to my box.

Using the output from hostname -A lead me to a solution in addition to this snippet in Ambari Agent's log file:

java.io.IOException: Login failure for dn/host-192-168-114-49.td.local@<REDACTED KERBEROS REALM> from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Notice the hostname is thought to be host-192-168-114-49.td.local however in hostnamectl it's set to dev09-ost-hivetest-h-hb02.td.local. These being out of sync was ultimately my issue.

I created this Jira in the Ambari project about this as well: https://issues.apache.org/jira/browse/AMBARI-15165