Registering Host on the ambari 1.7 fails - host entry doesn't exist in the database

[#1]

Hi,

I'm trying to install HDP using ambari cloud provisioning tool. The stack that i'm trying to setup is as follows (single node hadoop setup)

HDP2.2 Ambari 1.7 OpenSuse 11 SP3

As given in the guide, http://docs.hortonworks.com/HDPDocuments/Ambari-2...., i've set up the environment as required. Have turned off the firewall and have added the hosts as required. Aslo setup password less SSH to the localhost. The ambari server setup and start went through successfully.

When I go through cluster setup wizard, after selecting HDP 2.2 and then giving the host name, and the SSH key, I get the following error from the application.

ERROR: Host "xxx" doesn't exist in the database

AmbariExecption caught when trying to create a resource..

Any Help here would be greatly appreciated...

Entire Dump:

INFO:root:BootStrapping hosts
['XXX'] using
/usr/lib/python2.6/site-packages/ambari_server cluster primary OS: suse11 with
user 'root' sshKey File /var/run/ambari-server/bootstrap/1/sshKey password File
null using tmp dir /var/run/ambari-server/bootstrap/1 ambari:
XXX; server_port: 8080; ambari version: 1.7.0
INFO:root:Executing parallel bootstrap
ERROR:root:ERROR: Bootstrap of host XXX fails
because previous action finished with non-zero exit code (1)
ERROR MESSAGE: Execute of '<bound method Bootstrap.copyNeededFiles of
<Bootstrap(Thread-1, started daemon 140472791299840)>>' failed
STDOUT: Try to execute '<bound method Bootstrap.copyNeededFiles of
<Bootstrap(Thread-1, started daemon 140472791299840)>>'
INFO:root:Finished parallel bootstrap
20:19:59,393  INFO [pool-1-thread-1] BSHostStatusCollector:55 - Request
directory /var/run/ambari-server/bootstrap/1
20:19:59,393  INFO [pool-1-thread-1] BSHostStatusCollector:62 - HostList
for polling on [XXX]
20:20:01,759 ERROR [qtp367299178-20] ActionDBAccessorImpl:241 - Host
XXX doesn't exist in database
20:20:01,814 ERROR [qtp367299178-20] AbstractResourceProvider:280 - Caught
AmbariException when creating a resource
org.apache.ambari.server.AmbariException: Host
XXX doesn't exist in database
  at
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.persistActions(ActionDBAccessorImpl.java:242)
  at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)
  at
org.apache.ambari.server.actionmanager.ActionManager.sendActions(ActionManager.java:96)
  at org.apache.ambari.server.actionmanager.ActionManager.sendActions(ActionManager.java:85)
  at
org.apache.ambari.server.controller.AmbariManagementControllerImpl.createAction(AmbariManagementControllerImpl.java:3071)
  at
org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:121)
  at
org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:118)
  at
org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
  at
org.apache.ambari.server.controller.internal.RequestResourceProvider.createResources(RequestResourceProvider.java:118)
  at
org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:274)
  at
org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:75)
  at
org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
  at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
  at
org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
  at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:103)
  at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:72)
  at
org.apache.ambari.server.api.services.RequestService.createRequests(RequestService.java:137)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
  at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
  at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
  at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
  at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
  at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
  at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
  at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
  at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
  at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
  at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
  at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
  at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
  at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
  at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
  at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
  at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
  at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
  at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
  at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
  at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
  at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
  at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:166)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
  at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
  at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
  at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
  at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
  at
org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
  at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
  at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
  at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
  at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
  at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
  at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
  at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
  at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)
  at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)
  at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
  at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)
  at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
  at
org.apache.ambari.server.controller.FailsafeHandlerList.handle(FailsafeHandlerList.java:132)
  at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
  at org.eclipse.jetty.server.Server.handle(Server.java:363)
  at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)
  at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:931)
  at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:992)
  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856)
  at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
  at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
  at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
  at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
  at java.lang.Thread.run(Thread.java:745)
20:20:01,819 ERROR [qtp367299178-20] BaseManagementHandler:57 - Caught a system
exception while attempting to create a resource

Thanks!

[#2]

@Jamal J Please see this error Host inblrllssc281.apj.global.corp.sap doesn't exist in database

1) It's always good idea not to provide complete host entries while submitting AH post.

2) Please verify the hostnames and make sure that they can talk to each other.

3) if it's new install ( after couple of failed attempts to install ) then you can run ambari-server reset to reset the database and run the install. Please do your research before running reset

[#3 · Accepted solution]

It looks like Ambari server is on inblrllssc.apj.global.sap.corp and you are trying to register inblrllssc281.apj.global.corp.sap

Make sure you can "ssh inblrllssc281.apj.global.corp.sap" as root. Then retry, or do "ambari-server reset" and retry [there was a bug related to this in Ambari-1.7: https://issues.apache.org/jira/browse/AMBARI-8513 ]

By the way, is there a special reason you are still using Ambari-1.7? If not, I recommend you to try the latest version of Ambari-2.1.2.1. Many old bugs are fixed and it also has many new features.

[#4]

Hi Minovic,

Used Ambari 1.7, following a documentation.. Moved to Ambari 2.1.2.1, and the issue got resolved..

Thanks for your suggestion here!