Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

confirm hosts faield! I have spend lot of time but could not deal with it

Highlighted

confirm hosts faield! I have spend lot of time but could not deal with it

New Contributor

i am reinstalling the ambari-server on ubuntu14 and meet some trouble.i have checked the configure on my host (I am going to install the ambari-server and agent on the same host) for several time.

/etc/hosts:

zjt@amax:~$ cat /etc/hosts
127.0.0.1  localhost localhost.localdomain
192.168.5.162   master.com  amax

SSH :ssh localhost successfully

zjt@amax:~$ ssh localhost
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-62-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

269 packages can be updated.
217 updates are security updates.

New release '16.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Your Hardware Enablement Stack (HWE) is supported until April 2019.
Last login: Sat Sep  9 19:20:18 2017 from 192.168.5.162

I am sure that the ufw is inactive and i have opend the ntp.The user zjt can login without psw.

The error message as follow:

==========================
Creating target directory...
==========================

Command start time 2017-09-09 19:20:16
chmod: could not access"/var/lib/ambari-agent/data": no such file or directory.

Connection to master.com closed.
SSH command execution finished
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:16

==========================
Copying common functions script...
==========================

Command start time 2017-09-09 19:20:16

scp /usr/lib/python2.6/site-packages/ambari_commons
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:17

==========================
Copying OS type check script...
==========================

Command start time 2017-09-09 19:20:17

scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:17

==========================
Running OS type check...
==========================

Command start time 2017-09-09 19:20:17
Cluster primary/cluster OS family is ubuntu14 and local/current OS family is ubuntu14

Connection to master.com closed.
SSH command execution finished
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:17

==========================
Checking 'sudo' package on remote host...
==========================

Command start time 2017-09-09 19:20:17
sudo                        install

Connection to master.com closed.
SSH command execution finished
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:17

==========================
Copying repo file to 'tmp' folder...
==========================

Command start time 2017-09-09 19:20:17

scp /etc/apt/sources.list.d/ambari.list
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:17

==========================
Moving file to repo dir...
==========================

Command start time 2017-09-09 19:20:17

Connection to master.com closed.
SSH command execution finished
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:18

==========================
Changing permissions for ambari.repo...
==========================

Command start time 2017-09-09 19:20:18

Connection to master.com closed.
SSH command execution finished
host=master.com, exitcode=0
Command end time 2017-09-09 19:20:18

==========================
Update apt cache of repository...
==========================

Command start time 2017-09-09 19:20:18
Automatic Agent registration timed out (timeout = 300 seconds). Check your network connectivity and retry registration, or use manual agent registration.

and the ambari-server log

09 Sept 2017 14:13:33,315  INFO [pool-13-thread-1] BSHostStatusCollector:55 - Request directory /var/run/ambari-server/bootstrap/7
09 Sept 2017 14:13:33,316  INFO [pool-13-thread-1] BSHostStatusCollector:62 - HostList for polling on [master.com]
09 Sept 2017 14:13:34,818 ERROR [qtp-client-1982] AbstractResourceProvider:279 - Caught AmbariException when creating a resource
org.apache.ambari.server.HostNotFoundException: Host not found, hostname=
    at org.apache.ambari.server.state.cluster.ClustersImpl.getHost(ClustersImpl.java:343)
    at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:108)
    at org.apache.ambari.server.controller.AmbariManagementControllerImpl.findConfigurationTagsWithOverrides(AmbariManagementControllerImpl.java:1820)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
    at com.sun.proxy.$Proxy83.findConfigurationTagsWithOverrides(Unknown Source)
    at org.apache.ambari.server.controller.AmbariActionExecutionHelper.addExecutionCommandsToStage(AmbariActionExecutionHelper.java:372)
    at org.apache.ambari.server.controller.AmbariManagementControllerImpl.createAction(AmbariManagementControllerImpl.java:3366)
    at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:165)
    at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:162)
    at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:272)
    at org.apache.ambari.server.controller.internal.RequestResourceProvider.createResources(RequestResourceProvider.java:162)
    at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
    at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
    at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
    at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
    at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:105)
    at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:74)
    at org.apache.ambari.server.api.services.RequestService.createRequests(RequestService.java:145)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:182)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
    at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
    at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
    at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
    at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:209)
    at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:198)
    at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:132)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:748)
09 Sept 2017 14:13:34,819 ERROR [qtp-client-1982] BaseManagementHandler:57 - Caught a system exception while attempting to create a resource

It's something didn't clear up completely before I reinstall the ambari-server again? I have installed the ambari-server and registered the agent successfully on this host with the same configure several weaks ago. If anyone know what's wrong with my host or you have met the same trouble please help me!

thank you very much!

3 REPLIES 3
Highlighted

Re: confirm hosts faield! I have spend lot of time but could not deal with it

Super Mentor

@zheng zt

Are you sure that the hostname is correctly set?

Please make sure that the following commands returns the correct hostname (FQDN). Now only for master host, All the hosts that are part of the cluster should have correct FQDN set and the following command should return the value properly.

# hostname -f

.

Also the /etc/hosts file entry inside every host (ambari agent machines) need to be set correctly.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-installation/content/edit_the_net...

.

Your error at the end is :

AutomaticAgent registration timed out(timeout =300 seconds).Check your network connectivity and retry registration,or use manual agent registration.

Which indicates a communication issue. Which might be due to N/W or Firewall as well. Please make sure that ambari agents can access the following ports (8080, 8440, 8441, 8670) of amabri server without any issue.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-administration/content/default_ne...

Hence on agent machine please try to run the following commands to verify it's hostname and its communication with amabri server:

# hostname -f
# nc  -v master.com  8440
# nc  -v master.com  8441

.

Re: confirm hosts faield! I have spend lot of time but could not deal with it

New Contributor

@Jay SenSharma

Thanks for your reply.

I want to edit /etc/sysconfig/network but no such file on ubuntu 14.Should I add on /etc/network/interfaces?

the commands as you refer above return:

zjt@amax:/etc/sysctl.d$ hostname -f
master.com
zjt@amax:~$ nc -v  master.com 8440
Connection to master.com 8440 port [tcp/*] succeeded!
zjt@amax:~$ nc -v  master.com 8441
Connection to master.com 8441 port [tcp/*] succeeded!
Highlighted

Re: confirm hosts faield! I have spend lot of time but could not deal with it

Mentor

@zheng zt

The below entry if from my centos7 it could be different on Ubuntu , please don't tamper with the first 2 entries they can affect they can cause networking problems.

My example below has 4 hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
###################################
# <IP> <HOSTNAME>.<DOMAIN> <ALIAS> 
####################################
xxx.xxx.x.xxx  master.com   master
xxx.xxx.x.xxx  master2.com   master2
xxx.xxx.x.xxx  datanode1.com   datanode1
xxx.xxx.x.xxx  datanode2.com   datanode2
xxx.xxx.x.xxx  datanode3.com   datanode3

The hostname in ambari-agent.ini should be the output of

# hostname -f

ambari-agent.ini

[server]
hostname=[xxxxxx]
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30

Hope that helps

Don't have an account?
Coming from Hortonworks? Activate your account here