Created 08-06-2018 05:22 PM
I have 3 datanodes and ip address and hostnames of these datanode are changed. I followed steps given in https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.18/bk_ambari-reference/content/ch_changing_ho... but getting update-host-name failed.
Created 08-07-2018 11:00 AM
Hi @Shobhna Dhami ,
thanks for attaching the trace.
From Trace, it seems like you have a mytest.example.com that already exists in database.
I would suggest you raise a case with hortonworks support if you are a customer so that our engineers can fix this issue for you.
Or else.
Follow this steps .
1) take a DB backup ( so that you can restore incase you got something wrong in database)
pg_dump -U ambari ambari > ./ambari_$(date +"%Y%m%d%H%M%S").sql
2) stop Ambari-server
3) login to Postgres ( default password will be bigdata)
[root@aanaik1 ~]# psql -U ambari ambari Password for user ambari: psql (9.2.23) Type "help" for help. ambari=> select host_id,host_name,ipv4,public_host_name from ambari.hosts; ambari=> SELECT * FROM ambari.hoststate; <br>
you might find some duplicate entries for the host_name mytest.example.com.
you need to remove existing hosts after verifying that they do not have any services installed and not in use.
you can run the command :
ambari=> select * from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); ambari=> delete from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); ambari=> select * from hosts where host_name like '%mytest.example.com%' ; ambari=> delete from hosts where host_name like '%mytest.example.com%' ;
and then try to run the host update command again.
This commands are as per my understanding on the issue. you might need to investigate based on contents of ambari.hosts and ambari.hoststate table as you are more aware then me about your cluster configuration.
I would suggest if it's your production cluster then you can actually contact hortonworks support for this operation here: https://hortonworks.com/services/support/
hope this helps.
If this helps.
Please accept the answer so that it will be helpfull for other users facing same issue. .
Created 08-06-2018 06:17 PM
1. Make sure you replace cluster1 with the actual cluster name.
2. Format should be "old_host_name" : "new_host_name".
If still fails then check ambari-server log for any clues - if not abel to resolve then post the error message from the logs here.
Created 08-07-2018 04:52 AM
Hi @amarnath reddy pappu thanks for replying .
error message from server logs:
ERROR [ambari-client-thread-377621] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.hive2.utils.ServiceFormattedException at org.apache.ambari.view.hive2.resources.jobs.ProgressRetriever.jobCheck(ProgressRetriever.java:55) at org.apache.ambari.view.hive2.resources.jobs.ProgressRetriever.getProgress(ProgressRetriever.java:42) at org.apache.ambari.view.hive2.resources.jobs.JobService.getProgress(JobService.java:418) at sun.reflect.GeneratedMethodAccessor2305.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:257) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.jwt.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:96) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) at org.apache.ambari.server.security.authentication.AmbariAuthenticationFilter.doFilter(AmbariAuthenticationFilter.java:88) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at
Created 08-07-2018 05:44 AM
Hi @Shobhna Dhami,
Can you post the last few lines of the exception in server logs which will actually specify the reason in code format like below.
I am code format
also the below exception seems to be some exception related to hive views and i dont think its related to changing hostnames.
Please see the log timestamp and post the logs related to correct time when the issue have occured.
you can even try to run the command again and post the logs of current timestamp.
Created 08-07-2018 05:24 AM
Hi @Shobhna Dhami ,
you might need to stop ambari-agent in all hosts , stop ambari-server
Update host name as per the doc ( i see you are reffering to ambari-2.2.2.18 doc) refer to the version you are using.
then start ambari-server and ambari-agent .
Then try to stop and start datanodes.
If the hostname ip is changed then ambari-agent may be in lost heartbeat state
You can mannually try to stop datanode if you need : refer to : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_reference/content/stopping_hdp_services....
this URL refers to HDP-2.6.5 please refer to your corresponding HDP.
additionally if your command is failing please analyse ambari-server logs at
/var/log/ambari-server/ambari-server.log and search the Error trace here in community. you will find lot of similar article.
if not please attach those exceptions here with 'code' format ( easy to read)
will be happy to help you.
Created 08-07-2018 09:47 AM
07 Aug 2018 15:04:49,587 ERROR [main] HostUpdateHelper:562 - Unexpected error, host names update failed javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name" Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists. Error Code: 0 Call: UPDATE hosts SET host_name = ? WHERE (host_id = ?) bind => [2 parameters bound] at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153) at org.apache.ambari.server.update.HostUpdateHelper.updateHostsInDB(HostUpdateHelper.java:405) at org.apache.ambari.server.update.HostUpdateHelper.main(HostUpdateHelper.java:546) Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name" Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists. Error Code: 0 Call: UPDATE hosts SET host_name = ? WHERE (host_id = ?) bind => [2 parameters bound] at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633) at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149) at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845) at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614) at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169) at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134) ... 3 more Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name" Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892) ... 15 more 07 Aug 2018 15:04:49,815 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.4 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 07 Aug 2018 15:04:50,093 INFO [Stack Version Loading Thread] LatestRepoCallable:219 - Stack HDP-2.4 cannot resolve OS debian6 to the supported ones: suse11,redhat7,debian7,redhat6,ubuntu14,ubuntu12. Family: null 07 Aug 2018 15:04:50,450 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.5 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 07 Aug 2018 15:04:51,107 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.3.ECS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json 07 Aug 2018 15:04:52,507 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.1.GlusterFS from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json 07 Aug 2018 15:04:52,583 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.3.GlusterFS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json 07 Aug 2018 15:04:53,291 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.0 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
Created 08-07-2018 11:00 AM
Hi @Shobhna Dhami ,
thanks for attaching the trace.
From Trace, it seems like you have a mytest.example.com that already exists in database.
I would suggest you raise a case with hortonworks support if you are a customer so that our engineers can fix this issue for you.
Or else.
Follow this steps .
1) take a DB backup ( so that you can restore incase you got something wrong in database)
pg_dump -U ambari ambari > ./ambari_$(date +"%Y%m%d%H%M%S").sql
2) stop Ambari-server
3) login to Postgres ( default password will be bigdata)
[root@aanaik1 ~]# psql -U ambari ambari Password for user ambari: psql (9.2.23) Type "help" for help. ambari=> select host_id,host_name,ipv4,public_host_name from ambari.hosts; ambari=> SELECT * FROM ambari.hoststate; <br>
you might find some duplicate entries for the host_name mytest.example.com.
you need to remove existing hosts after verifying that they do not have any services installed and not in use.
you can run the command :
ambari=> select * from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); ambari=> delete from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); ambari=> select * from hosts where host_name like '%mytest.example.com%' ; ambari=> delete from hosts where host_name like '%mytest.example.com%' ;
and then try to run the host update command again.
This commands are as per my understanding on the issue. you might need to investigate based on contents of ambari.hosts and ambari.hoststate table as you are more aware then me about your cluster configuration.
I would suggest if it's your production cluster then you can actually contact hortonworks support for this operation here: https://hortonworks.com/services/support/
hope this helps.
If this helps.
Please accept the answer so that it will be helpfull for other users facing same issue. .
Created 08-07-2018 12:14 PM
Instead of removing it from database can I add new host with new hostname and remove host existing earlier.
Created 08-07-2018 12:39 PM
Hi @shobha,
yeah you can do that. but its always better to run the select command mentioned abouve to see whats wrong in your database
Created 08-07-2018 04:17 PM
As Akhil explained you seems to have new hostname already in the hosts tables. You can try following workaround.
1. Modify "slnxhadoop04.dhcp.noid.in.sopra" hostname to "slnxhadoop04.dhcp.noid.in.sopra.tmp" (temporary hostnmae) in your host update file ( repeat this if you have old hostnames mapped with new hostnames )
2. Now run "ambari-server update-host-names host_names_changes.json"
3. If 2nd step is successfull then go to step4 other wise identify problematic host and repeat step1 and step2.
4. Now create a new host update file mapping like below
"slnxhadoop04.dhcp.noid.in.sopra.tmp" "slnxhadoop04.dhcp.noid.in.sopra"
and run the "ambari-server update-host-names host_names_changes.json" command.
This should resolve your issue.