Support Questions

Find answers, ask questions, and share your expertise

How can I change hostname of datanodes in ambari?

avatar
Explorer

I have 3 datanodes and ip address and hostnames of these datanode are changed. I followed steps given in https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.18/bk_ambari-reference/content/ch_changing_ho... but getting update-host-name failed.

1 ACCEPTED SOLUTION

avatar

Hi @Shobhna Dhami ,

thanks for attaching the trace.

From Trace, it seems like you have a mytest.example.com that already exists in database.

I would suggest you raise a case with hortonworks support if you are a customer so that our engineers can fix this issue for you.

Or else.

Follow this steps .

1) take a DB backup ( so that you can restore incase you got something wrong in database)

pg_dump -U ambari ambari > ./ambari_$(date +"%Y%m%d%H%M%S").sql

2) stop Ambari-server

3) login to Postgres ( default password will be bigdata)

[root@aanaik1 ~]# psql -U ambari ambari
Password for user ambari:
psql (9.2.23)
Type "help" for help.
ambari=> select host_id,host_name,ipv4,public_host_name from ambari.hosts;
ambari=> SELECT * FROM ambari.hoststate;
<br>

you might find some duplicate entries for the host_name mytest.example.com.

you need to remove existing hosts after verifying that they do not have any services installed and not in use.

you can run the command :

ambari=> select * from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); 
ambari=> delete from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); 
ambari=> select * from hosts where host_name like '%mytest.example.com%' ;
ambari=> delete from hosts where host_name like '%mytest.example.com%' ;

and then try to run the host update command again.

This commands are as per my understanding on the issue. you might need to investigate based on contents of ambari.hosts and ambari.hoststate table as you are more aware then me about your cluster configuration.

I would suggest if it's your production cluster then you can actually contact hortonworks support for this operation here: https://hortonworks.com/services/support/

hope this helps.

If this helps.

Please accept the answer so that it will be helpfull for other users facing same issue. .

View solution in original post

9 REPLIES 9

avatar

@Shobhna Dhami

1. Make sure you replace cluster1 with the actual cluster name.

2. Format should be "old_host_name" : "new_host_name".

If still fails then check ambari-server log for any clues - if not abel to resolve then post the error message from the logs here.

avatar
Explorer

Hi @amarnath reddy pappu thanks for replying .

error message from server logs:

ERROR [ambari-client-thread-377621] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.hive2.utils.ServiceFormattedException at org.apache.ambari.view.hive2.resources.jobs.ProgressRetriever.jobCheck(ProgressRetriever.java:55) at org.apache.ambari.view.hive2.resources.jobs.ProgressRetriever.getProgress(ProgressRetriever.java:42) at org.apache.ambari.view.hive2.resources.jobs.JobService.getProgress(JobService.java:418) at sun.reflect.GeneratedMethodAccessor2305.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:257) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.jwt.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:96) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) at org.apache.ambari.server.security.authentication.AmbariAuthenticationFilter.doFilter(AmbariAuthenticationFilter.java:88) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at

avatar

Hi @Shobhna Dhami,

Can you post the last few lines of the exception in server logs which will actually specify the reason in code format like below.

I am code format

also the below exception seems to be some exception related to hive views and i dont think its related to changing hostnames.

Please see the log timestamp and post the logs related to correct time when the issue have occured.

you can even try to run the command again and post the logs of current timestamp.

avatar

Hi @Shobhna Dhami ,

you might need to stop ambari-agent in all hosts , stop ambari-server

Update host name as per the doc ( i see you are reffering to ambari-2.2.2.18 doc) refer to the version you are using.

then start ambari-server and ambari-agent .

Then try to stop and start datanodes.

If the hostname ip is changed then ambari-agent may be in lost heartbeat state

You can mannually try to stop datanode if you need : refer to : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_reference/content/stopping_hdp_services....

this URL refers to HDP-2.6.5 please refer to your corresponding HDP.

additionally if your command is failing please analyse ambari-server logs at

/var/log/ambari-server/ambari-server.log and search the Error trace here in community. you will find lot of similar article.

if not please attach those exceptions here with 'code' format ( easy to read)

will be happy to help you.

avatar
Explorer
07 Aug 2018 15:04:49,587 ERROR [main] HostUpdateHelper:562 - Unexpected error, host names update failed
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name"
  Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists.
Error Code: 0
Call: UPDATE hosts SET host_name = ? WHERE (host_id = ?)
bind => [2 parameters bound]
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at org.apache.ambari.server.update.HostUpdateHelper.updateHostsInDB(HostUpdateHelper.java:405)
at org.apache.ambari.server.update.HostUpdateHelper.main(HostUpdateHelper.java:546)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name"
  Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists.
Error Code: 0
Call: UPDATE hosts SET host_name = ? WHERE (host_id = ?)
bind => [2 parameters bound]
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
... 3 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name"
  Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
... 15 more
07 Aug 2018 15:04:49,815  INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.4 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:50,093  INFO [Stack Version Loading Thread] LatestRepoCallable:219 - Stack HDP-2.4 cannot resolve OS debian6 to the supported ones: suse11,redhat7,debian7,redhat6,ubuntu14,ubuntu12. Family: null
07 Aug 2018 15:04:50,450  INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.5 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:51,107  INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.3.ECS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:52,507  INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.1.GlusterFS from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:52,583  INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.3.GlusterFS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:53,291  INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.0 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json

avatar

Hi @Shobhna Dhami ,

thanks for attaching the trace.

From Trace, it seems like you have a mytest.example.com that already exists in database.

I would suggest you raise a case with hortonworks support if you are a customer so that our engineers can fix this issue for you.

Or else.

Follow this steps .

1) take a DB backup ( so that you can restore incase you got something wrong in database)

pg_dump -U ambari ambari > ./ambari_$(date +"%Y%m%d%H%M%S").sql

2) stop Ambari-server

3) login to Postgres ( default password will be bigdata)

[root@aanaik1 ~]# psql -U ambari ambari
Password for user ambari:
psql (9.2.23)
Type "help" for help.
ambari=> select host_id,host_name,ipv4,public_host_name from ambari.hosts;
ambari=> SELECT * FROM ambari.hoststate;
<br>

you might find some duplicate entries for the host_name mytest.example.com.

you need to remove existing hosts after verifying that they do not have any services installed and not in use.

you can run the command :

ambari=> select * from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); 
ambari=> delete from hoststate where host_id in (select host_id from hosts where host_name like '%mytest.example.com%'); 
ambari=> select * from hosts where host_name like '%mytest.example.com%' ;
ambari=> delete from hosts where host_name like '%mytest.example.com%' ;

and then try to run the host update command again.

This commands are as per my understanding on the issue. you might need to investigate based on contents of ambari.hosts and ambari.hoststate table as you are more aware then me about your cluster configuration.

I would suggest if it's your production cluster then you can actually contact hortonworks support for this operation here: https://hortonworks.com/services/support/

hope this helps.

If this helps.

Please accept the answer so that it will be helpfull for other users facing same issue. .

avatar
Explorer

Hi @Akhil S Naik

Instead of removing it from database can I add new host with new hostname and remove host existing earlier.

avatar

Hi @shobha,

yeah you can do that. but its always better to run the select command mentioned abouve to see whats wrong in your database

avatar

@Shobhna Dhami

As Akhil explained you seems to have new hostname already in the hosts tables. You can try following workaround.

1. Modify "slnxhadoop04.dhcp.noid.in.sopra" hostname to "slnxhadoop04.dhcp.noid.in.sopra.tmp" (temporary hostnmae) in your host update file ( repeat this if you have old hostnames mapped with new hostnames )

2. Now run "ambari-server update-host-names host_names_changes.json"

3. If 2nd step is successfull then go to step4 other wise identify problematic host and repeat step1 and step2.

4. Now create a new host update file mapping like below

"slnxhadoop04.dhcp.noid.in.sopra.tmp" "slnxhadoop04.dhcp.noid.in.sopra"

and run the "ambari-server update-host-names host_names_changes.json" command.

This should resolve your issue.