Support Questions
Find answers, ask questions, and share your expertise

App Timeline server duplicated (sort of) after failed move

Solved Go to solution

Re: App Timeline server duplicated (sort of) after failed move

Explorer

Remember, you asked for it. :)

20 Jan 2017 15:27:44,017 ERROR [qtp-client-37739] AbstractResourceProvider:338 - Caught AmbariException when modifying a resource

org.apache.ambari.server.AmbariException: Host Component cannot be removed, clusterName=ROGERGPFS, serviceName=YARN, componentName=APP_TIMELINE_SERVER, hostname=cg-hm09.ncsa.illinois.edu, request={ clusterName=ROGERGPFS, serviceName=YARN, componentName=APP_TIMELINE_SERVER, hostname=cg-hm09.ncsa.illinois.edu, desiredState=null, state=null, desiredStackId=null, staleConfig=null, adminState=null}

at org.apache.ambari.server.controller.AmbariManagementControllerImpl.deleteHostComponents(AmbariManagementControllerImpl.java:2731)

at org.apache.ambari.server.controller.internal.HostComponentResourceProvider$3.invoke(HostComponentResourceProvider.java:321)

at org.apache.ambari.server.controller.internal.HostComponentResourceProvider$3.invoke(HostComponentResourceProvider.java:318)

at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:331)

at org.apache.ambari.server.controller.internal.HostComponentResourceProvider.deleteResources(HostComponentResourceProvider.java:318)

at org.apache.ambari.server.controller.internal.ClusterControllerImpl.deleteResources(ClusterControllerImpl.java:330)

at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.delete(PersistenceManagerImpl.java:111)

at org.apache.ambari.server.api.handlers.DeleteHandler.persist(DeleteHandler.java:44)

at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)

at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)

at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:105)

at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:74)

at org.apache.ambari.server.api.services.HostComponentService.deleteHostComponent(HostComponentService.java:203)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)

{continued}

Re: App Timeline server duplicated (sort of) after failed move

Contributor

Ok, the ATS instance that you are trying to delete is in one of the states that makes it non-deletable.

Can you get me the output of:

curl -uadmin:PW -k https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu/host_components/APP...

Re: App Timeline server duplicated (sort of) after failed move

Contributor

@Wayne Hoyenga

Ok, you just need to issue Stop on that ATS instance on cg-hm09. Please go to Hosts -> cg-hm09 and choose Stop on App Timeline Server from the component list. Then try the delete API call again.

Re: App Timeline server duplicated (sort of) after failed move

Contributor

@Wayne Hoyenga

Did the unwanted alerts disappear too?

Re: App Timeline server duplicated (sort of) after failed move

Explorer

Yea! That worked. Thank you.

Re: App Timeline server duplicated (sort of) after failed move

Explorer

{ "href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu/host_components/APP_TIMELINE_SERVER", "HostRoles" : { "cluster_name" : "ROGERGPFS", "component_name" : "APP_TIMELINE_SERVER", "desired_stack_id" : "HDP-2.3", "desired_state" : "STARTED", "hdp_version" : "HDP-2.3.2.0-2602", "host_name" : "cg-hm09.ncsa.illinois.edu", "maintenance_state" : "ON", "service_name" : "YARN", "stack_id" : "HDP-2.3", "stale_configs" : false, "state" : "STARTED", "upgrade_state" : "NONE", "actual_configs" : { "accumulo-env" : { "default" : "version1460667914523" }, "accumulo-log4j" : { "default" : "version1460667914523" }, "accumulo-site" : { "default" : "version1460667914523" }, "ams-env" : { "default" : "version1" }, "ams-hbase-env" : { "default" : "version1" }, "ams-hbase-log4j" : { "default" : "version1" }, "ams-hbase-policy" : { "default" : "version1" }, "ams-hbase-security-site" : { "default" : "version1" }, "ams-hbase-site" : { "default" : "version1467740762357" }, "ams-log4j" : { "default" : "version1" }, "ams-site" : { "default" : "version1467740762357" }, "capacity-scheduler" : { "default" : "version1" }, "client" : { "default" : "version1460667914523" }, "cluster-env" : { "default" : "version1" }, "core-site" : { "default" : "version1484279982036" }, "falcon-env" : { "default" : "version1" }, "falcon-runtime.properties" : { "default" : "version1" }, "falcon-startup.properties" : { "default" : "version1" }, "gateway-log4j" : { "default" : "version1" }, "gateway-site" : { "default" : "version1" }, "hadoop-env" : { "default" : "version1" }, "hadoop-policy" : { "default" : "version1" }, "hbase-env" : { "default" : "version1" }, "hbase-log4j" : { "default" : "version1" }, "hbase-policy" : { "default" : "version1" }, "hbase-site" : { "default" : "version1" }, "hcat-env" : { "default" : "version1" }, "hdfs-log4j" : { "default" : "version1" }, "hdfs-site" : { "default" : "version1484262327604" }, "hive-env" : { "default" : "version1484280941838" }, "hive-exec-log4j" : { "default" : "version1" }, "hive-log4j" : { "default" : "version1" }, "hive-site" : { "default" : "version1484280941838" }, "hiveserver2-site" : { "default" : "version1" }, "knox-env" : { "default" : "version1" }, "ldap-log4j" : { "default" : "version1" }, "mapred-env" : { "default" : "version1" }, "mapred-site" : { "default" : "version1" }, "oozie-env" : { "default" : "version1442008913821" }, "oozie-log4j" : { "default" : "version1" }, "oozie-site" : { "default" : "version1484271458416" }, "pig-env" : { "default" : "version1" }, "pig-log4j" : { "default" : "version1" }, "pig-properties" : { "default" : "version1" }, "ranger-hbase-audit" : { "default" : "version1" }, "ranger-hbase-plugin-properties" : { "default" : "version1" }, "ranger-hbase-policymgr-ssl" : { "default" : "version1" }, "ranger-hbase-security" : { "default" : "version1" }, "ranger-hdfs-audit" : { "default" : "version1" }, "ranger-hdfs-plugin-properties" : { "default" : "version1" }, "ranger-hdfs-policymgr-ssl" : { "default" : "version1" }, "ranger-hdfs-security" : { "default" : "version1" }, "ranger-hive-audit" : { "default" : "version1" }, "ranger-hive-plugin-properties" : { "default" : "version1" }, "ranger-hive-policymgr-ssl" : { "default" : "version1" }, "ranger-hive-security" : { "default" : "version1" }, "ranger-knox-audit" : { "default" : "version1" }, "ranger-knox-plugin-properties" : { "default" : "version1" }, "ranger-knox-policymgr-ssl" : { "default" : "version1" }, "ranger-knox-security" : { "default" : "version1" }, "ranger-yarn-audit" : { "default" : "version1" }, "ranger-yarn-plugin-properties" : { "default" : "version1" }, "ranger-yarn-policymgr-ssl" : { "default" : "version1" }, "ranger-yarn-security" : { "default" : "version1" }, "spark-defaults" : { "default" : "version1" }, "spark-env" : { "default" : "version1" }, "spark-javaopts-properties" : { "default" : "version1" }, "spark-log4j-properties" : { "default" : "version1" }, "spark-metrics-properties" : { "default" : "version1" }, "sqoop-env" : { "default" : "version1" }, "ssl-client" : { "default" : "version1" }, "ssl-server" : { "default" : "version1" }, "tez-env" : { "default" : "version1" }, "tez-site" : { "default" : "version1" }, "topology" : { "default" : "version1" }, "users-ldif" : { "default" : "version1" }, "webhcat-env" : { "default" : "version1" }, "webhcat-log4j" : { "default" : "version1" }, "webhcat-site" : { "default" : "version1484280941838" }, "yarn-env" : { "default" : "version1484269362684" }, "yarn-log4j" : { "default" : "version1" }, "yarn-site" : { "default" : "version1484278518158" }, "zoo.cfg" : { "default" : "version1" }, "zookeeper-env" : { "default" : "version1" }, "zookeeper-log4j" : { "default" : "version1" } } }, "host" : { "href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu" }, "component" : [ { "href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/services/YARN/components/APP_TIMELINE_SERVER", "ServiceComponentInfo" : { "cluster_name" : "ROGERGPFS", "component_name" : "APP_TIMELINE_SERVER", "service_name" : "YARN" } } ], "processes" : [ ] }

Re: App Timeline server duplicated (sort of) after failed move

Explorer

@yusaku, yes, they did.

At some point, when it happens again, I may start another thread regarding why my ambari metrics server keeps dying. Its fairly annoying.

Re: App Timeline server duplicated (sort of) after failed move

Contributor

Awesome. If you could "accept" my answer, that would be great. AMS crashing issue might be due to the version of Ambari you are using (2.1.0 is quite old and there have been numerous stability improvements on AMS since then). If it is possible, I highly recommend you upgrade to Ambari 2.4.2.