Support Questions
Find answers, ask questions, and share your expertise

hang "Restart all required services" - how identify cause ?

hang "Restart all required services" - how identify cause ?

Contributor

HDP-2.6.1.0

6 REPLIES 6
Highlighted

Re: hang "Restart all required services" - how identify cause ?

Contributor

from what is it necessary to begin ?

Highlighted

Re: hang "Restart all required services" - how identify cause ?

Super Mentor

@Dmitro Vasilenko

If the "Restart all required services" button / action is hanging in ambari then it will be good to check if "Starting of Individual Services" are working fine or not?

Like Ambari UI --> MapReduce --> Service Actions --> Restart All

- Above is just to isolate if the issue is only with the "Restart all required services" button/action or even with restarting any individual service.

- When we click on "Restart all required services" in ambari UI then do we notice any warning / error message in "/var/log/ambari-server/ambari-server.log" ?

- Also looking at few ambari-agent logs will be helpful to know if the ambari agents are actually receiving the restart component requests or not?

- Are all the agents are sending the heart beat messages to ambari server? Means the "Hosts" tab in ambari UI shows all the hosts accessible (green)?

- Have we tried restarting ambari server (agents) ? Are the agents running on the same RPM version or any recent binary/package upgrade happened recently.

- Also looking at the Chrome Browser --> More Tools --> Developer Tools will help in finding the exact API calls, If they are showing any UI error or not when the restart all required services" button is clicked.

- Please let us know what is the size of the ambari cluster (Number of nodes) ? In large clusters some threads tuning might be needed to perform large bulk operations. So please check if you see any insufficient thread pool related message in your ambari server logs? The log line should look something like following (Values might be different in your case)

 WARN [main] AmbariServer:693 - The configured Jetty qtp-ambari-agent thread pool value of 25 is not sufficient on a host with 32 processors. Increasing the value to 28.

.

Highlighted

Re: hang "Restart all required services" - how identify cause ?

Contributor

Thanks you!

Have error in ambari log /var/log/ambari-server/ambari-server.log :

Trying restart LogSearch service and ambari server on hdf-02 ----------------------------

03 Aug 2017 14:42:04,276 ERROR [pool-11-thread-1] LoggingRequestHelperImpl:45 - Error occurred while trying to connect to the LogSearch service... java.io.FileNotFoundException: http://hdf-02:61888/api/v1/service/logs?component_name=spark_jobhistory_server&pageSize=1&host_name=... at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) at org.apache.ambari.server.controller.logging.LoggingRequestHelperImpl$DefaultNetworkConnection.readQueryResponseFromServer(LoggingRequestHelperImpl.java:482) at org.apache.ambari.server.controller.logging.LoggingRequestHelperImpl.sendQueryRequest(LoggingRequestHelperImpl.java:169) at org.apache.ambari.server.controller.logging.LoggingRequestHelperImpl.sendGetLogFileNamesRequest(LoggingRequestHelperImpl.java:308) at org.apache.ambari.server.controller.logging.LogSearchDataRetrievalService$LogSearchFileNameRequestRunnable.run(LogSearchDataRetrievalService.java:332) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) (END)

Highlighted

Re: hang "Restart all required services" - how identify cause ?

Super Mentor

@Dmitro Vasilenko

Can you please check the "logsearch" api version. Sometimes it happens that the logsearch APIs are not upgraded (when we perform ambari upgrade) Sometimes during the stack Upgrade. That may lead to FileNotFoundException that you are getting.

# ls -lart  /usr/lib/ambari-logsearch-portal/libs/
# yum upgrade ambari-logsearch-portal


Post upgrade steps:
https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/upgrade_ambari_lo...

.

Highlighted

Re: hang "Restart all required services" - how identify cause ?

Contributor

hi! probles resolve after restart ambari agent.

Highlighted

Re: hang "Restart all required services" - how identify cause ?

Contributor
Hi!
Apache Ambari  - Version 2.5.0.3  (ambari-logsearch-portal not upgraded ?)  :


[root@hdf-02 logs]# ls -lart  /usr/lib/ambari-logsearch-portal/libs/
total 48108
-rw-r--r-- 1 logsearch hadoop   792964 Nov 23  2016 zookeeper-3.4.6.jar
-rw-r--r-- 1 logsearch hadoop    94672 Nov 23  2016 xz-1.0.jar
-rw-r--r-- 1 logsearch hadoop    15010 Nov 23  2016 xmlenc-0.52.jar
-rw-r--r-- 1 logsearch hadoop   486013 Nov 23  2016 woodstox-core-asl-4.4.1.jar
-rw-r--r-- 1 logsearch hadoop    23346 Nov 23  2016 stax-api-1.0-2.jar
-rw-r--r-- 1 logsearch hadoop   161867 Nov 23  2016 stax2-api-3.1.4.jar
-rw-r--r-- 1 logsearch hadoop   767232 Nov 23  2016 spring-web-4.2.5.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   242551 Nov 23  2016 spring-tx-3.2.13.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   326872 Nov 23  2016 spring-security-web-4.0.4.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   103988 Nov 23  2016 spring-security-ldap-4.0.4.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop     7060 Nov 23  2016 spring-security-kerberos-web-1.0.1.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop    20626 Nov 23  2016 spring-security-kerberos-core-1.0.1.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop    11808 Nov 23  2016 spring-security-kerberos-client-1.0.1.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   357254 Nov 23  2016 spring-security-core-4.0.4.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   473345 Nov 23  2016 spring-security-config-4.0.4.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   376095 Nov 23  2016 spring-ldap-core-2.0.4.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   262988 Nov 23  2016 spring-expression-4.2.5.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop  1078771 Nov 23  2016 spring-core-4.2.5.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop  1096773 Nov 23  2016 spring-context-4.2.5.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   731562 Nov 23  2016 spring-beans-4.2.5.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   370067 Nov 23  2016 spring-aop-4.2.5.RELEASE.jar
-rw-r--r-- 1 logsearch hadoop   692068 Nov 23  2016 solr-solrj-5.5.2.jar
-rw-r--r-- 1 logsearch hadoop   995968 Nov 23  2016 snappy-java-1.0.4.1.jar
-rw-r--r-- 1 logsearch hadoop     8819 Nov 23  2016 slf4j-log4j12-1.7.2.jar
-rw-r--r-- 1 logsearch hadoop    26083 Nov 23  2016 slf4j-api-1.7.2.jar
-rw-r--r-- 1 logsearch hadoop   533455 Nov 23  2016 protobuf-java-2.5.0.jar
-rw-r--r-- 1 logsearch hadoop    29555 Nov 23  2016 paranamer-2.3.jar
-rw-r--r-- 1 logsearch hadoop    26720 Nov 23  2016 noggit-0.6.jar
-rw-r--r-- 1 logsearch hadoop   489884 Nov 23  2016 log4j-1.2.17.jar
-rw-r--r-- 1 logsearch hadoop    46367 Nov 23  2016 jsr311-api-1.1.1.jar
-rw-r--r-- 1 logsearch hadoop    33015 Nov 23  2016 jsr305-1.3.9.jar
-rw-r--r-- 1 logsearch hadoop   100636 Nov 23  2016 jsp-api-2.1.jar
-rw-r--r-- 1 logsearch hadoop   185746 Nov 23  2016 jsch-0.1.42.jar
-rw-r--r-- 1 logsearch hadoop    44138 Nov 23  2016 jetty-xml-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop   107758 Nov 23  2016 jetty-webapp-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop   359238 Nov 23  2016 jetty-util-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop   122014 Nov 23  2016 jetty-servlets-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop   115472 Nov 23  2016 jetty-servlet-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop   418554 Nov 23  2016 jetty-server-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop    95913 Nov 23  2016 jetty-security-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop    54108 Nov 23  2016 jetty-plus-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop    41263 Nov 23  2016 jetty-jndi-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop   108131 Nov 23  2016 jetty-io-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop   105601 Nov 23  2016 jetty-http-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop    15970 Nov 23  2016 jetty-continuation-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop    80135 Nov 23  2016 jetty-annotations-9.2.11.v20150529.jar
-rw-r--r-- 1 logsearch hadoop    67758 Nov 23  2016 jettison-1.1.jar
-rw-r--r-- 1 logsearch hadoop   539735 Nov 23  2016 jets3t-0.9.0.jar
-rw-r--r-- 1 logsearch hadoop    17418 Nov 23  2016 jersey-spring-1.19.jar
-rw-r--r-- 1 logsearch hadoop   128719 Nov 23  2016 jersey-servlet-1.19.jar
-rw-r--r-- 1 logsearch hadoop   702882 Nov 23  2016 jersey-server-1.19.jar
-rw-r--r-- 1 logsearch hadoop   165345 Nov 23  2016 jersey-json-1.19.jar
-rw-r--r-- 1 logsearch hadoop   436689 Nov 23  2016 jersey-core-1.19.jar
-rw-r--r-- 1 logsearch hadoop   134021 Nov 23  2016 jersey-client-1.19.jar
-rw-r--r-- 1 logsearch hadoop    22284 Nov 23  2016 jersey-apache-client-1.19.jar
-rw-r--r-- 1 logsearch hadoop 15211416 Nov 23  2016 jdk.tools-1.7.jar
-rw-r--r-- 1 logsearch hadoop    16519 Nov 23  2016 jcl-over-slf4j-1.7.7.jar
-rw-r--r-- 1 logsearch hadoop   890168 Nov 23  2016 jaxb-impl-2.2.3-1.jar
-rw-r--r-- 1 logsearch hadoop   105134 Nov 23  2016 jaxb-api-2.2.2.jar
-rw-r--r-- 1 logsearch hadoop    95806 Nov 23  2016 javax.servlet-api-3.1.0.jar
-rw-r--r-- 1 logsearch hadoop    18490 Nov 23  2016 java-xmlbuilder-0.4.jar
-rw-r--r-- 1 logsearch hadoop    26366 Nov 23  2016 javax.annotation-api-1.2.jar
-rw-r--r-- 1 logsearch hadoop    27074 Nov 23  2016 jackson-xc-1.9.9.jar
-rw-r--r-- 1 logsearch hadoop   780664 Nov 23  2016 jackson-mapper-asl-1.9.13.jar
-rw-r--r-- 1 logsearch hadoop    18324 Nov 23  2016 jackson-jaxrs-1.9.9.jar
-rw-r--r-- 1 logsearch hadoop   232102 Nov 23  2016 jackson-core-asl-1.9.9.jar
-rw-r--r-- 1 logsearch hadoop    40631 Nov 23  2016 httpmime-4.4.1.jar
-rw-r--r-- 1 logsearch hadoop   322234 Nov 23  2016 httpcore-4.4.1.jar
-rw-r--r-- 1 logsearch hadoop   720931 Nov 23  2016 httpclient-4.4.1.jar
-rw-r--r-- 1 logsearch hadoop  1475955 Nov 23  2016 htrace-core-3.1.0-incubating.jar
-rw-r--r-- 1 logsearch hadoop  3425940 Nov 23  2016 hadoop-common-2.7.0.jar
-rw-r--r-- 1 logsearch hadoop    70600 Nov 23  2016 hadoop-auth-2.7.0.jar
-rw-r--r-- 1 logsearch hadoop    17386 Nov 23  2016 hadoop-annotations-2.7.0.jar
-rw-r--r-- 1 logsearch hadoop  2225441 Nov 23  2016 guava-16.0.jar
-rw-r--r-- 1 logsearch hadoop   189285 Nov 23  2016 gson-2.2.2.jar
-rw-r--r-- 1 logsearch hadoop   270342 Nov 23  2016 curator-recipes-2.7.1.jar
-rw-r--r-- 1 logsearch hadoop   186273 Nov 23  2016 curator-framework-2.7.1.jar
-rw-r--r-- 1 logsearch hadoop    69500 Nov 23  2016 curator-client-2.7.1.jar
-rw-r--r-- 1 logsearch hadoop   273370 Nov 23  2016 commons-net-3.1.jar
-rw-r--r-- 1 logsearch hadoop  1599627 Nov 23  2016 commons-math3-3.1.1.jar
-rw-r--r-- 1 logsearch hadoop    60686 Nov 23  2016 commons-logging-1.1.1.jar
-rw-r--r-- 1 logsearch hadoop   279193 Nov 23  2016 commons-lang-2.5.jar
-rw-r--r-- 1 logsearch hadoop   185140 Nov 23  2016 commons-io-2.4.jar
-rw-r--r-- 1 logsearch hadoop   305001 Nov 23  2016 commons-httpclient-3.1.jar
-rw-r--r-- 1 logsearch hadoop   143602 Nov 23  2016 commons-digester-1.8.jar
-rw-r--r-- 1 logsearch hadoop   298829 Nov 23  2016 commons-configuration-1.6.jar
-rw-r--r-- 1 logsearch hadoop   241367 Nov 23  2016 commons-compress-1.4.1.jar
-rw-r--r-- 1 logsearch hadoop   575389 Nov 23  2016 commons-collections-3.2.1.jar
-rw-r--r-- 1 logsearch hadoop   263865 Nov 23  2016 commons-codec-1.8.jar
-rw-r--r-- 1 logsearch hadoop    52988 Nov 23  2016 commons-cli-1.3.1.jar
-rw-r--r-- 1 logsearch hadoop   206035 Nov 23  2016 commons-beanutils-core-1.8.0.jar
-rw-r--r-- 1 logsearch hadoop   188671 Nov 23  2016 commons-beanutils-1.7.0.jar
-rw-r--r-- 1 logsearch hadoop   299725 Nov 23  2016 cglib-3.2.4.jar
-rw-r--r-- 1 logsearch hadoop   303139 Nov 23  2016 avro-1.7.4.jar
-rw-r--r-- 1 logsearch hadoop    29038 Nov 23  2016 asm-tree-5.0.1.jar
-rw-r--r-- 1 logsearch hadoop    41718 Nov 23  2016 asm-commons-5.0.1.jar
-rw-r--r-- 1 logsearch hadoop    53217 Nov 23  2016 asm-5.0.1.jar
-rw-r--r-- 1 logsearch hadoop    82413 Nov 23  2016 api-util-1.0.0-M26.jar
-rw-r--r-- 1 logsearch hadoop    15418 Nov 23  2016 api-asn1-api-1.0.0-M26.jar
-rw-r--r-- 1 logsearch hadoop   689835 Nov 23  2016 apacheds-kerberos-codec-2.0.0-M19.jar
-rw-r--r-- 1 logsearch hadoop    44990 Nov 23  2016 apacheds-i18n-2.0.0-M19.jar
-rw-r--r-- 1 logsearch hadoop     4467 Nov 23  2016 aopalliance-1.0.jar
-rw-r--r-- 1 logsearch hadoop    12143 Nov 23  2016 ant-launcher-1.7.1.jar
-rw-r--r-- 1 logsearch hadoop  1323005 Nov 23  2016 ant-1.7.1.jar
-rw-r--r-- 1 logsearch hadoop    36518 Nov 23  2016 ambari-metrics-common-2.4.2.0.136.jar
-rw-r--r-- 1 logsearch hadoop    51972 Nov 23  2016 ambari-logsearch-solr-client-2.4.2.0.136.jar
-rw-r--r-- 1 logsearch hadoop     7693 Nov 23  2016 ambari-logsearch-appender-2.4.2.0.136.jar
-rw-r--r-- 1 logsearch hadoop    62983 Nov 23  2016 activation-1.1.jar
drwxr-xr-x 4 logsearch hadoop      110 Jan 17  2017 ..
drwxr-xr-x 2 logsearch hadoop     8192 Jan 17  2017 .
[root@hdf-02 logs]#