After upgrading Ambari (not entirely sure which version but think it was Ambari 2.4.2 to 2.5.0.3), Smartsense View came up unavailable with these messages: "Service Unavailable" "The SmartSense service is currently unavailable. Please make sure the SmartSense Server is up and running." See screenshot In the /var/log/ambari-server/ambari-server.log log you many see entries like this when you click on the Smartsense view. 14 Jul 2017 11:14:05,523 ERROR [ambari-client-thread-467377] ServerProxy:162 - Failed to execute GET /recommendations. Reason: Connection refused 14 Jul 2017 11:14:14,734 ERROR [ambari-client-thread-467506] ServerProxy:162 - Failed to execute GET /context. Reason: Connection refused 14 Jul 2017 11:14:14,900 ERROR [ambari-client-thread-467507] ServerProxy:162 - Failed to execute GET /checkconfig. Reason: Connection refused diff'ing the hst-server.ini config file against previous saved config versions yielded this: # diff conf/hst-server.ini conf_13_04_17_04_17.save/ 2,4d1 < enable.flex.subscription = false < smartsense.id=X-00000000-X-00000000 < flex.subscription.id = 6a4 > smartsense.id=X-00000000-X-00000000 27,28d24 < url = http://:9000 < max.heap = 2048 30d25 < run.as.user = 33d27 < version = 1.4.0.2.5.0.3-7 52,57d45 < [recommendation] < auto.download.interval = 300 < feedback.push.maxentries = 50 < feedback.push.interval = 1800 < expiry = 30 < Was missing a valid hostname in the url property. Find this in Ambari Smartsense Config tab under Advanced hst-server-conf and add the hostname of the HST server to the server.url property (can't share the screenshot because weirdly his property ghosted into view but now I can't re-find it in Ambari - know it's there because under the [server] section in the hst-server.ini there is a url = http://<hst server name>:9000 - this strangeness was probably the reason it took me so long to find the issue in the first place). Restart Smartsense via Ambari and the Smartsense View was working as usual. ... View more

I didn't want to clear the ATS and lose all the job history. I wanted to fix the corruption and preserve the ATS leveldb entries. 2017-06-15 19:17:43,871 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 43 missing files; e.g.: /data/hadoop/ats/leveldb/leveldb-timeline-store/domain-ldb/000015.sst <snipped> 2017-06-15 19:17:43,871 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping ApplicationHistoryServer metrics system... 2017-06-15 19:17:43,873 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2017-06-15 19:17:43,875 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - ApplicationHistoryServer metrics system stopped. 2017-06-15 19:17:43,875 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(605)) - ApplicationHistoryServer metrics system shutdown complete. 2017-06-15 19:17:43,876 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(171)) - Error starting ApplicationHistoryServer <snipped> 2017-06-15 19:17:43,877 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status -1 2017-06-15 19:17:43,880 INFO applicationhistoryservice.ApplicationHistoryServer (LogAdapter.java:info(45)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer This post on HCC was helpful: ATS issue. What wasn't obvious from that post is that there could be more than one leveldb "partition" (my term) corrupted. In my case, there was a corruption of the following which required these remedial steps ... I had to remove each of the following CURRENT files: /data/hadoop/ats/leveldb/leveldb-timeline-store/domain-ldb/CURRENT /data/hadoop/ats/leveldb/leveldb-timeline-store/starttime-ldb/CURRENT /data/hadoop/ats/leveldb/leveldb-timeline-store/owner-ldb/CURRENT /data/hadoop/yarn/timeline/timeline-state-store.ldb/CURRENT I kept copies of the CURRENT files in /tmp/leveldbissue like this: cd <dir where the leveldb files were reporting missing> mkdir /tmp/leveldbissue cp -ip CURRENT /tmp/leveldbissue/xxxx-ldb (where xxxx is the deepest dir where the leveldb files were reporting missing) rm CURRENT Each time a corrupted leveldb files were found, do the above and restart the ATS (via Ambari) and iterate until no more xxxxx-ldb/.ldb files reporting 'corruption'. Here are the files at the end of my iterations through 'corruptions'. $ cd /tmp/leveldbissue $ ls -alt CURR* -rw-r--r-- 1 root root 16 Jun 15 20:28 CURRENT.starttime-ldb -rw-r--r-- 1 yarn hadoop 16 Apr 13 04:51 CURRENT.timeline-state-store.ldb -rw-r--r-- 1 yarn hadoop 16 Apr 13 04:51 CURRENT.owner-ldb -rw-r--r-- 1 yarn hadoop 16 Apr 13 04:48 CURRENT.domain-ldb The process was fairly painless though the "recovery" process on ATS restart after removing the CURRENT files did take some time for the busy cluster I was working on at the time. If downtime is more of a concern than preserving the ATS job history, you could consider clearing the ATS data. Hope this helps - not a nice one to get in the small hours of the morning when you are on your own. ... View more