SmartSense based troubleshooting for YARN container apps

[#1]

I've been having following questions about SmartSense. Would anyone who has used it, be able to help?

1. We host hbase as YARN app and use slider for the same. I notice that SmartSense has a support for hbase monitoring/ troubleshooting. Just wanted to know if that is extensible to Hbase on Yarn too?

2. Does SmartSense help with piecing together troubleshooting information from so different logs? For example, YARN container app may be down because Yarn node manager went down, which in turn may be down because Yarn RM terminated all apps on that node manager. Piecing this information today requires looking into resource manager/ node manager logs along with Hbase logs.

Another case is that of say, an app going down because ZooKeeper has hit the maxClientCnxns issues and would not allow any more incoming connections from that host.

Those are just representative set of problems. Does SmartSense help there?

3. Does SmartSense also help identify issues such as Kerberos ticket renewal issue, SSL issues, open file handles issues?

Thanks,

Sumit

[#2 · Accepted solution]

@Sumit Nigam, answering your individual questions:

1. No, I expect SmartSense would not be able to analyze an HBase instance deployed via Slider at this time. SmartSense works by constructing a model of the cluster, including its configuration files, and then running a set of rules against those configuration files to generate recommendations. In the case of Slider, the HBase configuration files would reside inside its Slider Application Package, not the typical file system location. I don't believe SmartSense currently is equipped to inspect Slider application packages. @sheetal or @Paul Codding, could you please confirm (or deny) this?

2. SmartSense operates by running a set of rules against captured information of a cluster's configuration, including HDP component configuration files and host OS configuration. It does not perform an exhaustive capture of all logs in the cluster and execute rules against those logs. Some of the cases you described likely would be better served by runtime operational monitoring in Ambari.

3. SmartSense is capable of identifying and recommending use of secure mode and additional security best practices. There is also a rule that checks configured open file limits and makes recommendations if limits are not within an acceptable range.