I am looking for services, solutions that can be leveraged to monitor non-ambari cluster health and status.
I have a non-ambari cluster with kerberos security enabled. I need to monitor health status of the cluster, including most of the components (hbase, kafka, spark, solr) in HDP. If anyone is aware of any services solutions that they have come across or used to monitor non-ambari cluster, please do share with us.
To monitor services health on a non ambari cluster, you can use Nagios and Ganglia.
That being said, using Ambari gives you far more capabilities than Nagios and Ganglia combined and that's why you should consider using an Ambari managed cluster.
The whole purpose of having Ambari Managed cluster is that it makes the Whole cluster provisioning, monitoring and maintenance very easy and reliable. (It is recommended to use Ambari which can be also be setup over an existing HDP cluster as well, which is know as "Ambari Takeover")
However if you do not have ambari installed on your cluster then you might want to try some of the following approach:
1. Monitoring using JMX Metrics APIs for HDFS Daemons: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hdfs-administration/content/ch_jmx_metri...
2. HDP components JVM monitoring: https://community.hortonworks.com/articles/34603/monitor-hadoop-jvms-with-jvisualvm.html
3. Use the component specific Statistics monitoring features like for example in case of HBase: http://hbase.apache.org/book.html#hbase_metrics
4. One can use Ganglia: https://community.hortonworks.com/questions/78844/how-to-monitor-performance-in-hadoop.html
5. HDFS DataNode Moniroting: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hdfs-administration/content/ch_hdfs_data...
Conclusion: Ambari is best for provisioning, managing, monitoring and securing Apache Hadoop clusters.