question Re: Utility to for utilization. in Archives of Support Questions (Read Only)

Utility to for utilization.

ashneesharma88 — Tue, 31 Jan 2017 18:59:51 GMT

Is there any utility to find out the resource utilization for particular job level.

Like:- CPU, memory, per queue utilization, time utilization.

Re: Utility to for utilization.

aervits — Tue, 31 Jan 2017 20:09:29 GMT

1. If you're using Tez, it has a really nice view for those kinds of statistics. Ambari provides access to the view in a secure way.

2. SmartSense is an offering for Hortonworks customers and leverages Ambari Metrics Collector to provide you with metrics across Hive, HBase, HDFS, Spark, YARN, etc. Including capacity projections months ahead. It is provided via Apache Spark queries and Apache Zeppelin notebook.

3. You can look elsewhere like Dr. Elephant to build your own solution

I'm biased but SmartSense I believe gives the best overall view of a cluster. If you are a customer, I don't see a reason not to use it.

Re: Utility to for utilization.

ashneesharma88 — Wed, 01 Feb 2017 14:46:18 GMT

@Artem Ervits

Is there any docs link to go through for more details?

And this will be provide me the per job utilization details or overall cluster level detail?

Re: Utility to for utilization.

aervits — Wed, 01 Feb 2017 22:08:26 GMT

@Ashnee Sharma here you go http://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.3.1/bk_user-guide/content/activity_explorer.html

go through each page, it says specifically the level of detail per component. In the next release, we're going to add cluster capacity projections and more extensive analysis of components.

If this answers your question, please mark accept the answer as best.