This has me a little confused as well. There are three counters for this:
- (1) Ambari Dashboard has "CPU Usage" which always seems to look low. (makes sense for our env)
- (2) YARN has "CPU Utilization" for "% of total cores assigned to containers" which never goes under 40% (makes sense for our env)
- (3) YARN has "Cluster CPU" for "% of CPU utilization across node manager hosts" which is always between 50%-95% (This makes no sense in our env)
So, when our jobs run, 1 doesn't climb much, 2 doesn't move much, 3 quickly hits 95%. We want to run our jobs faster but between 1 & 3 I can't tell if we are high utilization of CPU or not.
Host page CPU seems to trend with 1 (quiet)
But am I hitting a ceiling? 3 can't go any higher