question Re: Script to get queue usage in graph. in Archives of Support Questions (Read Only)

Script to get queue usage in graph.

ashneesharma88 — Wed, 14 Sep 2016 18:08:08 GMT

Hi I am using ambari 1.7. Hdp 2.1

We have multiple queue's to run jobs. Now we want to take the output of queue usage in graph . Any idea how to do this.

Re: Script to get queue usage in graph.

pvillard — Wed, 14 Sep 2016 18:34:05 GMT

I'd recommend you upgrading to a more recent version of HDP. In recent versions of Ambari you can install Grafana that proposes such graphs and much more.

http://hortonworks.com/blog/advanced-metrics-visualization-dashboarding-apache-ambari/

Otherwise you have a real time representation of your current queues use in the YARN resource manager UI.

If you want to do something custom, you can have a look at the REST APIs exposed by YARN:

https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Hope this helps.

Re: Script to get queue usage in graph.

aervits — Wed, 14 Sep 2016 19:53:54 GMT

upgrade to HDP 2.5 with Smartsense 1.3 and you will get a Zeppelin dashboard with predefined queries to analyze your queues, along with great features built over the last few years you've been missing out on running an unsupported release.

Re: Script to get queue usage in graph.

ashneesharma88 — Thu, 15 Sep 2016 20:09:04 GMT

I wrote one script for this. We can't migrate right now because it will take time to upgrade, this is client system.

Re: Script to get queue usage in graph.

ashneesharma88 — Fri, 16 Sep 2016 15:07:25 GMT

To upgrade it will take time so I have written script to get the graph of queue usage data:-

Gnuplot installed on linux box

cat data.sh

#!/bin/bash

WD='/path where we will resotre the data

FN=`date +%Y%m%d%H%M%S`

curl "http://XXX.XXX.XXX:8088/ws/v1/cluster/scheduler" > $WD/$FN

To collect the output in one file for creating one graph for qurue usage:-

cat queue.sh

HM=home path where stored all config and script

WD=$HM/original

QD=$HM/queue-data

rm -rf $QD/*

cd $WD

for que in abc xyz xx

for i in `ls`

column=`awk -F',' '{ for (i=0; i<=NF; i++) {print i,$i}}' $i| grep $que | tail -1 | awk '{print $1}'`

uc=$((column-6))

auc=$((column-2))

awk -F',' '{print ARGV[3], $ucp, $aucp}' ucp="$uc" aucp="$auc" $i | awk -F' ' '{print $1" "$2" "$3}' | awk -F':' '{print $1" "$2" "$3}' | awk '{print $1" "$3" "$5}' >> $QD/$que

done

paste -d' ' $QD/abc $QD/xyz $QD/xx | gawk '{print $1 " " $3 " " $6 " " $9 " " $12 " " $15 " " $18 " " $21 " " $24" " $27}' >>$QD/all_que

Than create a conf file for create graph:-

cat 1.conf

set terminal png truecolor size 2000,600

set output "all-in-1.png"

set autoscale

set xdata time

set timefmt "%Y%m%d%H%M%S"

set format x "%Y/%m/%d\n%H:%M\n"

set style data lines

set key reverse Left outside

set xlabel "Time(in IST)"

set ylabel "Used Capacity"

set title "non-parental-queue-usage"

plot "/home/xyz" using 1:2 title "xyz", '' using 1:3 title "abc", '' using 1:4 title "xx"

/home/xyz is path from where we will get the details of que usage and time

In this conf on xlabel is use for TIme and ylable for used capacity of queue.

Now plot the graph using following command-

cat 1.conf | gnuplot

You will see the graph

Thanks for the reply.....!!!!!!!