Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Script to get queue usage in graph.

Hi I am using ambari 1.7. Hdp 2.1

We have multiple queue's to run jobs. Now we want to take the output of queue usage in graph . Any idea how to do this.

1 ACCEPTED SOLUTION

To upgrade it will take time so I have written script to get the graph of queue usage data:-

Gnuplot installed on linux box

cat data.sh

#!/bin/bash

WD='/path where we will resotre the data

FN=`date +%Y%m%d%H%M%S`

curl "http://XXX.XXX.XXX:8088/ws/v1/cluster/scheduler" > $WD/$FN

To collect the output in one file for creating one graph for qurue usage:-

cat queue.sh

HM=home path where stored all config and script

WD=$HM/original

QD=$HM/queue-data

rm -rf $QD/*

cd $WD

for que in abc xyz xx

do

for i in `ls`

do

column=`awk -F',' '{ for (i=0; i<=NF; i++) {print i,$i}}' $i| grep $que | tail -1 | awk '{print $1}'`

uc=$((column-6))

auc=$((column-2))

awk -F',' '{print ARGV[3], $ucp, $aucp}' ucp="$uc" aucp="$auc" $i | awk -F' ' '{print $1" "$2" "$3}' | awk -F':' '{print $1" "$2" "$3}' | awk '{print $1" "$3" "$5}' >> $QD/$que

done

done

paste -d' ' $QD/abc $QD/xyz $QD/xx | gawk '{print $1 " " $3 " " $6 " " $9 " " $12 " " $15 " " $18 " " $21 " " $24" " $27}' >>$QD/all_que

Than create a conf file for create graph:-

cat 1.conf

set terminal png truecolor size 2000,600

set output "all-in-1.png"

set autoscale

set xdata time

set timefmt "%Y%m%d%H%M%S"

set format x "%Y/%m/%d\n%H:%M\n"

set style data lines

set key reverse Left outside

set xlabel "Time(in IST)"

set ylabel "Used Capacity"

set title "non-parental-queue-usage"

plot "/home/xyz" using 1:2 title "xyz", '' using 1:3 title "abc", '' using 1:4 title "xx"

/home/xyz is path from where we will get the details of que usage and time

In this conf on xlabel is use for TIme and ylable for used capacity of queue.

Now plot the graph using following command-

cat 1.conf | gnuplot

You will see the graph

Thanks for the reply.....!!!!!!!

View solution in original post

4 REPLIES 4

I'd recommend you upgrading to a more recent version of HDP. In recent versions of Ambari you can install Grafana that proposes such graphs and much more.

http://hortonworks.com/blog/advanced-metrics-visualization-dashboarding-apache-ambari/

Otherwise you have a real time representation of your current queues use in the YARN resource manager UI.

If you want to do something custom, you can have a look at the REST APIs exposed by YARN:

https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Hope this helps.

Mentor

upgrade to HDP 2.5 with Smartsense 1.3 and you will get a Zeppelin dashboard with predefined queries to analyze your queues, along with great features built over the last few years you've been missing out on running an unsupported release.

I wrote one script for this. We can't migrate right now because it will take time to upgrade, this is client system.

To upgrade it will take time so I have written script to get the graph of queue usage data:-

Gnuplot installed on linux box

cat data.sh

#!/bin/bash

WD='/path where we will resotre the data

FN=`date +%Y%m%d%H%M%S`

curl "http://XXX.XXX.XXX:8088/ws/v1/cluster/scheduler" > $WD/$FN

To collect the output in one file for creating one graph for qurue usage:-

cat queue.sh

HM=home path where stored all config and script

WD=$HM/original

QD=$HM/queue-data

rm -rf $QD/*

cd $WD

for que in abc xyz xx

do

for i in `ls`

do

column=`awk -F',' '{ for (i=0; i<=NF; i++) {print i,$i}}' $i| grep $que | tail -1 | awk '{print $1}'`

uc=$((column-6))

auc=$((column-2))

awk -F',' '{print ARGV[3], $ucp, $aucp}' ucp="$uc" aucp="$auc" $i | awk -F' ' '{print $1" "$2" "$3}' | awk -F':' '{print $1" "$2" "$3}' | awk '{print $1" "$3" "$5}' >> $QD/$que

done

done

paste -d' ' $QD/abc $QD/xyz $QD/xx | gawk '{print $1 " " $3 " " $6 " " $9 " " $12 " " $15 " " $18 " " $21 " " $24" " $27}' >>$QD/all_que

Than create a conf file for create graph:-

cat 1.conf

set terminal png truecolor size 2000,600

set output "all-in-1.png"

set autoscale

set xdata time

set timefmt "%Y%m%d%H%M%S"

set format x "%Y/%m/%d\n%H:%M\n"

set style data lines

set key reverse Left outside

set xlabel "Time(in IST)"

set ylabel "Used Capacity"

set title "non-parental-queue-usage"

plot "/home/xyz" using 1:2 title "xyz", '' using 1:3 title "abc", '' using 1:4 title "xx"

/home/xyz is path from where we will get the details of que usage and time

In this conf on xlabel is use for TIme and ylable for used capacity of queue.

Now plot the graph using following command-

cat 1.conf | gnuplot

You will see the graph

Thanks for the reply.....!!!!!!!