Created 09-14-2016 11:08 AM
Hi I am using ambari 1.7. Hdp 2.1
We have multiple queue's to run jobs. Now we want to take the output of queue usage in graph . Any idea how to do this.
Created 09-16-2016 08:07 AM
To upgrade it will take time so I have written script to get the graph of queue usage data:-
Gnuplot installed on linux box
cat data.sh
#!/bin/bash
WD='/path where we will resotre the data
FN=`date +%Y%m%d%H%M%S`
curl "http://XXX.XXX.XXX:8088/ws/v1/cluster/scheduler" > $WD/$FN
To collect the output in one file for creating one graph for qurue usage:-
cat queue.sh
HM=home path where stored all config and script
WD=$HM/original
QD=$HM/queue-data
rm -rf $QD/*
cd $WD
for que in abc xyz xx
do
for i in `ls`
do
column=`awk -F',' '{ for (i=0; i<=NF; i++) {print i,$i}}' $i| grep $que | tail -1 | awk '{print $1}'`
uc=$((column-6))
auc=$((column-2))
awk -F',' '{print ARGV[3], $ucp, $aucp}' ucp="$uc" aucp="$auc" $i | awk -F' ' '{print $1" "$2" "$3}' | awk -F':' '{print $1" "$2" "$3}' | awk '{print $1" "$3" "$5}' >> $QD/$que
done
done
paste -d' ' $QD/abc $QD/xyz $QD/xx | gawk '{print $1 " " $3 " " $6 " " $9 " " $12 " " $15 " " $18 " " $21 " " $24" " $27}' >>$QD/all_que
Than create a conf file for create graph:-
cat 1.conf
set terminal png truecolor size 2000,600
set output "all-in-1.png"
set autoscale
set xdata time
set timefmt "%Y%m%d%H%M%S"
set format x "%Y/%m/%d\n%H:%M\n"
set style data lines
set key reverse Left outside
set xlabel "Time(in IST)"
set ylabel "Used Capacity"
set title "non-parental-queue-usage"
plot "/home/xyz" using 1:2 title "xyz", '' using 1:3 title "abc", '' using 1:4 title "xx"
/home/xyz is path from where we will get the details of que usage and time
In this conf on xlabel is use for TIme and ylable for used capacity of queue.
Now plot the graph using following command-
cat 1.conf | gnuplot
You will see the graph
Thanks for the reply.....!!!!!!!
Created 09-14-2016 11:34 AM
I'd recommend you upgrading to a more recent version of HDP. In recent versions of Ambari you can install Grafana that proposes such graphs and much more.
http://hortonworks.com/blog/advanced-metrics-visualization-dashboarding-apache-ambari/
Otherwise you have a real time representation of your current queues use in the YARN resource manager UI.
If you want to do something custom, you can have a look at the REST APIs exposed by YARN:
https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
Hope this helps.
Created 09-14-2016 12:53 PM
upgrade to HDP 2.5 with Smartsense 1.3 and you will get a Zeppelin dashboard with predefined queries to analyze your queues, along with great features built over the last few years you've been missing out on running an unsupported release.
Created 09-15-2016 01:09 PM
I wrote one script for this. We can't migrate right now because it will take time to upgrade, this is client system.
Created 09-16-2016 08:07 AM
To upgrade it will take time so I have written script to get the graph of queue usage data:-
Gnuplot installed on linux box
cat data.sh
#!/bin/bash
WD='/path where we will resotre the data
FN=`date +%Y%m%d%H%M%S`
curl "http://XXX.XXX.XXX:8088/ws/v1/cluster/scheduler" > $WD/$FN
To collect the output in one file for creating one graph for qurue usage:-
cat queue.sh
HM=home path where stored all config and script
WD=$HM/original
QD=$HM/queue-data
rm -rf $QD/*
cd $WD
for que in abc xyz xx
do
for i in `ls`
do
column=`awk -F',' '{ for (i=0; i<=NF; i++) {print i,$i}}' $i| grep $que | tail -1 | awk '{print $1}'`
uc=$((column-6))
auc=$((column-2))
awk -F',' '{print ARGV[3], $ucp, $aucp}' ucp="$uc" aucp="$auc" $i | awk -F' ' '{print $1" "$2" "$3}' | awk -F':' '{print $1" "$2" "$3}' | awk '{print $1" "$3" "$5}' >> $QD/$que
done
done
paste -d' ' $QD/abc $QD/xyz $QD/xx | gawk '{print $1 " " $3 " " $6 " " $9 " " $12 " " $15 " " $18 " " $21 " " $24" " $27}' >>$QD/all_que
Than create a conf file for create graph:-
cat 1.conf
set terminal png truecolor size 2000,600
set output "all-in-1.png"
set autoscale
set xdata time
set timefmt "%Y%m%d%H%M%S"
set format x "%Y/%m/%d\n%H:%M\n"
set style data lines
set key reverse Left outside
set xlabel "Time(in IST)"
set ylabel "Used Capacity"
set title "non-parental-queue-usage"
plot "/home/xyz" using 1:2 title "xyz", '' using 1:3 title "abc", '' using 1:4 title "xx"
/home/xyz is path from where we will get the details of que usage and time
In this conf on xlabel is use for TIme and ylable for used capacity of queue.
Now plot the graph using following command-
cat 1.conf | gnuplot
You will see the graph
Thanks for the reply.....!!!!!!!