I am trying to test a spark application and I need to collect the cluster metrics from Ambari within a specific time-frame. I record the start-time and end-time of my spark queries and then try to collect these metrics after a time-gap of at least 13 minutes. I receive all the values as null. However, from the Ambari Web UI, I can see that the metrics for that time-interval are available.
I tried a tweak of adding a millisecond to the start-time and the metrics magically appeared. However, that tweak now fails to work, in that now it requires me to add 2 milliseconds to the start-time. Can anybody help me get the metrics through the REST calls ?
I am testing the application from a Java program running on Eclipse in Windows 7. The queries are made on a Hive table over a JDBC connection, hosted on a VM of HDP 2.4 with 11GB RAM to it. The system has 16GB of total memory.
This issue seems strange. Can you increase the window size of the interval you request (around 10mins between start and and end times)? Also, can you share the full JSON you got when you get Nulls as well as when you get metrics? I can compare them to find out what is wrong.