Support Questions
Find answers, ask questions, and share your expertise

Ambari metrics collector is shutting down abruptly

Expert Contributor

Hi can somebody help me with this issue i got all of a sudden this morning. I have ambari metrics collector shut down abruptly twice at random times, but when i start it, it starts succesfully ..

logs are: 2017-05-11 10:07:52,213 INFO TimelineClusterAggregatorSecond: Saving 17188 metric aggregates. 2017-05-11 10:07:53,376 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:07:53 EDT 2017 2017-05-11 10:07:53,376 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:07:53 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:08:45 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494511200000, lagBy: 525 seconds. 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:08:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:08:45 EDT 2017 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494511200000, lagBy: 525 seconds. 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:08:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: 4302 row(s) updated. 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:49 EDT 2017 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:49 EDT 2017 2017-05-11 10:08:57,344 INFO TimelineMetricHostAggregatorMinute: 81766 row(s) updated. 2017-05-11 10:08:57,344 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:57,344 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:57 EDT 2017 2017-05-11 10:08:57,345 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:57 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:09:45 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:06:00 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:06:00 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511560000, lagBy: 225 seconds. 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:09:45 EDT 2017, startTime = Thu May 11 10:06:00 EDT 2017, endTime = Thu May 11 10:08:00 EDT 2017 2017-05-11 10:09:50,585 INFO TimelineClusterAggregatorSecond: Saving 17156 metric aggregates. 2017-05-11 10:09:53,912 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:09:53 EDT 2017 2017-05-11 10:09:53,912 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:09:53 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:11:45 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:08:00 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:08:00 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511680000, lagBy: 225 seconds. 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:11:45 EDT 2017, startTime = Thu May 11 10:08:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:11:50,076 INFO TimelineClusterAggregatorSecond: Saving 17192 metric aggregates. 2017-05-11 10:11:51,296 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:11:51 EDT 2017 2017-05-11 10:11:51,297 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:11:51 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511800000, lagBy: 225 seconds. 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:12:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494511500000, lagBy: 525 seconds. 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,523 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,523 INFO TimelineClusterAggregatorHourly: Last check point time: 1494507600000, lagBy: 4425 seconds. 2017-05-11 10:13:45,523 INFO TimelineClusterAggregatorHourly: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:13:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494511500000, lagBy: 525 seconds. 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494507600000, lagBy: 4425 seconds. 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:13:51,706 INFO TimelineClusterAggregatorMinute: 4298 row(s) updated. 2017-05-11 10:13:51,706 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:51,707 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:13:51 EDT 2017 2017-05-11 10:13:51,707 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:13:51 EDT 2017 2017-05-11 10:13:58,283 INFO TimelineClusterAggregatorSecond: Saving 17144 metric aggregates. 2017-05-11 10:14:05,778 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:10,223 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34288 actions to finish 2017-05-11 10:14:15,780 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:16,139 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11374 actions to finish 2017-05-11 10:14:20,229 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34288 actions to finish 2017-05-11 10:14:25,781 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:28,370 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6402 actions to finish 2017-05-11 10:14:30,236 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34288 actions to finish 2017-05-11 10:14:34,706 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6540 actions to finish 2017-05-11 10:14:35,784 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:38,860 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:14:38 EDT 2017 2017-05-11 10:14:38,860 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:14:38 EDT 2017 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: 4336 row(s) updated. 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: Aggregated cluster metrics for METRIC_AGGREGATE_HOURLY, with startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 10:14:44 EDT 2017 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 10:14:44 EDT 2017 2017-05-11 10:14:45,794 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:48,857 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10120 actions to finish 2017-05-11 10:14:54,291 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 5358 actions to finish 2017-05-11 10:14:55,795 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:57,838 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21566 actions to finish 2017-05-11 10:15:05,798 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:15:07,007 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 12896 actions to finish 2017-05-11 10:15:07,843 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21566 actions to finish 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: 81757 row(s) updated. 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:15:09 EDT 2017 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:15:09 EDT 2017 2017-05-11 10:15:11,125 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2016 actions to finish 2017-05-11 10:15:23,315 INFO TimelineMetricHostAggregatorHourly: 81802 row(s) updated. 2017-05-11 10:15:23,315 INFO TimelineMetricHostAggregatorHourly: Aggregated host metrics for METRIC_RECORD_HOURLY, with startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:15:23,315 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 10:15:23 EDT 2017 2017-05-11 10:15:23,316 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 10:15:23 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:15:45 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:12:00 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:12:00 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511920000, lagBy: 225 seconds. 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:15:45 EDT 2017, startTime = Thu May 11 10:12:00 EDT 2017, endTime = Thu May 11 10:14:00 EDT 2017 2017-05-11 10:15:50,093 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:15:51,712 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:15:51 EDT 2017 2017-05-11 10:15:51,712 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:15:51 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:17:45 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:14:00 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:14:00 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512040000, lagBy: 225 seconds. 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:17:45 EDT 2017, startTime = Thu May 11 10:14:00 EDT 2017, endTime = Thu May 11 10:16:00 EDT 2017 2017-05-11 10:17:50,212 INFO TimelineClusterAggregatorSecond: Saving 17196 metric aggregates. 2017-05-11 10:17:51,316 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:17:51 EDT 2017 2017-05-11 10:17:51,316 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:17:51 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:18:45 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494511800000, lagBy: 525 seconds. 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:18:45 EDT 2017, startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:18:45 EDT 2017 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494511800000, lagBy: 525 seconds. 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:18:45 EDT 2017, startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: 4302 row(s) updated. 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:46 EDT 2017 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:46 EDT 2017 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: 81753 row(s) updated. 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:54 EDT 2017 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:54 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:19:45 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:16:00 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:16:00 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512160000, lagBy: 225 seconds. 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:19:45 EDT 2017, startTime = Thu May 11 10:16:00 EDT 2017, endTime = Thu May 11 10:18:00 EDT 2017 2017-05-11 10:19:54,032 INFO TimelineClusterAggregatorSecond: Saving 17184 metric aggregates. 2017-05-11 10:19:55,196 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:19:55 EDT 2017 2017-05-11 10:19:55,196 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:19:55 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:21:45 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:18:00 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:18:00 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512280000, lagBy: 225 seconds. 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:21:45 EDT 2017, startTime = Thu May 11 10:18:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:21:49,978 INFO TimelineClusterAggregatorSecond: Saving 17252 metric aggregates. 2017-05-11 10:21:53,438 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:21:53 EDT 2017 2017-05-11 10:21:53,438 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:21:53 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:23:45 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512400000, lagBy: 225 seconds. 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:23:45 EDT 2017, startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:22:00 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:23:45 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494512100000, lagBy: 525 seconds. 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:23:45 EDT 2017, startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:23:45 EDT 2017 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494512100000, lagBy: 525 seconds. 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:23:45 EDT 2017, startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:49,249 INFO TimelineClusterAggregatorMinute: 4322 row(s) updated. 2017-05-11 10:23:49,250 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:49,250 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:23:49 EDT 2017 2017-05-11 10:23:49,250 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:23:49 EDT 2017 2017-05-11 10:23:52,849 INFO TimelineClusterAggregatorSecond: Saving 17148 metric aggregates. 2017-05-11 10:24:00,893 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83562 actions to finish 2017-05-11 10:24:04,533 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34296 actions to finish 2017-05-11 10:24:04,555 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:24:04 EDT 2017 2017-05-11 10:24:04,555 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:24:04 EDT 2017 2017-05-11 10:24:06,617 INFO TimelineMetricHostAggregatorMinute: 81781 row(s) updated. 2017-05-11 10:24:06,617 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:24:06,618 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:24:06 EDT 2017 2017-05-11 10:24:06,618 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:24:06 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:25:45 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:22:00 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:22:00 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512520000, lagBy: 225 seconds. 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:25:45 EDT 2017, startTime = Thu May 11 10:22:00 EDT 2017, endTime = Thu May 11 10:24:00 EDT 2017 2017-05-11 10:25:52,140 INFO TimelineClusterAggregatorSecond: Saving 17200 metric aggregates. 2017-05-11 10:25:55,716 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:25:55 EDT 2017 2017-05-11 10:25:55,716 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:25:55 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:27:45 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:24:00 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:24:00 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512640000, lagBy: 225 seconds. 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:27:45 EDT 2017, startTime = Thu May 11 10:24:00 EDT 2017, endTime = Thu May 11 10:26:00 EDT 2017 2017-05-11 10:27:52,191 INFO TimelineClusterAggregatorSecond: Saving 17184 metric aggregates. 2017-05-11 10:27:53,496 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:27:53 EDT 2017 2017-05-11 10:27:53,496 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:27:53 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:28:45 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494512400000, lagBy: 525 seconds. 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:28:45 EDT 2017, startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:28:45 EDT 2017 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494512400000, lagBy: 525 seconds. 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:28:45 EDT 2017, startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: 4301 row(s) updated. 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:47 EDT 2017 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:47 EDT 2017 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: 81760 row(s) updated. 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:54 EDT 2017 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:54 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:29:45 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:26:00 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:26:00 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512760000, lagBy: 225 seconds. 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:29:45 EDT 2017, startTime = Thu May 11 10:26:00 EDT 2017, endTime = Thu May 11 10:28:00 EDT 2017 2017-05-11 10:29:50,043 INFO TimelineClusterAggregatorSecond: Saving 17144 metric aggregates. 2017-05-11 10:29:53,349 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:29:53 EDT 2017 2017-05-11 10:29:53,349 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:29:53 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:31:45 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:28:00 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:28:00 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512880000, lagBy: 225 seconds. 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:31:45 EDT 2017, startTime = Thu May 11 10:28:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:31:50,293 INFO TimelineClusterAggregatorSecond: Saving 17160 metric aggregates. 2017-05-11 10:31:51,293 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:31:51 EDT 2017 2017-05-11 10:31:51,294 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:31:51 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:33:45 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513000000, lagBy: 225 seconds. 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:33:45 EDT 2017, startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:32:00 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:33:45 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494512700000, lagBy: 525 seconds. 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:33:45 EDT 2017, startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:33:45 EDT 2017 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494512700000, lagBy: 525 seconds. 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:33:45 EDT 2017, startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:49,528 INFO TimelineClusterAggregatorMinute: 4301 row(s) updated. 2017-05-11 10:33:49,528 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:49,529 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:33:49 EDT 2017 2017-05-11 10:33:49,529 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:33:49 EDT 2017 2017-05-11 10:33:55,444 INFO TimelineClusterAggregatorSecond: Saving 17088 metric aggregates. 2017-05-11 10:34:03,179 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83520 actions to finish 2017-05-11 10:34:08,200 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34176 actions to finish 2017-05-11 10:34:08,200 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22106 actions to finish 2017-05-11 10:34:08,225 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:34:08 EDT 2017 2017-05-11 10:34:08,225 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:34:08 EDT 2017 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: 81760 row(s) updated. 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:34:10 EDT 2017 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:34:10 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:35:45 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:32:00 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:32:00 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513120000, lagBy: 225 seconds. 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:35:45 EDT 2017, startTime = Thu May 11 10:32:00 EDT 2017, endTime = Thu May 11 10:34:00 EDT 2017 2017-05-11 10:35:50,394 INFO TimelineClusterAggregatorSecond: Saving 17176 metric aggregates. 2017-05-11 10:35:53,974 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:35:53 EDT 2017 2017-05-11 10:35:53,974 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:35:53 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:37:45 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:34:00 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:34:00 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513240000, lagBy: 225 seconds. 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:37:45 EDT 2017, startTime = Thu May 11 10:34:00 EDT 2017, endTime = Thu May 11 10:36:00 EDT 2017 2017-05-11 10:37:50,219 INFO TimelineClusterAggregatorSecond: Saving 17176 metric aggregates. 2017-05-11 10:37:53,999 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:37:53 EDT 2017 2017-05-11 10:37:53,999 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:37:53 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:38:45 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513000000, lagBy: 525 seconds. 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:38:45 EDT 2017, startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:38:45 EDT 2017 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513000000, lagBy: 525 seconds. 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:38:45 EDT 2017, startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: 4298 row(s) updated. 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:47 EDT 2017 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:47 EDT 2017 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: 81753 row(s) updated. 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:52 EDT 2017 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:52 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:39:45 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:36:00 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:36:00 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513360000, lagBy: 225 seconds. 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:39:45 EDT 2017, startTime = Thu May 11 10:36:00 EDT 2017, endTime = Thu May 11 10:38:00 EDT 2017 2017-05-11 10:39:49,836 INFO TimelineClusterAggregatorSecond: Saving 17140 metric aggregates. 2017-05-11 10:39:50,902 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:39:50 EDT 2017 2017-05-11 10:39:50,902 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:39:50 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:41:45 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:38:00 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:38:00 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513480000, lagBy: 225 seconds. 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:41:45 EDT 2017, startTime = Thu May 11 10:38:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:41:50,617 INFO TimelineClusterAggregatorSecond: Saving 17224 metric aggregates. 2017-05-11 10:41:51,775 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:41:51 EDT 2017 2017-05-11 10:41:51,775 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:41:51 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:43:45 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513600000, lagBy: 225 seconds. 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:43:45 EDT 2017, startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:42:00 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:43:45 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513300000, lagBy: 525 seconds. 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:43:45 EDT 2017, startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:43:45 EDT 2017 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513300000, lagBy: 525 seconds. 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:43:45 EDT 2017, startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: 4309 row(s) updated. 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:43:49 EDT 2017 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:43:49 EDT 2017 2017-05-11 10:43:54,791 INFO TimelineClusterAggregatorSecond: Saving 17128 metric aggregates. 2017-05-11 10:44:01,487 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:44:01 EDT 2017 2017-05-11 10:44:01,488 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:44:01 EDT 2017 2017-05-11 10:44:02,684 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83536 actions to finish 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: 81768 row(s) updated. 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:44:04 EDT 2017 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:44:04 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:45:45 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:42:00 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:42:00 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513720000, lagBy: 225 seconds. 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:45:45 EDT 2017, startTime = Thu May 11 10:42:00 EDT 2017, endTime = Thu May 11 10:44:00 EDT 2017 2017-05-11 10:45:50,424 INFO TimelineClusterAggregatorSecond: Saving 17164 metric aggregates. 2017-05-11 10:45:51,606 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:45:51 EDT 2017 2017-05-11 10:45:51,606 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:45:51 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:47:45 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:44:00 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:44:00 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513840000, lagBy: 225 seconds. 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:47:45 EDT 2017, startTime = Thu May 11 10:44:00 EDT 2017, endTime = Thu May 11 10:46:00 EDT 2017 2017-05-11 10:47:50,155 INFO TimelineClusterAggregatorSecond: Saving 17164 metric aggregates. 2017-05-11 10:47:51,406 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:47:51 EDT 2017 2017-05-11 10:47:51,406 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:47:51 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:48:45 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513600000, lagBy: 525 seconds. 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:48:45 EDT 2017, startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:48:45 EDT 2017 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513600000, lagBy: 525 seconds. 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:48:45 EDT 2017, startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: 4294 row(s) updated. 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:48 EDT 2017 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:48 EDT 2017 2017-05-11 10:48:54,171 INFO TimelineMetricHostAggregatorMinute: 81750 row(s) updated. 2017-05-11 10:48:54,171 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:54,172 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:54 EDT 2017 2017-05-11 10:48:54,172 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:54 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:49:45 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:46:00 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:46:00 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513960000, lagBy: 225 seconds. 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:49:45 EDT 2017, startTime = Thu May 11 10:46:00 EDT 2017, endTime = Thu May 11 10:48:00 EDT 2017 2017-05-11 10:49:49,760 INFO TimelineClusterAggregatorSecond: Saving 17148 metric aggregates. 2017-05-11 10:49:53,353 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:49:53 EDT 2017 2017-05-11 10:49:53,353 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:49:53 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:51:45 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:48:00 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:48:00 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514080000, lagBy: 225 seconds. 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:51:45 EDT 2017, startTime = Thu May 11 10:48:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:51:50,505 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:51:51,622 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:51:51 EDT 2017 2017-05-11 10:51:51,622 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:51:51 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:53:45 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514200000, lagBy: 225 seconds. 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:53:45 EDT 2017, startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:52:00 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:53:45 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513900000, lagBy: 525 seconds. 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:53:45 EDT 2017, startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:53:45 EDT 2017 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513900000, lagBy: 525 seconds. 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:53:45 EDT 2017, startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: 4306 row(s) updated. 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:53:49 EDT 2017 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:53:49 EDT 2017 2017-05-11 10:53:53,442 INFO TimelineClusterAggregatorSecond: Saving 17104 metric aggregates. 2017-05-11 10:54:01,116 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83530 actions to finish 2017-05-11 10:54:04,123 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:54:04 EDT 2017 2017-05-11 10:54:04,123 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:54:04 EDT 2017 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: 81765 row(s) updated. 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:54:07 EDT 2017 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:54:07 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:55:45 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:52:00 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:52:00 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514320000, lagBy: 225 seconds. 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:55:45 EDT 2017, startTime = Thu May 11 10:52:00 EDT 2017, endTime = Thu May 11 10:54:00 EDT 2017 2017-05-11 10:55:50,403 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:55:51,472 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:55:51 EDT 2017 2017-05-11 10:55:51,473 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:55:51 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:57:45 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:54:00 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:54:00 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514440000, lagBy: 225 seconds. 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:57:45 EDT 2017, startTime = Thu May 11 10:54:00 EDT 2017, endTime = Thu May 11 10:56:00 EDT 2017 2017-05-11 10:57:50,176 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:57:55,360 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:57:55 EDT 2017 2017-05-11 10:57:55,360 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:57:55 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:58:45 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494514200000, lagBy: 525 seconds. 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:58:45 EDT 2017, startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:58:45 EDT 2017 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494514200000, lagBy: 525 seconds. 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:58:45 EDT 2017, startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:47,112 INFO TimelineClusterAggregatorMinute: 4293 row(s) updated. 2017-05-11 10:58:47,113 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:47,113 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:47 EDT 2017 2017-05-11 10:58:47,113 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:47 EDT 2017 2017-05-11 10:58:55,460 INFO TimelineMetricHostAggregatorMinute: 81746 row(s) updated. 2017-05-11 10:58:55,461 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:55,461 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:55 EDT 2017 2017-05-11 10:58:55,461 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:55 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:59:45 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:56:00 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:56:00 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514560000, lagBy: 225 seconds. 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:59:45 EDT 2017, startTime = Thu May 11 10:56:00 EDT 2017, endTime = Thu May 11 10:58:00 EDT 2017 2017-05-11 10:59:50,395 INFO TimelineClusterAggregatorSecond: Saving 17128 metric aggregates. 2017-05-11 10:59:51,533 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:59:51 EDT 2017 2017-05-11 10:59:51,533 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:59:51 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:01:45 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:58:00 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:58:00 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514680000, lagBy: 225 seconds. 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:01:45 EDT 2017, startTime = Thu May 11 10:58:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:01:50,245 INFO TimelineClusterAggregatorSecond: Saving 17256 metric aggregates. 2017-05-11 11:01:51,528 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:01:51 EDT 2017 2017-05-11 11:01:51,528 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:01:51 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:03:45 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514800000, lagBy: 225 seconds. 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:03:45 EDT 2017, startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:02:00 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:03:45 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494514500000, lagBy: 525 seconds. 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:03:45 EDT 2017, startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:03:45 EDT 2017 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494514500000, lagBy: 525 seconds. 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:03:45 EDT 2017, startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: 4316 row(s) updated. 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:03:49 EDT 2017 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:03:49 EDT 2017 2017-05-11 11:03:55,333 INFO TimelineClusterAggregatorSecond: Saving 17252 metric aggregates. 2017-05-11 11:04:03,165 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83546 actions to finish 2017-05-11 11:04:06,437 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:04:06 EDT 2017 2017-05-11 11:04:06,437 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:04:06 EDT 2017 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: 81773 row(s) updated. 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:04:09 EDT 2017 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:04:09 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:05:45 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:02:00 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:02:00 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514920000, lagBy: 225 seconds. 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:05:45 EDT 2017, startTime = Thu May 11 11:02:00 EDT 2017, endTime = Thu May 11 11:04:00 EDT 2017 2017-05-11 11:05:50,025 INFO TimelineClusterAggregatorSecond: Saving 17212 metric aggregates. 2017-05-11 11:05:51,347 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:05:51 EDT 2017 2017-05-11 11:05:51,347 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:05:51 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:07:45 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:04:00 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:04:00 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515040000, lagBy: 225 seconds. 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:07:45 EDT 2017, startTime = Thu May 11 11:04:00 EDT 2017, endTime = Thu May 11 11:06:00 EDT 2017 2017-05-11 11:07:52,400 INFO TimelineClusterAggregatorSecond: Saving 17184 metric aggregates. 2017-05-11 11:07:53,665 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:07:53 EDT 2017 2017-05-11 11:07:53,665 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:07:53 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:08:45 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494514800000, lagBy: 525 seconds. 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:08:45 EDT 2017, startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:08:45 EDT 2017 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494514800000, lagBy: 525 seconds. 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:08:45 EDT 2017, startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: 4319 row(s) updated. 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:47 EDT 2017 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:47 EDT 2017 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: 81768 row(s) updated. 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:52 EDT 2017 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:52 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:09:45 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:06:00 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:06:00 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515160000, lagBy: 225 seconds. 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:09:45 EDT 2017, startTime = Thu May 11 11:06:00 EDT 2017, endTime = Thu May 11 11:08:00 EDT 2017 2017-05-11 11:09:49,962 INFO TimelineClusterAggregatorSecond: Saving 17148 metric aggregates. 2017-05-11 11:09:54,032 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:09:54 EDT 2017 2017-05-11 11:09:54,032 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:09:54 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:11:45 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:08:00 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:08:00 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515280000, lagBy: 225 seconds. 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:11:45 EDT 2017, startTime = Thu May 11 11:08:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:11:50,670 INFO TimelineClusterAggregatorSecond: Saving 17220 metric aggregates. 2017-05-11 11:11:51,771 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:11:51 EDT 2017 2017-05-11 11:11:51,771 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:11:51 EDT 2017 2017-05-11 11:13:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515400000, lagBy: 225 seconds. 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 11:10:00 EDT 2017, endTime = Thu May 11 11:12:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494515100000, lagBy: 525 seconds. 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Last check point time: 1494511200000, lagBy: 4425 seconds. 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:13:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494515100000, lagBy: 525 seconds. 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494511200000, lagBy: 4425 seconds. 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: 4308 row(s) updated. 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:13:52 EDT 2017 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:13:52 EDT 2017 2017-05-11 11:14:07,832 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:09,329 INFO TimelineClusterAggregatorSecond: Saving 17220 metric aggregates. 2017-05-11 11:14:17,839 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:21,889 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:24,531 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10522 actions to finish 2017-05-11 11:14:27,847 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:32,309 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:33,518 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:14:37,855 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:42,312 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:43,521 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:14:43,899 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:14:47,862 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:51,525 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:14:52,313 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:54,876 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:14:54,876 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:14:57,205 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:14:57,870 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:01,531 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:02,322 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:04,879 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:15:04,879 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:07,213 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:07,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:11,541 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:12,327 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:14,884 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:15:14,886 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:17,215 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:17,879 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:21,552 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:22,337 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:24,894 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:27,181 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 0 2017-05-11 11:15:27,221 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:27,887 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:31,553 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:32,339 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:34,895 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:36,450 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Time to save: 87121, thread = pool-6-thread-1 2017-05-11 11:15:36,451 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:15:36 EDT 2017 2017-05-11 11:15:36,451 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:15:36 EDT 2017 2017-05-11 11:15:38,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:38,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:15:46 EDT 2017 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:12:00 EDT 2017 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:12:00 EDT 2017 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515520000, lagBy: 226 seconds. 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:15:46 EDT 2017, startTime = Thu May 11 11:12:00 EDT 2017, endTime = Thu May 11 11:14:00 EDT 2017 2017-05-11 11:15:46,460 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:15:46,460 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:15:46,461 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:15:46,581 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:15:48,341 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:48,342 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:56,579 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:15:56,579 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:15:56,579 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:15:56,589 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:15:58,348 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:58,352 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:16:02,833 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:04,869 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:05,041 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:06,580 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:16:06,589 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:16:06,589 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:16:06,590 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:08,350 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:16:08,360 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:16:13,784 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:14,875 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:15,047 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:16,588 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:16:16,597 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:16,598 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:16:16,599 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:16:18,358 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:16:18,368 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: 81767 row(s) updated. 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:16:21 EDT 2017 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:16:21 EDT 2017 2017-05-11 11:16:23,805 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:24,883 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:25,054 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:26,604 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:16:26,604 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:26,605 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:16:32,687 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:16:33,808 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:16:33,815 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:34,885 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:35,065 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:35,299 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:16:36,613 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:42,696 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:16:43,813 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:16:43,823 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:44,598 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:16:45,072 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:45,303 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:16:46,623 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:52,705 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:16:53,815 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:16:53,885 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:16:54,601 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:16:55,078 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:55,312 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:16:56,624 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:17:02,712 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:17:03,175 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:03,822 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:17:03,890 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:04,605 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:05,079 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:05,321 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:17:06,629 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:17:08,855 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:12,716 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:17:13,181 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:13,826 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:17:13,896 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:14,611 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:15,087 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:15,326 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:17:16,639 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:17:18,866 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:22,720 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:17:23,183 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:23,828 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:17:23,898 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:24,615 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:25,094 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:25,327 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:17:25,639 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:28,871 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:33,187 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:33,906 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:34,623 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:35,103 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:35,640 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:38,881 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:40,570 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:17:40,570 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:17:43,196 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:43,365 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 0 2017-05-11 11:17:44,627 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:45,109 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:45,648 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:48,885 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:50,580 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:17:50,580 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:17:53,201 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:53,402 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:17:55,649 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:56,378 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2548 actions to finish 2017-05-11 11:18:00,582 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:00,583 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:03,202 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:03,403 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:04,466 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8686 actions to finish 2017-05-11 11:18:04,506 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:05,655 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:06,389 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2548 actions to finish 2017-05-11 11:18:10,585 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:10,585 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:13,208 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:13,409 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:14,145 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:14,477 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8686 actions to finish 2017-05-11 11:18:14,509 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:15,664 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:20,588 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:20,588 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:20,718 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:23,217 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:23,419 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:24,155 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:24,478 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8686 actions to finish 2017-05-11 11:18:24,513 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:25,665 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:29,619 INFO TimelineClusterAggregatorHourly: 4343 row(s) updated. 2017-05-11 11:18:29,620 INFO TimelineClusterAggregatorHourly: Aggregated cluster metrics for METRIC_AGGREGATE_HOURLY, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:18:29,620 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 11:18:29 EDT 2017 2017-05-11 11:18:29,620 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 11:18:29 EDT 2017 2017-05-11 11:18:31,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:31,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:31,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:33,218 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:33,429 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:34,156 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:34,519 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:35,674 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:41,339 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:41,339 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:43,228 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:43,366 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 1 2017-05-11 11:18:43,434 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:44,164 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:44,527 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:18:45 EDT 2017 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494515400000, lagBy: 525 seconds. 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:18:45 EDT 2017, startTime = Thu May 11 11:10:00 EDT 2017, endTime = Thu May 11 11:15:00 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:18:45 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494515400000, lagBy: 525 seconds. 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:18:45 EDT 2017, startTime = Thu May 11 11:10:00 EDT 2017, endTime = Thu May 11 11:15:00 EDT 2017 2017-05-11 11:18:45,547 ERROR TimelineMetricHostAggregatorHourly: Exception during aggregating metrics. java.sql.SQLTimeoutException: Operation timed out. at org.apache.phoenix.exception.SQLExceptionCode$14.newException(SQLExceptionCode.java:363) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:802) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:714) at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72) at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93) at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58) at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778) at org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:188) at org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:109) at org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:796) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:329) at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(PhoenixPreparedStatement.java:199) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.doWork(AbstractTimelineAggregator.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.runOnce(AbstractTimelineAggregator.java:110) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.run(AbstractTimelineAggregator.java:94) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-05-11 11:18:45,552 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 11:18:45 EDT 2017 2017-05-11 11:18:48,208 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:18:48,218 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:18:51,344 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:51,344 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:53,439 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:54,170 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:54,532 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:57,412 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:18:58,217 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:18:58,227 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:01,352 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:19:02,778 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:04,178 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:19:04,537 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:19:07,420 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:19:08,226 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:19:08,236 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:11,418 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 20230 actions to finish 2017-05-11 11:19:12,784 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:14,547 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:19:17,429 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:19:18,235 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:19:18,246 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:19,982 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6752 actions to finish 2017-05-11 11:19:21,421 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 20230 actions to finish 2017-05-11 11:19:22,791 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:24,553 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:19:25,822 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8322 actions to finish 2017-05-11 11:19:27,437 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:19:28,244 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:19:28,254 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:29,985 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6752 actions to finish 2017-05-11 11:19:31,425 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 20230 actions to finish 2017-05-11 11:19:32,798 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:34,090 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 2 2017-05-11 11:19:34,091 FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Error getting metrics from TimelineMetricStore. Shutting down by TimelineMetricStoreWatcher. 2017-05-11 11:19:34,092 INFO org.apache.hadoop.util.ExitUtil: Exiting with status -1 2017-05-11 11:19:34,102 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@:6188 2017-05-11 11:19:34,103 ERROR org.apache.phoenix.cache.ServerCacheClient: Error trying to remove hash cache for region=METRIC_RECORD,,1488665232951.aa0e4527400db4664d8a14735d897ef8., hostname=,44249,1494501076341, seqNum=3619070 java.io.InterruptedIOException: Interrupted calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService for row at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1742) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1692) at org.apache.phoenix.cache.ServerCacheClient.removeServerCache(ServerCacheClient.java:324) at org.apache.phoenix.cache.ServerCacheClient.access$000(ServerCacheClient.java:82) at org.apache.phoenix.cache.ServerCacheClient$ServerCache.close(ServerCacheClient.java:142) at org.apache.phoenix.util.SQLCloseables.closeAllQuietly(SQLCloseables.java:53) at org.apache.phoenix.util.SQLCloseables.closeAll(SQLCloseables.java:43) at org.apache.phoenix.execute.BaseQueryPlan$1.close(BaseQueryPlan.java:334) at org.apache.phoenix.jdbc.PhoenixResultSet.close(PhoenixResultSet.java:162) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getLatestMetricRecords(PhoenixHBaseAccessor.java:934) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getMetricRecords(PhoenixHBaseAccessor.java:776) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:359) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1736) ... 60 more 2017-05-11 11:19:34,103 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR javax.ws.rs.WebApplicationException: java.sql.SQLException: ERROR 2005 (INT07): Interrupted exception. Sub plan [0] execution interrupted. at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:377) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.sql.SQLException: ERROR 2005 (INT07): Interrupted exception. Sub plan [0] execution interrupted. at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:441) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:194) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:143) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:138) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:281) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:266) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:265) at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeQuery(PhoenixPreparedStatement.java:186) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getLatestMetricRecords(PhoenixHBaseAccessor.java:926) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getMetricRecords(PhoenixHBaseAccessor.java:776) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:359) ... 48 more Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:186) ... 59 more 2017-05-11 11:19:34,110 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR javax.ws.rs.WebApplicationException: org.apache.phoenix.exception.PhoenixIOException at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:377) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: org.apache.phoenix.exception.PhoenixIOException at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:808) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:714) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getLatestMetricRecords(PhoenixHBaseAccessor.java:927) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getMetricRecords(PhoenixHBaseAccessor.java:776) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:359) ... 48 more Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:204) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:766) ... 57 more 2017-05-11 11:19:34,128 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:19:36,209 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping phoenix metrics system... 2017-05-11 11:19:36,210 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system stopped. 2017-05-11 11:19:36,210 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system shutdown complete. 2017-05-11 11:19:36,211 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Stopping ApplicationHistory 2017-05-11 11:19:36,211 INFO org.apache.hadoop.ipc.Server: Stopping server on 60200 2017-05-11 11:19:36,213 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60200 2017-05-11 11:19:36,213 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2017-05-11 11:19:36,213 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer ************************************************************/ 2017-05-11 11:21:03,366 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting ApplicationHistoryServer STARTUP_MSG: user = ams STARTUP_MSG: host = STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.3.2.5.3.0-28 STARTUP_MSG: classpath = /usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9828acfdec41a121f0121f556b09e2d112259e92; compiled by 'jenkins' on 2016-11-21T23:57Z STARTUP_MSG: java = 1.8.0_77 ************************************************************/ 2017-05-11 11:21:03,372 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT] 2017-05-11 11:21:03,399 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found hbase site configuration: file:/etc/ambari-metrics-collector/conf/hbase-site.xml 2017-05-11 11:21:03,399 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found metric service configuration: file:/etc/ambari-metrics-collector/conf/ams-site.xml 2017-05-11 11:21:03,621 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Explicitly disabled application timeline store. 2017-05-11 11:21:03,622 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Creating metrics store. 2017-05-11 11:21:03,625 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: ApplicationHistory Init 2017-05-11 11:21:03,625 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Explicitly disabled application timeline store. 2017-05-11 11:21:03,858 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Initializing metrics schema... 2017-05-11 11:21:03,921 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,016 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,049 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,113 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,322 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5ddeb7cb connecting to ZooKeeper ensemble=:61181 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-28--1, built on 11/21/2016 23:52 GMT 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=str3 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_77 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_77/jre 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/ams-hbase/lib/hadoop-native 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ams 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ams 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/ams 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@5b799640 2017-05-11 11:21:04,348 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:21:04,355 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:21:04,361 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf8195bad0001, negotiated timeout = 120000 2017-05-11 11:21:04,424 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,569 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix 2017-05-11 11:21:04,621 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties 2017-05-11 11:21:04,694 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-05-11 11:21:04,694 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system started 2017-05-11 11:21:04,830 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:22:13,347 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68415 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,44249,1494501076341, seqNum=0 2017-05-11 11:22:33,537 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=88607 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,44249,1494501076341, seqNum=0 2017-05-11 11:22:53,577 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=108647 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,44249,1494501076341, seqNum=0 2017-05-11 11:23:14,008 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:15,755 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Metrics schema initialized. 2017-05-11 11:23:16,348 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 4842, metadata objects from store. 2017-05-11 11:23:16,380 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 50 host objects from store. 2017-05-11 11:23:16,398 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6134ac4a connecting to ZooKeeper ensemble=str3:61181 2017-05-11 11:23:16,398 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@777c9dc9 2017-05-11 11:23:16,400 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:23:16,400 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:23:16,402 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf8195bad0011, negotiated timeout = 120000 2017-05-11 11:23:16,418 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Using group by aggregators for aggregating host and cluster metrics. 2017-05-11 11:23:16,424 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.TimelineMetricAppAggregator: AppIds configured for aggregation: [datanode, nodemanager, hbase] 2017-05-11 11:23:16,431 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Started watchdog for timeline metrics store with initial delay = 600, delay = 30 2017-05-11 11:23:16,431 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system already initialized! 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineMetricHostAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,438 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Instantiating AHSWebApp at str3:6188 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 WARN TimelineMetricHostAggregatorMinute: Last Checkpoint is too old, discarding last checkpoint. lastCheckPointTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:12:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 WARN TimelineClusterAggregatorSecond: Last Checkpoint is too old, discarding last checkpoint. lastCheckPointTime = Thu May 11 11:12:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorHourly: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorMinute: Saving checkpoint time. Thu May 11 11:15:00 EDT 2017 2017-05-11 11:23:16,445 WARN TimelineClusterAggregatorMinute: Last Checkpoint is too old, discarding last checkpoint. lastCheckPointTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorMinute: Saving checkpoint time. Thu May 11 11:15:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorSecond: Saving checkpoint time. Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorMinute: Last check point time: 1494515700000, lagBy: 496 seconds. 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494515700000, lagBy: 496 seconds. 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494511200000, lagBy: 4996 seconds. 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorSecond: Last check point time: 1494516000000, lagBy: 196 seconds. 2017-05-11 11:23:16,447 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:16,447 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:22:00 EDT 2017 2017-05-11 11:23:16,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2017-05-11 11:23:16,541 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2017-05-11 11:23:16,547 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.applicationhistory is not defined 2017-05-11 11:23:16,555 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2017-05-11 11:23:16,558 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context applicationhistory 2017-05-11 11:23:16,559 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2017-05-11 11:23:16,559 INFO org.apache.hadoop.security.HttpCrossOriginFilterInitializer: CORS filter not enabled. Please set hadoop.http.cross-origin.enabled to 'true' to enable it 2017-05-11 11:23:16,563 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /applicationhistory/* 2017-05-11 11:23:16,563 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2017-05-11 11:23:16,948 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2017-05-11 11:23:16,954 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 6188 2017-05-11 11:23:16,954 INFO org.mortbay.log: jetty-6.1.26.hwx 2017-05-11 11:23:16,988 INFO org.mortbay.log: Extract jar:file:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar!/webapps/applicationhistory to /tmp/Jetty_str3_bi_grn_hcvlny_cv_net_6188_applicationhistory____3kd262/webapp 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: 0 row(s) updated. 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:18 EDT 2017 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:18 EDT 2017 2017-05-11 11:23:18,342 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@str3:6188 2017-05-11 11:23:18,342 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app applicationhistory started at 6188 2017-05-11 11:23:18,432 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2017-05-11 11:23:18,447 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 60200 2017-05-11 11:23:18,488 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ApplicationHistoryProtocolPB to the server 2017-05-11 11:23:18,489 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60200: starting 2017-05-11 11:23:18,489 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2017-05-11 11:23:18,499 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService: Instantiated ApplicationHistoryClientService at str3/.70:60200 2017-05-11 11:23:18,499 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Starting ApplicationHistory 2017-05-11 11:23:20,242 INFO TimelineClusterAggregatorSecond: Saving 15664 metric aggregates. 2017-05-11 11:23:23,410 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:23:23 EDT 2017 2017-05-11 11:23:23,411 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:23:23 EDT 2017 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: 81770 row(s) updated. 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:24 EDT 2017 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:24 EDT 2017 2017-05-11 11:23:28,091 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: RECEIVED SIGNAL 15: SIGTERM 2017-05-11 11:23:28,098 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@str3:6188 2017-05-11 11:23:28,117 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:28,199 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping phoenix metrics system... 2017-05-11 11:23:28,200 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system stopped. 2017-05-11 11:23:28,200 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system shutdown complete. 2017-05-11 11:23:28,200 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Stopping ApplicationHistory 2017-05-11 11:23:28,200 INFO org.apache.hadoop.ipc.Server: Stopping server on 60200 2017-05-11 11:23:28,201 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60200 2017-05-11 11:23:28,201 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2017-05-11 11:23:28,202 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at str3/.70 ************************************************************/ 2017-05-11 11:23:43,719 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting ApplicationHistoryServer STARTUP_MSG: user = ams STARTUP_MSG: host = str3/.70 STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.3.2.5.3.0-28 STARTUP_MSG: classpath = /usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9828acfdec41a121f0121f556b09e2d112259e92; compiled by 'jenkins' on 2016-11-21T23:57Z STARTUP_MSG: java = 1.8.0_77 ************************************************************/ 2017-05-11 11:23:43,727 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT] 2017-05-11 11:23:43,757 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found hbase site configuration: file:/etc/ambari-metrics-collector/conf/hbase-site.xml 2017-05-11 11:23:43,757 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found metric service configuration: file:/etc/ambari-metrics-collector/conf/ams-site.xml 2017-05-11 11:23:43,976 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Explicitly disabled application timeline store. 2017-05-11 11:23:43,978 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Creating metrics store. 2017-05-11 11:23:43,981 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: ApplicationHistory Init 2017-05-11 11:23:43,981 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Explicitly disabled application timeline store. 2017-05-11 11:23:44,222 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Initializing metrics schema... 2017-05-11 11:23:44,284 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,381 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,414 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,509 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,707 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5ddeb7cb connecting to ZooKeeper ensemble=str3:61181 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-28--1, built on 11/21/2016 23:52 GMT 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=str3 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_77 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_77/jre 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/ams-hbase/lib/hadoop-native 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ams 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ams 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/ams 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@5b799640 2017-05-11 11:23:44,732 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:23:44,738 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:23:44,744 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf81bce100001, negotiated timeout = 120000 2017-05-11 11:23:44,802 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,934 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix 2017-05-11 11:23:44,980 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties 2017-05-11 11:23:45,048 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-05-11 11:23:45,048 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system started 2017-05-11 11:23:45,179 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:24:53,597 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68321 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,40557,1494516064903, seqNum=0 2017-05-11 11:25:13,663 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=88388 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,40557,1494516064903, seqNum=0 2017-05-11 11:25:33,708 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=108433 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,40557,1494516064903, seqNum=0 2017-05-11 11:25:54,138 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:25:58,934 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Metrics schema initialized. 2017-05-11 11:25:59,481 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 4842, metadata objects from store. 2017-05-11 11:25:59,514 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 50 host objects from store. 2017-05-11 11:25:59,531 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6134ac4a connecting to ZooKeeper ensemble=str3:61181 2017-05-11 11:25:59,532 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@777c9dc9 2017-05-11 11:25:59,533 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:25:59,534 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:25:59,535 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf81bce100011, negotiated timeout = 120000 2017-05-11 11:25:59,552 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Using group by aggregators for aggregating host and cluster metrics. 2017-05-11 11:25:59,561 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.TimelineMetricAppAggregator: AppIds configured for aggregation: [datanode, nodemanager, hbase] 2017-05-11 11:25:59,570 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Started watchdog for timeline metrics store with initial delay = 600, delay = 30 2017-05-11 11:25:59,570 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system already initialized! 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineMetricHostAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,581 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Instantiating AHSWebApp at str3:6188 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:22:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:22:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorSecond: Last check point time: 1494516120000, lagBy: 239 seconds. 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 11:22:00 EDT 2017, endTime = Thu May 11 11:24:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorMinute: Last check point time: 1494516000000, lagBy: 359 seconds. 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494516000000, lagBy: 359 seconds. 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorHourly: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494511200000, lagBy: 5159 seconds. 2017-05-11 11:25:59,585 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:25:59,683 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2017-05-11 11:25:59,694 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2017-05-11 11:25:59,699 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.applicationhistory is not defined 2017-05-11 11:25:59,705 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2017-05-11 11:25:59,707 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context applicationhistory 2017-05-11 11:25:59,708 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2017-05-11 11:25:59,708 INFO org.apache.hadoop.security.HttpCrossOriginFilterInitializer: CORS filter not enabled. Please set hadoop.http.cross-origin.enabled to 'true' to enable it 2017-05-11 11:25:59,711 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /applicationhistory/* 2017-05-11 11:25:59,711 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2017-05-11 11:26:00,145 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2017-05-11 11:26:00,153 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 6188 2017-05-11 11:26:00,154 INFO org.mortbay.log: jetty-6.1.26.hwx 2017-05-11 11:26:00,203 INFO org.mortbay.log: Extract jar:file:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar!/webapps/applicationhistory to /tmp/Jetty_str3_bi_grn_hcvlny_cv_net_6188_applicationhistory____3kd262/webapp 2017-05-11 11:26:00,687 INFO TimelineClusterAggregatorSecond: Saving 7552 metric aggregates. 2017-05-11 11:26:02,018 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@str3:6188 2017-05-11 11:26:02,018 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app applicationhistory started at 6188 2017-05-11 11:26:02,100 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2017-05-11 11:26:02,112 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 60200 2017-05-11 11:26:02,137 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ApplicationHistoryProtocolPB to the server 2017-05-11 11:26:02,138 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60200: starting 2017-05-11 11:26:02,138 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2017-05-11 11:26:02,148 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService: Instantiated ApplicationHistoryClientService at str3/.70:60200 2017-05-11 11:26:02,149 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Starting ApplicationHistory 2017-05-11 11:26:02,253 INFO TimelineMetricHostAggregatorMinute: 3677 row(s) updated. 2017-05-11 11:26:02,253 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:26:02,253 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,254 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,256 INFO TimelineClusterAggregatorMinute: 3916 row(s) updated. 2017-05-11 11:26:02,257 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:26:02,257 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,257 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,348 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,349 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: 81808 row(s) updated. 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: Aggregated host metrics for METRIC_RECORD_HOURLY, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 11:26:10 EDT 2017 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 11:26:10 EDT 2017

3 REPLIES 3

Cloudera Employee

Hi @PJ

- Please verify your AMS heap settings are configured properly as per the below documentation.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-operations/content/ams_general_gu...

- Do a cleanup on "hbase.tmp.dir".

a) Shutdown AMS.

b) Move "hbase.tmp.dir".

c) Start AMS

Please monitor AMS and check if you are getting same exceptions.

Did you enable HBase metrics?

Expert Contributor

I just increased ambari metricsheapsize from 756MB to 4 GB.No issues since then ...

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.