Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari metrics collector is shutting down abruptly

Ambari metrics collector is shutting down abruptly

Expert Contributor

Hi can somebody help me with this issue i got all of a sudden this morning. I have ambari metrics collector shut down abruptly twice at random times, but when i start it, it starts succesfully ..

logs are: 2017-05-11 10:07:52,213 INFO TimelineClusterAggregatorSecond: Saving 17188 metric aggregates. 2017-05-11 10:07:53,376 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:07:53 EDT 2017 2017-05-11 10:07:53,376 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:07:53 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:08:45 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494511200000, lagBy: 525 seconds. 2017-05-11 10:08:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:08:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:08:45 EDT 2017 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494511200000, lagBy: 525 seconds. 2017-05-11 10:08:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:08:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: 4302 row(s) updated. 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:49 EDT 2017 2017-05-11 10:08:49,194 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:49 EDT 2017 2017-05-11 10:08:57,344 INFO TimelineMetricHostAggregatorMinute: 81766 row(s) updated. 2017-05-11 10:08:57,344 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 10:05:00 EDT 2017 2017-05-11 10:08:57,344 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:57 EDT 2017 2017-05-11 10:08:57,345 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:08:57 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:09:45 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:06:00 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:06:00 EDT 2017 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511560000, lagBy: 225 seconds. 2017-05-11 10:09:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:09:45 EDT 2017, startTime = Thu May 11 10:06:00 EDT 2017, endTime = Thu May 11 10:08:00 EDT 2017 2017-05-11 10:09:50,585 INFO TimelineClusterAggregatorSecond: Saving 17156 metric aggregates. 2017-05-11 10:09:53,912 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:09:53 EDT 2017 2017-05-11 10:09:53,912 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:09:53 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:11:45 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:08:00 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:08:00 EDT 2017 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511680000, lagBy: 225 seconds. 2017-05-11 10:11:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:11:45 EDT 2017, startTime = Thu May 11 10:08:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:11:50,076 INFO TimelineClusterAggregatorSecond: Saving 17192 metric aggregates. 2017-05-11 10:11:51,296 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:11:51 EDT 2017 2017-05-11 10:11:51,297 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:11:51 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511800000, lagBy: 225 seconds. 2017-05-11 10:13:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:12:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494511500000, lagBy: 525 seconds. 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,522 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,523 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,523 INFO TimelineClusterAggregatorHourly: Last check point time: 1494507600000, lagBy: 4425 seconds. 2017-05-11 10:13:45,523 INFO TimelineClusterAggregatorHourly: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:13:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:05:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494511500000, lagBy: 525 seconds. 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 10:13:45 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 09:00:00 EDT 2017 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494507600000, lagBy: 4425 seconds. 2017-05-11 10:13:45,533 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 10:13:45 EDT 2017, startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:13:51,706 INFO TimelineClusterAggregatorMinute: 4298 row(s) updated. 2017-05-11 10:13:51,706 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:13:51,707 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:13:51 EDT 2017 2017-05-11 10:13:51,707 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:13:51 EDT 2017 2017-05-11 10:13:58,283 INFO TimelineClusterAggregatorSecond: Saving 17144 metric aggregates. 2017-05-11 10:14:05,778 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:10,223 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34288 actions to finish 2017-05-11 10:14:15,780 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:16,139 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11374 actions to finish 2017-05-11 10:14:20,229 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34288 actions to finish 2017-05-11 10:14:25,781 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:28,370 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6402 actions to finish 2017-05-11 10:14:30,236 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34288 actions to finish 2017-05-11 10:14:34,706 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6540 actions to finish 2017-05-11 10:14:35,784 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:38,860 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:14:38 EDT 2017 2017-05-11 10:14:38,860 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:14:38 EDT 2017 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: 4336 row(s) updated. 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: Aggregated cluster metrics for METRIC_AGGREGATE_HOURLY, with startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 10:14:44 EDT 2017 2017-05-11 10:14:44,307 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 10:14:44 EDT 2017 2017-05-11 10:14:45,794 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:48,857 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10120 actions to finish 2017-05-11 10:14:54,291 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 5358 actions to finish 2017-05-11 10:14:55,795 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:14:57,838 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21566 actions to finish 2017-05-11 10:15:05,798 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83514 actions to finish 2017-05-11 10:15:07,007 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 12896 actions to finish 2017-05-11 10:15:07,843 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21566 actions to finish 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: 81757 row(s) updated. 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:05:00 EDT 2017, endTime = Thu May 11 10:10:00 EDT 2017 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:15:09 EDT 2017 2017-05-11 10:15:09,786 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:15:09 EDT 2017 2017-05-11 10:15:11,125 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2016 actions to finish 2017-05-11 10:15:23,315 INFO TimelineMetricHostAggregatorHourly: 81802 row(s) updated. 2017-05-11 10:15:23,315 INFO TimelineMetricHostAggregatorHourly: Aggregated host metrics for METRIC_RECORD_HOURLY, with startTime = Thu May 11 09:00:00 EDT 2017, endTime = Thu May 11 10:00:00 EDT 2017 2017-05-11 10:15:23,315 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 10:15:23 EDT 2017 2017-05-11 10:15:23,316 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 10:15:23 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:15:45 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:12:00 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:12:00 EDT 2017 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494511920000, lagBy: 225 seconds. 2017-05-11 10:15:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:15:45 EDT 2017, startTime = Thu May 11 10:12:00 EDT 2017, endTime = Thu May 11 10:14:00 EDT 2017 2017-05-11 10:15:50,093 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:15:51,712 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:15:51 EDT 2017 2017-05-11 10:15:51,712 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:15:51 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:17:45 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:14:00 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:14:00 EDT 2017 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512040000, lagBy: 225 seconds. 2017-05-11 10:17:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:17:45 EDT 2017, startTime = Thu May 11 10:14:00 EDT 2017, endTime = Thu May 11 10:16:00 EDT 2017 2017-05-11 10:17:50,212 INFO TimelineClusterAggregatorSecond: Saving 17196 metric aggregates. 2017-05-11 10:17:51,316 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:17:51 EDT 2017 2017-05-11 10:17:51,316 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:17:51 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:18:45 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494511800000, lagBy: 525 seconds. 2017-05-11 10:18:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:18:45 EDT 2017, startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:18:45 EDT 2017 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:10:00 EDT 2017 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494511800000, lagBy: 525 seconds. 2017-05-11 10:18:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:18:45 EDT 2017, startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: 4302 row(s) updated. 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:46 EDT 2017 2017-05-11 10:18:46,873 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:46 EDT 2017 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: 81753 row(s) updated. 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:10:00 EDT 2017, endTime = Thu May 11 10:15:00 EDT 2017 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:54 EDT 2017 2017-05-11 10:18:54,790 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:18:54 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:19:45 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:16:00 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:16:00 EDT 2017 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512160000, lagBy: 225 seconds. 2017-05-11 10:19:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:19:45 EDT 2017, startTime = Thu May 11 10:16:00 EDT 2017, endTime = Thu May 11 10:18:00 EDT 2017 2017-05-11 10:19:54,032 INFO TimelineClusterAggregatorSecond: Saving 17184 metric aggregates. 2017-05-11 10:19:55,196 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:19:55 EDT 2017 2017-05-11 10:19:55,196 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:19:55 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:21:45 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:18:00 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:18:00 EDT 2017 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512280000, lagBy: 225 seconds. 2017-05-11 10:21:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:21:45 EDT 2017, startTime = Thu May 11 10:18:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:21:49,978 INFO TimelineClusterAggregatorSecond: Saving 17252 metric aggregates. 2017-05-11 10:21:53,438 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:21:53 EDT 2017 2017-05-11 10:21:53,438 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:21:53 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:23:45 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512400000, lagBy: 225 seconds. 2017-05-11 10:23:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:23:45 EDT 2017, startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:22:00 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:23:45 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494512100000, lagBy: 525 seconds. 2017-05-11 10:23:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:23:45 EDT 2017, startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:23:45 EDT 2017 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:15:00 EDT 2017 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494512100000, lagBy: 525 seconds. 2017-05-11 10:23:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:23:45 EDT 2017, startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:49,249 INFO TimelineClusterAggregatorMinute: 4322 row(s) updated. 2017-05-11 10:23:49,250 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:23:49,250 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:23:49 EDT 2017 2017-05-11 10:23:49,250 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:23:49 EDT 2017 2017-05-11 10:23:52,849 INFO TimelineClusterAggregatorSecond: Saving 17148 metric aggregates. 2017-05-11 10:24:00,893 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83562 actions to finish 2017-05-11 10:24:04,533 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34296 actions to finish 2017-05-11 10:24:04,555 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:24:04 EDT 2017 2017-05-11 10:24:04,555 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:24:04 EDT 2017 2017-05-11 10:24:06,617 INFO TimelineMetricHostAggregatorMinute: 81781 row(s) updated. 2017-05-11 10:24:06,617 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:15:00 EDT 2017, endTime = Thu May 11 10:20:00 EDT 2017 2017-05-11 10:24:06,618 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:24:06 EDT 2017 2017-05-11 10:24:06,618 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:24:06 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:25:45 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:22:00 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:22:00 EDT 2017 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512520000, lagBy: 225 seconds. 2017-05-11 10:25:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:25:45 EDT 2017, startTime = Thu May 11 10:22:00 EDT 2017, endTime = Thu May 11 10:24:00 EDT 2017 2017-05-11 10:25:52,140 INFO TimelineClusterAggregatorSecond: Saving 17200 metric aggregates. 2017-05-11 10:25:55,716 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:25:55 EDT 2017 2017-05-11 10:25:55,716 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:25:55 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:27:45 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:24:00 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:24:00 EDT 2017 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512640000, lagBy: 225 seconds. 2017-05-11 10:27:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:27:45 EDT 2017, startTime = Thu May 11 10:24:00 EDT 2017, endTime = Thu May 11 10:26:00 EDT 2017 2017-05-11 10:27:52,191 INFO TimelineClusterAggregatorSecond: Saving 17184 metric aggregates. 2017-05-11 10:27:53,496 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:27:53 EDT 2017 2017-05-11 10:27:53,496 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:27:53 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:28:45 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494512400000, lagBy: 525 seconds. 2017-05-11 10:28:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:28:45 EDT 2017, startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:28:45 EDT 2017 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:20:00 EDT 2017 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494512400000, lagBy: 525 seconds. 2017-05-11 10:28:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:28:45 EDT 2017, startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: 4301 row(s) updated. 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:47 EDT 2017 2017-05-11 10:28:47,021 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:47 EDT 2017 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: 81760 row(s) updated. 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:20:00 EDT 2017, endTime = Thu May 11 10:25:00 EDT 2017 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:54 EDT 2017 2017-05-11 10:28:54,604 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:28:54 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:29:45 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:26:00 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:26:00 EDT 2017 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512760000, lagBy: 225 seconds. 2017-05-11 10:29:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:29:45 EDT 2017, startTime = Thu May 11 10:26:00 EDT 2017, endTime = Thu May 11 10:28:00 EDT 2017 2017-05-11 10:29:50,043 INFO TimelineClusterAggregatorSecond: Saving 17144 metric aggregates. 2017-05-11 10:29:53,349 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:29:53 EDT 2017 2017-05-11 10:29:53,349 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:29:53 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:31:45 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:28:00 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:28:00 EDT 2017 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494512880000, lagBy: 225 seconds. 2017-05-11 10:31:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:31:45 EDT 2017, startTime = Thu May 11 10:28:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:31:50,293 INFO TimelineClusterAggregatorSecond: Saving 17160 metric aggregates. 2017-05-11 10:31:51,293 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:31:51 EDT 2017 2017-05-11 10:31:51,294 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:31:51 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:33:45 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513000000, lagBy: 225 seconds. 2017-05-11 10:33:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:33:45 EDT 2017, startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:32:00 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:33:45 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494512700000, lagBy: 525 seconds. 2017-05-11 10:33:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:33:45 EDT 2017, startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:33:45 EDT 2017 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:25:00 EDT 2017 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494512700000, lagBy: 525 seconds. 2017-05-11 10:33:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:33:45 EDT 2017, startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:49,528 INFO TimelineClusterAggregatorMinute: 4301 row(s) updated. 2017-05-11 10:33:49,528 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:33:49,529 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:33:49 EDT 2017 2017-05-11 10:33:49,529 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:33:49 EDT 2017 2017-05-11 10:33:55,444 INFO TimelineClusterAggregatorSecond: Saving 17088 metric aggregates. 2017-05-11 10:34:03,179 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83520 actions to finish 2017-05-11 10:34:08,200 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34176 actions to finish 2017-05-11 10:34:08,200 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22106 actions to finish 2017-05-11 10:34:08,225 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:34:08 EDT 2017 2017-05-11 10:34:08,225 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:34:08 EDT 2017 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: 81760 row(s) updated. 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:25:00 EDT 2017, endTime = Thu May 11 10:30:00 EDT 2017 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:34:10 EDT 2017 2017-05-11 10:34:10,153 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:34:10 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:35:45 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:32:00 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:32:00 EDT 2017 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513120000, lagBy: 225 seconds. 2017-05-11 10:35:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:35:45 EDT 2017, startTime = Thu May 11 10:32:00 EDT 2017, endTime = Thu May 11 10:34:00 EDT 2017 2017-05-11 10:35:50,394 INFO TimelineClusterAggregatorSecond: Saving 17176 metric aggregates. 2017-05-11 10:35:53,974 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:35:53 EDT 2017 2017-05-11 10:35:53,974 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:35:53 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:37:45 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:34:00 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:34:00 EDT 2017 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513240000, lagBy: 225 seconds. 2017-05-11 10:37:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:37:45 EDT 2017, startTime = Thu May 11 10:34:00 EDT 2017, endTime = Thu May 11 10:36:00 EDT 2017 2017-05-11 10:37:50,219 INFO TimelineClusterAggregatorSecond: Saving 17176 metric aggregates. 2017-05-11 10:37:53,999 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:37:53 EDT 2017 2017-05-11 10:37:53,999 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:37:53 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:38:45 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513000000, lagBy: 525 seconds. 2017-05-11 10:38:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:38:45 EDT 2017, startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:38:45 EDT 2017 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:30:00 EDT 2017 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513000000, lagBy: 525 seconds. 2017-05-11 10:38:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:38:45 EDT 2017, startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: 4298 row(s) updated. 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:47 EDT 2017 2017-05-11 10:38:47,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:47 EDT 2017 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: 81753 row(s) updated. 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:30:00 EDT 2017, endTime = Thu May 11 10:35:00 EDT 2017 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:52 EDT 2017 2017-05-11 10:38:52,726 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:38:52 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:39:45 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:36:00 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:36:00 EDT 2017 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513360000, lagBy: 225 seconds. 2017-05-11 10:39:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:39:45 EDT 2017, startTime = Thu May 11 10:36:00 EDT 2017, endTime = Thu May 11 10:38:00 EDT 2017 2017-05-11 10:39:49,836 INFO TimelineClusterAggregatorSecond: Saving 17140 metric aggregates. 2017-05-11 10:39:50,902 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:39:50 EDT 2017 2017-05-11 10:39:50,902 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:39:50 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:41:45 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:38:00 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:38:00 EDT 2017 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513480000, lagBy: 225 seconds. 2017-05-11 10:41:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:41:45 EDT 2017, startTime = Thu May 11 10:38:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:41:50,617 INFO TimelineClusterAggregatorSecond: Saving 17224 metric aggregates. 2017-05-11 10:41:51,775 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:41:51 EDT 2017 2017-05-11 10:41:51,775 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:41:51 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:43:45 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513600000, lagBy: 225 seconds. 2017-05-11 10:43:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:43:45 EDT 2017, startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:42:00 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:43:45 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513300000, lagBy: 525 seconds. 2017-05-11 10:43:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:43:45 EDT 2017, startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:43:45 EDT 2017 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:35:00 EDT 2017 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513300000, lagBy: 525 seconds. 2017-05-11 10:43:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:43:45 EDT 2017, startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: 4309 row(s) updated. 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:43:49 EDT 2017 2017-05-11 10:43:49,172 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:43:49 EDT 2017 2017-05-11 10:43:54,791 INFO TimelineClusterAggregatorSecond: Saving 17128 metric aggregates. 2017-05-11 10:44:01,487 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:44:01 EDT 2017 2017-05-11 10:44:01,488 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:44:01 EDT 2017 2017-05-11 10:44:02,684 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83536 actions to finish 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: 81768 row(s) updated. 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:35:00 EDT 2017, endTime = Thu May 11 10:40:00 EDT 2017 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:44:04 EDT 2017 2017-05-11 10:44:04,979 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:44:04 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:45:45 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:42:00 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:42:00 EDT 2017 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513720000, lagBy: 225 seconds. 2017-05-11 10:45:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:45:45 EDT 2017, startTime = Thu May 11 10:42:00 EDT 2017, endTime = Thu May 11 10:44:00 EDT 2017 2017-05-11 10:45:50,424 INFO TimelineClusterAggregatorSecond: Saving 17164 metric aggregates. 2017-05-11 10:45:51,606 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:45:51 EDT 2017 2017-05-11 10:45:51,606 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:45:51 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:47:45 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:44:00 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:44:00 EDT 2017 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513840000, lagBy: 225 seconds. 2017-05-11 10:47:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:47:45 EDT 2017, startTime = Thu May 11 10:44:00 EDT 2017, endTime = Thu May 11 10:46:00 EDT 2017 2017-05-11 10:47:50,155 INFO TimelineClusterAggregatorSecond: Saving 17164 metric aggregates. 2017-05-11 10:47:51,406 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:47:51 EDT 2017 2017-05-11 10:47:51,406 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:47:51 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:48:45 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513600000, lagBy: 525 seconds. 2017-05-11 10:48:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:48:45 EDT 2017, startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:48:45 EDT 2017 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:40:00 EDT 2017 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513600000, lagBy: 525 seconds. 2017-05-11 10:48:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:48:45 EDT 2017, startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: 4294 row(s) updated. 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:48 EDT 2017 2017-05-11 10:48:48,754 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:48 EDT 2017 2017-05-11 10:48:54,171 INFO TimelineMetricHostAggregatorMinute: 81750 row(s) updated. 2017-05-11 10:48:54,171 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:40:00 EDT 2017, endTime = Thu May 11 10:45:00 EDT 2017 2017-05-11 10:48:54,172 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:54 EDT 2017 2017-05-11 10:48:54,172 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:48:54 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:49:45 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:46:00 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:46:00 EDT 2017 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494513960000, lagBy: 225 seconds. 2017-05-11 10:49:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:49:45 EDT 2017, startTime = Thu May 11 10:46:00 EDT 2017, endTime = Thu May 11 10:48:00 EDT 2017 2017-05-11 10:49:49,760 INFO TimelineClusterAggregatorSecond: Saving 17148 metric aggregates. 2017-05-11 10:49:53,353 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:49:53 EDT 2017 2017-05-11 10:49:53,353 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:49:53 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:51:45 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:48:00 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:48:00 EDT 2017 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514080000, lagBy: 225 seconds. 2017-05-11 10:51:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:51:45 EDT 2017, startTime = Thu May 11 10:48:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:51:50,505 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:51:51,622 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:51:51 EDT 2017 2017-05-11 10:51:51,622 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:51:51 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:53:45 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514200000, lagBy: 225 seconds. 2017-05-11 10:53:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:53:45 EDT 2017, startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:52:00 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:53:45 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494513900000, lagBy: 525 seconds. 2017-05-11 10:53:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:53:45 EDT 2017, startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:53:45 EDT 2017 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:45:00 EDT 2017 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494513900000, lagBy: 525 seconds. 2017-05-11 10:53:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:53:45 EDT 2017, startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: 4306 row(s) updated. 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:53:49 EDT 2017 2017-05-11 10:53:49,465 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:53:49 EDT 2017 2017-05-11 10:53:53,442 INFO TimelineClusterAggregatorSecond: Saving 17104 metric aggregates. 2017-05-11 10:54:01,116 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83530 actions to finish 2017-05-11 10:54:04,123 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:54:04 EDT 2017 2017-05-11 10:54:04,123 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:54:04 EDT 2017 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: 81765 row(s) updated. 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:45:00 EDT 2017, endTime = Thu May 11 10:50:00 EDT 2017 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:54:07 EDT 2017 2017-05-11 10:54:07,375 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:54:07 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:55:45 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:52:00 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:52:00 EDT 2017 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514320000, lagBy: 225 seconds. 2017-05-11 10:55:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:55:45 EDT 2017, startTime = Thu May 11 10:52:00 EDT 2017, endTime = Thu May 11 10:54:00 EDT 2017 2017-05-11 10:55:50,403 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:55:51,472 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:55:51 EDT 2017 2017-05-11 10:55:51,473 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:55:51 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:57:45 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:54:00 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:54:00 EDT 2017 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514440000, lagBy: 225 seconds. 2017-05-11 10:57:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:57:45 EDT 2017, startTime = Thu May 11 10:54:00 EDT 2017, endTime = Thu May 11 10:56:00 EDT 2017 2017-05-11 10:57:50,176 INFO TimelineClusterAggregatorSecond: Saving 17172 metric aggregates. 2017-05-11 10:57:55,360 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:57:55 EDT 2017 2017-05-11 10:57:55,360 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:57:55 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:58:45 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494514200000, lagBy: 525 seconds. 2017-05-11 10:58:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 10:58:45 EDT 2017, startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 10:58:45 EDT 2017 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:50:00 EDT 2017 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494514200000, lagBy: 525 seconds. 2017-05-11 10:58:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 10:58:45 EDT 2017, startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:47,112 INFO TimelineClusterAggregatorMinute: 4293 row(s) updated. 2017-05-11 10:58:47,113 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:47,113 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:47 EDT 2017 2017-05-11 10:58:47,113 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:47 EDT 2017 2017-05-11 10:58:55,460 INFO TimelineMetricHostAggregatorMinute: 81746 row(s) updated. 2017-05-11 10:58:55,461 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:50:00 EDT 2017, endTime = Thu May 11 10:55:00 EDT 2017 2017-05-11 10:58:55,461 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:55 EDT 2017 2017-05-11 10:58:55,461 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 10:58:55 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 10:59:45 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:56:00 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:56:00 EDT 2017 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514560000, lagBy: 225 seconds. 2017-05-11 10:59:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 10:59:45 EDT 2017, startTime = Thu May 11 10:56:00 EDT 2017, endTime = Thu May 11 10:58:00 EDT 2017 2017-05-11 10:59:50,395 INFO TimelineClusterAggregatorSecond: Saving 17128 metric aggregates. 2017-05-11 10:59:51,533 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:59:51 EDT 2017 2017-05-11 10:59:51,533 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 10:59:51 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:01:45 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 10:58:00 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 10:58:00 EDT 2017 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514680000, lagBy: 225 seconds. 2017-05-11 11:01:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:01:45 EDT 2017, startTime = Thu May 11 10:58:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:01:50,245 INFO TimelineClusterAggregatorSecond: Saving 17256 metric aggregates. 2017-05-11 11:01:51,528 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:01:51 EDT 2017 2017-05-11 11:01:51,528 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:01:51 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:03:45 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514800000, lagBy: 225 seconds. 2017-05-11 11:03:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:03:45 EDT 2017, startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:02:00 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:03:45 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494514500000, lagBy: 525 seconds. 2017-05-11 11:03:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:03:45 EDT 2017, startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:03:45 EDT 2017 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 10:55:00 EDT 2017 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494514500000, lagBy: 525 seconds. 2017-05-11 11:03:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:03:45 EDT 2017, startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: 4316 row(s) updated. 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:03:49 EDT 2017 2017-05-11 11:03:49,293 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:03:49 EDT 2017 2017-05-11 11:03:55,333 INFO TimelineClusterAggregatorSecond: Saving 17252 metric aggregates. 2017-05-11 11:04:03,165 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83546 actions to finish 2017-05-11 11:04:06,437 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:04:06 EDT 2017 2017-05-11 11:04:06,437 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:04:06 EDT 2017 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: 81773 row(s) updated. 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 10:55:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:04:09 EDT 2017 2017-05-11 11:04:09,780 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:04:09 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:05:45 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:02:00 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:02:00 EDT 2017 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494514920000, lagBy: 225 seconds. 2017-05-11 11:05:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:05:45 EDT 2017, startTime = Thu May 11 11:02:00 EDT 2017, endTime = Thu May 11 11:04:00 EDT 2017 2017-05-11 11:05:50,025 INFO TimelineClusterAggregatorSecond: Saving 17212 metric aggregates. 2017-05-11 11:05:51,347 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:05:51 EDT 2017 2017-05-11 11:05:51,347 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:05:51 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:07:45 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:04:00 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:04:00 EDT 2017 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515040000, lagBy: 225 seconds. 2017-05-11 11:07:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:07:45 EDT 2017, startTime = Thu May 11 11:04:00 EDT 2017, endTime = Thu May 11 11:06:00 EDT 2017 2017-05-11 11:07:52,400 INFO TimelineClusterAggregatorSecond: Saving 17184 metric aggregates. 2017-05-11 11:07:53,665 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:07:53 EDT 2017 2017-05-11 11:07:53,665 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:07:53 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:08:45 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494514800000, lagBy: 525 seconds. 2017-05-11 11:08:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:08:45 EDT 2017, startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:08:45 EDT 2017 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494514800000, lagBy: 525 seconds. 2017-05-11 11:08:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:08:45 EDT 2017, startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: 4319 row(s) updated. 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:47 EDT 2017 2017-05-11 11:08:47,038 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:47 EDT 2017 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: 81768 row(s) updated. 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:00:00 EDT 2017, endTime = Thu May 11 11:05:00 EDT 2017 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:52 EDT 2017 2017-05-11 11:08:52,736 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:08:52 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:09:45 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:06:00 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:06:00 EDT 2017 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515160000, lagBy: 225 seconds. 2017-05-11 11:09:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:09:45 EDT 2017, startTime = Thu May 11 11:06:00 EDT 2017, endTime = Thu May 11 11:08:00 EDT 2017 2017-05-11 11:09:49,962 INFO TimelineClusterAggregatorSecond: Saving 17148 metric aggregates. 2017-05-11 11:09:54,032 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:09:54 EDT 2017 2017-05-11 11:09:54,032 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:09:54 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:11:45 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:08:00 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:08:00 EDT 2017 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515280000, lagBy: 225 seconds. 2017-05-11 11:11:45,521 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:11:45 EDT 2017, startTime = Thu May 11 11:08:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:11:50,670 INFO TimelineClusterAggregatorSecond: Saving 17220 metric aggregates. 2017-05-11 11:11:51,771 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:11:51 EDT 2017 2017-05-11 11:11:51,771 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:11:51 EDT 2017 2017-05-11 11:13:45,521 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,521 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515400000, lagBy: 225 seconds. 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 11:10:00 EDT 2017, endTime = Thu May 11 11:12:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494515100000, lagBy: 525 seconds. 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Last check point time: 1494511200000, lagBy: 4425 seconds. 2017-05-11 11:13:45,522 INFO TimelineClusterAggregatorHourly: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:13:45,532 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:05:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494515100000, lagBy: 525 seconds. 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:13:45 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494511200000, lagBy: 4425 seconds. 2017-05-11 11:13:45,533 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 11:13:45 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: 4308 row(s) updated. 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:13:52 EDT 2017 2017-05-11 11:13:52,015 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:13:52 EDT 2017 2017-05-11 11:14:07,832 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:09,329 INFO TimelineClusterAggregatorSecond: Saving 17220 metric aggregates. 2017-05-11 11:14:17,839 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:21,889 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:24,531 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10522 actions to finish 2017-05-11 11:14:27,847 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:32,309 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:33,518 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:14:37,855 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:42,312 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:43,521 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:14:43,899 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:14:47,862 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:14:51,525 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:14:52,313 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:14:54,876 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:14:54,876 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:14:57,205 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:14:57,870 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:01,531 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:02,322 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:04,879 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:15:04,879 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:07,213 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:07,877 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:11,541 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:12,327 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:14,884 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10586 actions to finish 2017-05-11 11:15:14,886 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:17,215 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:17,879 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:21,552 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:22,337 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:24,894 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:27,181 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 0 2017-05-11 11:15:27,221 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:27,887 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:31,553 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7328 actions to finish 2017-05-11 11:15:32,339 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 34440 actions to finish 2017-05-11 11:15:34,895 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8732 actions to finish 2017-05-11 11:15:36,450 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Time to save: 87121, thread = pool-6-thread-1 2017-05-11 11:15:36,451 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:15:36 EDT 2017 2017-05-11 11:15:36,451 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:15:36 EDT 2017 2017-05-11 11:15:38,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:38,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:15:46 EDT 2017 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:12:00 EDT 2017 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:12:00 EDT 2017 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Last check point time: 1494515520000, lagBy: 226 seconds. 2017-05-11 11:15:46,027 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:15:46 EDT 2017, startTime = Thu May 11 11:12:00 EDT 2017, endTime = Thu May 11 11:14:00 EDT 2017 2017-05-11 11:15:46,460 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:15:46,460 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:15:46,461 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:15:46,581 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:15:48,341 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:15:48,342 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:56,579 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:15:56,579 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:15:56,579 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:15:56,589 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:15:58,348 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:15:58,352 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:16:02,833 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:04,869 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:05,041 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:06,580 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:16:06,589 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:16:06,589 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:16:06,590 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:08,350 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:16:08,360 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:16:13,784 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:14,875 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:15,047 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:16,588 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8550 actions to finish 2017-05-11 11:16:16,597 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:16,598 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:16:16,599 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:16:18,358 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 83534 actions to finish 2017-05-11 11:16:18,368 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10308 actions to finish 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: 81767 row(s) updated. 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:05:00 EDT 2017, endTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:16:21 EDT 2017 2017-05-11 11:16:21,628 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:16:21 EDT 2017 2017-05-11 11:16:23,805 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:24,883 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:25,054 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:26,604 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9810 actions to finish 2017-05-11 11:16:26,604 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:26,605 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10550 actions to finish 2017-05-11 11:16:32,687 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:16:33,808 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:16:33,815 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:34,885 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7054 actions to finish 2017-05-11 11:16:35,065 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:35,299 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:16:36,613 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:42,696 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:16:43,813 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:16:43,823 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10134 actions to finish 2017-05-11 11:16:44,598 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:16:45,072 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:45,303 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:16:46,623 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:16:52,705 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:16:53,815 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:16:53,885 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:16:54,601 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:16:55,078 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:16:55,312 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:16:56,624 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:17:02,712 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:17:03,175 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:03,822 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:17:03,890 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:04,605 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:05,079 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:05,321 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:17:06,629 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:17:08,855 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:12,716 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:17:13,181 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:13,826 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:17:13,896 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:14,611 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:15,087 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:15,326 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:17:16,639 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22544 actions to finish 2017-05-11 11:17:18,866 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:22,720 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8938 actions to finish 2017-05-11 11:17:23,183 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:23,828 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 11320 actions to finish 2017-05-11 11:17:23,898 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:24,615 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:25,094 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:25,327 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9596 actions to finish 2017-05-11 11:17:25,639 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:28,871 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:33,187 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:33,906 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7442 actions to finish 2017-05-11 11:17:34,623 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:35,103 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:35,640 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:38,881 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:40,570 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:17:40,570 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:17:43,196 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:43,365 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 0 2017-05-11 11:17:44,627 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9952 actions to finish 2017-05-11 11:17:45,109 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 21474 actions to finish 2017-05-11 11:17:45,648 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:48,885 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10104 actions to finish 2017-05-11 11:17:50,580 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:17:50,580 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:17:53,201 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:17:53,402 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:17:55,649 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:17:56,378 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2548 actions to finish 2017-05-11 11:18:00,582 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:00,583 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:03,202 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:03,403 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:04,466 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8686 actions to finish 2017-05-11 11:18:04,506 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:05,655 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:06,389 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2548 actions to finish 2017-05-11 11:18:10,585 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:10,585 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:13,208 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:13,409 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:14,145 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:14,477 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8686 actions to finish 2017-05-11 11:18:14,509 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:15,664 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:20,588 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:20,588 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:20,718 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:23,217 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:23,419 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:24,155 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:24,478 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8686 actions to finish 2017-05-11 11:18:24,513 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:25,665 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:29,619 INFO TimelineClusterAggregatorHourly: 4343 row(s) updated. 2017-05-11 11:18:29,620 INFO TimelineClusterAggregatorHourly: Aggregated cluster metrics for METRIC_AGGREGATE_HOURLY, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:18:29,620 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 11:18:29 EDT 2017 2017-05-11 11:18:29,620 INFO TimelineClusterAggregatorHourly: End aggregation cycle @ Thu May 11 11:18:29 EDT 2017 2017-05-11 11:18:31,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:31,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 7408 actions to finish 2017-05-11 11:18:31,334 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:33,218 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:33,429 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:34,156 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:34,519 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:35,674 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10860 actions to finish 2017-05-11 11:18:41,339 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:41,339 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:43,228 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 22600 actions to finish 2017-05-11 11:18:43,366 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 1 2017-05-11 11:18:43,434 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:44,164 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:44,527 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:18:45 EDT 2017 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Last check point time: 1494515400000, lagBy: 525 seconds. 2017-05-11 11:18:45,522 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:18:45 EDT 2017, startTime = Thu May 11 11:10:00 EDT 2017, endTime = Thu May 11 11:15:00 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:18:45 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494515400000, lagBy: 525 seconds. 2017-05-11 11:18:45,533 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:18:45 EDT 2017, startTime = Thu May 11 11:10:00 EDT 2017, endTime = Thu May 11 11:15:00 EDT 2017 2017-05-11 11:18:45,547 ERROR TimelineMetricHostAggregatorHourly: Exception during aggregating metrics. java.sql.SQLTimeoutException: Operation timed out. at org.apache.phoenix.exception.SQLExceptionCode$14.newException(SQLExceptionCode.java:363) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:802) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:714) at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72) at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93) at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58) at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778) at org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:188) at org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:109) at org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:796) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:329) at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(PhoenixPreparedStatement.java:199) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.doWork(AbstractTimelineAggregator.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.runOnce(AbstractTimelineAggregator.java:110) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.run(AbstractTimelineAggregator.java:94) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-05-11 11:18:45,552 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 11:18:45 EDT 2017 2017-05-11 11:18:48,208 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:18:48,218 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:18:51,344 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9878 actions to finish 2017-05-11 11:18:51,344 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:18:53,439 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10660 actions to finish 2017-05-11 11:18:54,170 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:18:54,532 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:18:57,412 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:18:58,217 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:18:58,227 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:01,352 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6754 actions to finish 2017-05-11 11:19:02,778 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:04,178 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 9978 actions to finish 2017-05-11 11:19:04,537 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:19:07,420 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:19:08,226 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:19:08,236 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:11,418 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 20230 actions to finish 2017-05-11 11:19:12,784 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:14,547 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:19:17,429 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:19:18,235 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:19:18,246 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:19,982 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6752 actions to finish 2017-05-11 11:19:21,421 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 20230 actions to finish 2017-05-11 11:19:22,791 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:24,553 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 23970 actions to finish 2017-05-11 11:19:25,822 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8322 actions to finish 2017-05-11 11:19:27,437 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8928 actions to finish 2017-05-11 11:19:28,244 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 8286 actions to finish 2017-05-11 11:19:28,254 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 10220 actions to finish 2017-05-11 11:19:29,985 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 6752 actions to finish 2017-05-11 11:19:31,425 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 20230 actions to finish 2017-05-11 11:19:32,798 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 16624 actions to finish 2017-05-11 11:19:34,090 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Failed to get metrics from TimelineMetricStore, attempt = 2 2017-05-11 11:19:34,091 FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricStoreWatcher: Error getting metrics from TimelineMetricStore. Shutting down by TimelineMetricStoreWatcher. 2017-05-11 11:19:34,092 INFO org.apache.hadoop.util.ExitUtil: Exiting with status -1 2017-05-11 11:19:34,102 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@:6188 2017-05-11 11:19:34,103 ERROR org.apache.phoenix.cache.ServerCacheClient: Error trying to remove hash cache for region=METRIC_RECORD,,1488665232951.aa0e4527400db4664d8a14735d897ef8., hostname=,44249,1494501076341, seqNum=3619070 java.io.InterruptedIOException: Interrupted calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService for row at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1742) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1692) at org.apache.phoenix.cache.ServerCacheClient.removeServerCache(ServerCacheClient.java:324) at org.apache.phoenix.cache.ServerCacheClient.access$000(ServerCacheClient.java:82) at org.apache.phoenix.cache.ServerCacheClient$ServerCache.close(ServerCacheClient.java:142) at org.apache.phoenix.util.SQLCloseables.closeAllQuietly(SQLCloseables.java:53) at org.apache.phoenix.util.SQLCloseables.closeAll(SQLCloseables.java:43) at org.apache.phoenix.execute.BaseQueryPlan$1.close(BaseQueryPlan.java:334) at org.apache.phoenix.jdbc.PhoenixResultSet.close(PhoenixResultSet.java:162) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getLatestMetricRecords(PhoenixHBaseAccessor.java:934) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getMetricRecords(PhoenixHBaseAccessor.java:776) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:359) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1736) ... 60 more 2017-05-11 11:19:34,103 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR javax.ws.rs.WebApplicationException: java.sql.SQLException: ERROR 2005 (INT07): Interrupted exception. Sub plan [0] execution interrupted. at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:377) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.sql.SQLException: ERROR 2005 (INT07): Interrupted exception. Sub plan [0] execution interrupted. at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:441) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:194) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:143) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:138) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:281) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:266) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:265) at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeQuery(PhoenixPreparedStatement.java:186) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getLatestMetricRecords(PhoenixHBaseAccessor.java:926) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getMetricRecords(PhoenixHBaseAccessor.java:776) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:359) ... 48 more Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:186) ... 59 more 2017-05-11 11:19:34,110 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR javax.ws.rs.WebApplicationException: org.apache.phoenix.exception.PhoenixIOException at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:377) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: org.apache.phoenix.exception.PhoenixIOException at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:808) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:714) at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176) at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getLatestMetricRecords(PhoenixHBaseAccessor.java:927) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getMetricRecords(PhoenixHBaseAccessor.java:776) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:228) at org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:359) ... 48 more Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:204) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:766) ... 57 more 2017-05-11 11:19:34,128 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:19:36,209 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping phoenix metrics system... 2017-05-11 11:19:36,210 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system stopped. 2017-05-11 11:19:36,210 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system shutdown complete. 2017-05-11 11:19:36,211 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Stopping ApplicationHistory 2017-05-11 11:19:36,211 INFO org.apache.hadoop.ipc.Server: Stopping server on 60200 2017-05-11 11:19:36,213 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60200 2017-05-11 11:19:36,213 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2017-05-11 11:19:36,213 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer ************************************************************/ 2017-05-11 11:21:03,366 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting ApplicationHistoryServer STARTUP_MSG: user = ams STARTUP_MSG: host = STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.3.2.5.3.0-28 STARTUP_MSG: classpath = /usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9828acfdec41a121f0121f556b09e2d112259e92; compiled by 'jenkins' on 2016-11-21T23:57Z STARTUP_MSG: java = 1.8.0_77 ************************************************************/ 2017-05-11 11:21:03,372 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT] 2017-05-11 11:21:03,399 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found hbase site configuration: file:/etc/ambari-metrics-collector/conf/hbase-site.xml 2017-05-11 11:21:03,399 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found metric service configuration: file:/etc/ambari-metrics-collector/conf/ams-site.xml 2017-05-11 11:21:03,621 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Explicitly disabled application timeline store. 2017-05-11 11:21:03,622 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Creating metrics store. 2017-05-11 11:21:03,625 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: ApplicationHistory Init 2017-05-11 11:21:03,625 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Explicitly disabled application timeline store. 2017-05-11 11:21:03,858 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Initializing metrics schema... 2017-05-11 11:21:03,921 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,016 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,049 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,113 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,322 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5ddeb7cb connecting to ZooKeeper ensemble=:61181 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-28--1, built on 11/21/2016 23:52 GMT 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=str3 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_77 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_77/jre 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/ams-hbase/lib/hadoop-native 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2017-05-11 11:21:04,328 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ams 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ams 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/ams 2017-05-11 11:21:04,329 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@5b799640 2017-05-11 11:21:04,348 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:21:04,355 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:21:04,361 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf8195bad0001, negotiated timeout = 120000 2017-05-11 11:21:04,424 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:21:04,569 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix 2017-05-11 11:21:04,621 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties 2017-05-11 11:21:04,694 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-05-11 11:21:04,694 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system started 2017-05-11 11:21:04,830 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:22:13,347 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68415 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,44249,1494501076341, seqNum=0 2017-05-11 11:22:33,537 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=88607 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,44249,1494501076341, seqNum=0 2017-05-11 11:22:53,577 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=108647 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,44249,1494501076341, seqNum=0 2017-05-11 11:23:14,008 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:15,755 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Metrics schema initialized. 2017-05-11 11:23:16,348 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 4842, metadata objects from store. 2017-05-11 11:23:16,380 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 50 host objects from store. 2017-05-11 11:23:16,398 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6134ac4a connecting to ZooKeeper ensemble=str3:61181 2017-05-11 11:23:16,398 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@777c9dc9 2017-05-11 11:23:16,400 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:23:16,400 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:23:16,402 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf8195bad0011, negotiated timeout = 120000 2017-05-11 11:23:16,418 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Using group by aggregators for aggregating host and cluster metrics. 2017-05-11 11:23:16,424 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.TimelineMetricAppAggregator: AppIds configured for aggregation: [datanode, nodemanager, hbase] 2017-05-11 11:23:16,431 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Started watchdog for timeline metrics store with initial delay = 600, delay = 30 2017-05-11 11:23:16,431 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system already initialized! 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineMetricHostAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,434 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:23:16 EDT 2017 2017-05-11 11:23:16,438 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Instantiating AHSWebApp at str3:6188 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 WARN TimelineMetricHostAggregatorMinute: Last Checkpoint is too old, discarding last checkpoint. lastCheckPointTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:12:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 WARN TimelineClusterAggregatorSecond: Last Checkpoint is too old, discarding last checkpoint. lastCheckPointTime = Thu May 11 11:12:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorHourly: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorMinute: Saving checkpoint time. Thu May 11 11:15:00 EDT 2017 2017-05-11 11:23:16,445 WARN TimelineClusterAggregatorMinute: Last Checkpoint is too old, discarding last checkpoint. lastCheckPointTime = Thu May 11 11:10:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorMinute: Saving checkpoint time. Thu May 11 11:15:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorSecond: Saving checkpoint time. Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:16,445 INFO TimelineClusterAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorMinute: Last check point time: 1494515700000, lagBy: 496 seconds. 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494515700000, lagBy: 496 seconds. 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494511200000, lagBy: 4996 seconds. 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineClusterAggregatorSecond: Last check point time: 1494516000000, lagBy: 196 seconds. 2017-05-11 11:23:16,447 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:23:16,446 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:16,447 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:23:16 EDT 2017, startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:22:00 EDT 2017 2017-05-11 11:23:16,528 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2017-05-11 11:23:16,541 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2017-05-11 11:23:16,547 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.applicationhistory is not defined 2017-05-11 11:23:16,555 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2017-05-11 11:23:16,558 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context applicationhistory 2017-05-11 11:23:16,559 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2017-05-11 11:23:16,559 INFO org.apache.hadoop.security.HttpCrossOriginFilterInitializer: CORS filter not enabled. Please set hadoop.http.cross-origin.enabled to 'true' to enable it 2017-05-11 11:23:16,563 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /applicationhistory/* 2017-05-11 11:23:16,563 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2017-05-11 11:23:16,948 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2017-05-11 11:23:16,954 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 6188 2017-05-11 11:23:16,954 INFO org.mortbay.log: jetty-6.1.26.hwx 2017-05-11 11:23:16,988 INFO org.mortbay.log: Extract jar:file:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar!/webapps/applicationhistory to /tmp/Jetty_str3_bi_grn_hcvlny_cv_net_6188_applicationhistory____3kd262/webapp 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: 0 row(s) updated. 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:18 EDT 2017 2017-05-11 11:23:18,180 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:18 EDT 2017 2017-05-11 11:23:18,342 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@str3:6188 2017-05-11 11:23:18,342 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app applicationhistory started at 6188 2017-05-11 11:23:18,432 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2017-05-11 11:23:18,447 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 60200 2017-05-11 11:23:18,488 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ApplicationHistoryProtocolPB to the server 2017-05-11 11:23:18,489 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60200: starting 2017-05-11 11:23:18,489 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2017-05-11 11:23:18,499 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService: Instantiated ApplicationHistoryClientService at str3/.70:60200 2017-05-11 11:23:18,499 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Starting ApplicationHistory 2017-05-11 11:23:20,242 INFO TimelineClusterAggregatorSecond: Saving 15664 metric aggregates. 2017-05-11 11:23:23,410 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:23:23 EDT 2017 2017-05-11 11:23:23,411 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:23:23 EDT 2017 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: 81770 row(s) updated. 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:15:00 EDT 2017, endTime = Thu May 11 11:20:00 EDT 2017 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:24 EDT 2017 2017-05-11 11:23:24,617 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:23:24 EDT 2017 2017-05-11 11:23:28,091 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: RECEIVED SIGNAL 15: SIGTERM 2017-05-11 11:23:28,098 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@str3:6188 2017-05-11 11:23:28,117 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:28,199 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping phoenix metrics system... 2017-05-11 11:23:28,200 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system stopped. 2017-05-11 11:23:28,200 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system shutdown complete. 2017-05-11 11:23:28,200 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Stopping ApplicationHistory 2017-05-11 11:23:28,200 INFO org.apache.hadoop.ipc.Server: Stopping server on 60200 2017-05-11 11:23:28,201 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60200 2017-05-11 11:23:28,201 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2017-05-11 11:23:28,202 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at str3/.70 ************************************************************/ 2017-05-11 11:23:43,719 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting ApplicationHistoryServer STARTUP_MSG: user = ams STARTUP_MSG: host = str3/.70 STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.3.2.5.3.0-28 STARTUP_MSG: classpath = /usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9828acfdec41a121f0121f556b09e2d112259e92; compiled by 'jenkins' on 2016-11-21T23:57Z STARTUP_MSG: java = 1.8.0_77 ************************************************************/ 2017-05-11 11:23:43,727 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT] 2017-05-11 11:23:43,757 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found hbase site configuration: file:/etc/ambari-metrics-collector/conf/hbase-site.xml 2017-05-11 11:23:43,757 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.TimelineMetricConfiguration: Found metric service configuration: file:/etc/ambari-metrics-collector/conf/ams-site.xml 2017-05-11 11:23:43,976 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Explicitly disabled application timeline store. 2017-05-11 11:23:43,978 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Creating metrics store. 2017-05-11 11:23:43,981 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: ApplicationHistory Init 2017-05-11 11:23:43,981 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Explicitly disabled application timeline store. 2017-05-11 11:23:44,222 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Initializing metrics schema... 2017-05-11 11:23:44,284 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,381 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,414 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,509 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,707 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5ddeb7cb connecting to ZooKeeper ensemble=str3:61181 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-28--1, built on 11/21/2016 23:52 GMT 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=str3 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_77 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_77/jre 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/ambari-metrics-collector/jcodings-1.0.8.jar:/usr/lib/ambari-metrics-collector/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-nodemanager-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-protocol-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-core-1.8.0.jar:/usr/lib/ambari-metrics-collector/asm-all-5.0.2.jar:/usr/lib/ambari-metrics-collector/jersey-json-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/guice-assistedinject-3.0.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jetty-util-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/javax.inject-1.jar:/usr/lib/ambari-metrics-collector/hadoop-annotations-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/tephra-api-0.7.0.jar:/usr/lib/ambari-metrics-collector/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-metrics-collector/tephra-core-0.7.0.jar:/usr/lib/ambari-metrics-collector/stringtemplate-3.2.1.jar:/usr/lib/ambari-metrics-collector/java-xmlbuilder-0.4.jar:/usr/lib/ambari-metrics-collector/httpclient-4.2.5.jar:/usr/lib/ambari-metrics-collector/aopalliance-1.0.jar:/usr/lib/ambari-metrics-collector/gson-2.2.4.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/phoenix-core-4.7.0.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/annotations-1.3.2.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5.jar:/usr/lib/ambari-metrics-collector/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/zookeeper-3.4.6.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jersey-servlet-1.11.jar:/usr/lib/ambari-metrics-collector/commons-configuration-1.6.jar:/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/okhttp-2.4.0.jar:/usr/lib/ambari-metrics-collector/guava-14.0.1.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-common-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xercesImpl-2.11.0.jar:/usr/lib/ambari-metrics-collector/jcip-annotations-1.0.jar:/usr/lib/ambari-metrics-collector/twill-discovery-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/xmlenc-0.52.jar:/usr/lib/ambari-metrics-collector/azure-storage-4.2.0.jar:/usr/lib/ambari-metrics-collector/commons-compress-1.4.1.jar:/usr/lib/ambari-metrics-collector/tephra-hbase-compat-1.1-0.7.0.jar:/usr/lib/ambari-metrics-collector/jetty-sslengine-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/curator-framework-2.7.1.jar:/usr/lib/ambari-metrics-collector/curator-recipes-2.7.1.jar:/usr/lib/ambari-metrics-collector/hadoop-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-all-4.0.23.Final.jar:/usr/lib/ambari-metrics-collector/jersey-guice-1.11.jar:/usr/lib/ambari-metrics-collector/hadoop-auth-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jaxb-impl-2.2.3-1.jar:/usr/lib/ambari-metrics-collector/hadoop-distcp-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jline-2.11.jar:/usr/lib/ambari-metrics-collector/commons-collections-3.2.1.jar:/usr/lib/ambari-metrics-collector/fastutil-6.5.6.jar:/usr/lib/ambari-metrics-collector/okio-1.4.0.jar:/usr/lib/ambari-metrics-collector/jets3t-0.9.0.jar:/usr/lib/ambari-metrics-collector/metrics-core-2.2.0.jar:/usr/lib/ambari-metrics-collector/commons-logging-1.1.1.jar:/usr/lib/ambari-metrics-collector/joda-time-1.6.jar:/usr/lib/ambari-metrics-collector/curator-client-2.7.1.jar:/usr/lib/ambari-metrics-collector/guice-servlet-3.0.jar:/usr/lib/ambari-metrics-collector/commons-math-2.2.jar:/usr/lib/ambari-metrics-collector/libthrift-0.9.0.jar:/usr/lib/ambari-metrics-collector/commons-lang-2.5.jar:/usr/lib/ambari-metrics-collector/snappy-java-1.0.4.1.jar:/usr/lib/ambari-metrics-collector/jsr305-2.0.1.jar:/usr/lib/ambari-metrics-collector/commons-math3-3.4.1.jar:/usr/lib/ambari-metrics-collector/commons-cli-1.2.jar:/usr/lib/ambari-metrics-collector/httpcore-4.2.4.jar:/usr/lib/ambari-metrics-collector/joni-2.1.2.jar:/usr/lib/ambari-metrics-collector/jersey-client-1.11.jar:/usr/lib/ambari-metrics-collector/guice-3.0.jar:/usr/lib/ambari-metrics-collector/protobuf-java-2.5.0.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/commons-daemon-1.0.13.jar:/usr/lib/ambari-metrics-collector/jamon-runtime-2.4.1.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0.1.jar:/usr/lib/ambari-metrics-collector/hadoop-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/commons-beanutils-1.7.0.jar:/usr/lib/ambari-metrics-collector/snappy-0.3.jar:/usr/lib/ambari-metrics-collector/hbase-hadoop2-compat-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/avro-1.7.4.jar:/usr/lib/ambari-metrics-collector/api-asn1-api-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/commons-digester-1.8.jar:/usr/lib/ambari-metrics-collector/twill-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hbase-common-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/netty-3.6.2.Final.jar:/usr/lib/ambari-metrics-collector/commons-net-3.1.jar:/usr/lib/ambari-metrics-collector/jsp-api-2.1.jar:/usr/lib/ambari-metrics-collector/leveldbjni-all-1.8.jar:/usr/lib/ambari-metrics-collector/jsp-2.1-6.1.14.jar:/usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.4.2.0.136.jar:/usr/lib/ambari-metrics-collector/twill-common-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.0.1.jar:/usr/lib/ambari-metrics-collector/api-util-1.0.0-M20.jar:/usr/lib/ambari-metrics-collector/log4j-1.2.17.jar:/usr/lib/ambari-metrics-collector/jackson-core-2.2.3.jar:/usr/lib/ambari-metrics-collector/jackson-jaxrs-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-server-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/antlr-runtime-3.5.jar:/usr/lib/ambari-metrics-collector/jaxb-api-2.2.2.jar:/usr/lib/ambari-metrics-collector/json-smart-1.1.1.jar:/usr/lib/ambari-metrics-collector/jdk.tools-1.7.jar:/usr/lib/ambari-metrics-collector/commons-lang3-3.3.2.jar:/usr/lib/ambari-metrics-collector/jsch-0.1.42.jar:/usr/lib/ambari-metrics-collector/jackson-xc-1.9.2.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/hbase-client-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ambari-metrics-collector/servlet-api-2.5-6.1.14.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/activation-1.1.jar:/usr/lib/ambari-metrics-collector/slf4j-api-1.7.2.jar:/usr/lib/ambari-metrics-collector/antlr-2.7.7.jar:/usr/lib/ambari-metrics-collector/nimbus-jose-jwt-3.9.jar:/usr/lib/ambari-metrics-collector/hbase-annotations-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-metrics-collector/disruptor-3.3.0.jar:/usr/lib/ambari-metrics-collector/htrace-core-3.1.0-incubating.jar:/usr/lib/ambari-metrics-collector/findbugs-annotations-1.3.9-1.jar:/usr/lib/ambari-metrics-collector/paranamer-2.3.jar:/usr/lib/ambari-metrics-collector/twill-discovery-core-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/metrics-core-3.1.0.jar:/usr/lib/ambari-metrics-collector/jetty-6.1.26.hwx.jar:/usr/lib/ambari-metrics-collector/asm-3.1.jar:/usr/lib/ambari-metrics-collector/jersey-server-1.11.jar:/usr/lib/ambari-metrics-collector/sqlline-1.1.8.jar:/usr/lib/ambari-metrics-collector/commons-csv-1.0.jar:/usr/lib/ambari-metrics-collector/twill-api-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/commons-io-2.4.jar:/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/jettison-1.1.jar:/usr/lib/ambari-metrics-collector/jersey-core-1.11.jar:/usr/lib/ambari-metrics-collector/commons-pool2-2.3.jar:/usr/lib/ambari-metrics-collector/hbase-procedure-1.1.2.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/xml-apis-1.4.01.jar:/usr/lib/ambari-metrics-collector/xz-1.0.jar:/usr/lib/ambari-metrics-collector/commons-codec-1.7.jar:/usr/lib/ambari-metrics-collector/twill-zookeeper-0.6.0-incubating.jar:/usr/lib/ambari-metrics-collector/hadoop-yarn-server-common-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/stax-api-1.0-2.jar:/usr/lib/ambari-metrics-collector/hadoop-azure-2.7.3.2.5.3.0-28.jar:/usr/lib/ambari-metrics-collector/azure-keyvault-core-0.8.0.jar:/etc/ambari-metrics-collector/conf 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/ams-hbase/lib/hadoop-native 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2017-05-11 11:23:44,713 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ams 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ams 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/ams 2017-05-11 11:23:44,714 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@5b799640 2017-05-11 11:23:44,732 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:23:44,738 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:23:44,744 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf81bce100001, negotiated timeout = 120000 2017-05-11 11:23:44,802 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:23:44,934 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix 2017-05-11 11:23:44,980 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties 2017-05-11 11:23:45,048 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-05-11 11:23:45,048 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system started 2017-05-11 11:23:45,179 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:24:53,597 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68321 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,40557,1494516064903, seqNum=0 2017-05-11 11:25:13,663 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=88388 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,40557,1494516064903, seqNum=0 2017-05-11 11:25:33,708 INFO org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=108433 ms ago, cancelled=false, msg=row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=str3,40557,1494516064903, seqNum=0 2017-05-11 11:25:54,138 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-11 11:25:58,934 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Metrics schema initialized. 2017-05-11 11:25:59,481 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 4842, metadata objects from store. 2017-05-11 11:25:59,514 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.discovery.TimelineMetricMetadataManager: Retrieved 50 host objects from store. 2017-05-11 11:25:59,531 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6134ac4a connecting to ZooKeeper ensemble=str3:61181 2017-05-11 11:25:59,532 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=str3:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@777c9dc9 2017-05-11 11:25:59,533 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server str3/.70:61181. Will not attempt to authenticate using SASL (unknown error) 2017-05-11 11:25:59,534 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to str3/.70:61181, initiating session 2017-05-11 11:25:59,535 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server str3/.70:61181, sessionid = 0x15bf81bce100011, negotiated timeout = 120000 2017-05-11 11:25:59,552 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Using group by aggregators for aggregating host and cluster metrics. 2017-05-11 11:25:59,561 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.TimelineMetricAppAggregator: AppIds configured for aggregation: [datanode, nodemanager, hbase] 2017-05-11 11:25:59,570 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: Started watchdog for timeline metrics store with initial delay = 600, delay = 30 2017-05-11 11:25:59,570 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system already initialized! 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineMetricHostAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineMetricHostAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorHourly: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorDaily: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,572 INFO TimelineMetricHostAggregatorMinute: Started Timeline aggregator thread @ Thu May 11 11:25:59 EDT 2017 2017-05-11 11:25:59,581 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Instantiating AHSWebApp at str3:6188 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorHourly: Last Checkpoint read : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorDaily: Last Checkpoint read : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu May 11 11:22:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorMinute: Last Checkpoint read : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu May 11 11:22:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorMinute: Last Checkpoint read : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorHourly: Last Checkpoint read : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorMinute: Rounded off checkpoint : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorSecond: Last check point time: 1494516120000, lagBy: 239 seconds. 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorMinute: Rounded off checkpoint : Thu May 11 11:20:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineMetricHostAggregatorDaily: Rounded off checkpoint : Wed May 10 20:00:00 EDT 2017 2017-05-11 11:25:59,583 INFO TimelineClusterAggregatorHourly: Rounded off checkpoint : Thu May 11 11:00:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 11:22:00 EDT 2017, endTime = Thu May 11 11:24:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorDaily: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorMinute: Last check point time: 1494516000000, lagBy: 359 seconds. 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorMinute: Last check point time: 1494516000000, lagBy: 359 seconds. 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorMinute: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorHourly: Rounded off checkpoint : Thu May 11 10:00:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorMinute: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:25:59,584 INFO TimelineClusterAggregatorHourly: Last checkpoint too recent for aggregation. Sleeping for 1 cycle. 2017-05-11 11:25:59,584 INFO TimelineMetricHostAggregatorHourly: Last check point time: 1494511200000, lagBy: 5159 seconds. 2017-05-11 11:25:59,585 INFO TimelineMetricHostAggregatorHourly: Start aggregation cycle @ Thu May 11 11:25:59 EDT 2017, startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:25:59,683 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2017-05-11 11:25:59,694 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2017-05-11 11:25:59,699 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.applicationhistory is not defined 2017-05-11 11:25:59,705 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2017-05-11 11:25:59,707 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context applicationhistory 2017-05-11 11:25:59,708 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2017-05-11 11:25:59,708 INFO org.apache.hadoop.security.HttpCrossOriginFilterInitializer: CORS filter not enabled. Please set hadoop.http.cross-origin.enabled to 'true' to enable it 2017-05-11 11:25:59,711 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /applicationhistory/* 2017-05-11 11:25:59,711 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2017-05-11 11:26:00,145 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2017-05-11 11:26:00,153 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 6188 2017-05-11 11:26:00,154 INFO org.mortbay.log: jetty-6.1.26.hwx 2017-05-11 11:26:00,203 INFO org.mortbay.log: Extract jar:file:/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.7.3.2.5.3.0-28.jar!/webapps/applicationhistory to /tmp/Jetty_str3_bi_grn_hcvlny_cv_net_6188_applicationhistory____3kd262/webapp 2017-05-11 11:26:00,687 INFO TimelineClusterAggregatorSecond: Saving 7552 metric aggregates. 2017-05-11 11:26:02,018 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@str3:6188 2017-05-11 11:26:02,018 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app applicationhistory started at 6188 2017-05-11 11:26:02,100 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2017-05-11 11:26:02,112 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 60200 2017-05-11 11:26:02,137 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ApplicationHistoryProtocolPB to the server 2017-05-11 11:26:02,138 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60200: starting 2017-05-11 11:26:02,138 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2017-05-11 11:26:02,148 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService: Instantiated ApplicationHistoryClientService at str3/.70:60200 2017-05-11 11:26:02,149 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Starting ApplicationHistory 2017-05-11 11:26:02,253 INFO TimelineMetricHostAggregatorMinute: 3677 row(s) updated. 2017-05-11 11:26:02,253 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:26:02,253 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,254 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,256 INFO TimelineClusterAggregatorMinute: 3916 row(s) updated. 2017-05-11 11:26:02,257 INFO TimelineClusterAggregatorMinute: Aggregated cluster metrics for METRIC_AGGREGATE_MINUTE, with startTime = Thu May 11 11:20:00 EDT 2017, endTime = Thu May 11 11:25:00 EDT 2017 2017-05-11 11:26:02,257 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,257 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,348 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:02,349 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu May 11 11:26:02 EDT 2017 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: 81808 row(s) updated. 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: Aggregated host metrics for METRIC_RECORD_HOURLY, with startTime = Thu May 11 10:00:00 EDT 2017, endTime = Thu May 11 11:00:00 EDT 2017 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 11:26:10 EDT 2017 2017-05-11 11:26:10,623 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Thu May 11 11:26:10 EDT 2017

3 REPLIES 3
Highlighted

Re: Ambari metrics collector is shutting down abruptly

New Contributor

Hi @PJ

- Please verify your AMS heap settings are configured properly as per the below documentation.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-operations/content/ams_general_gu...

- Do a cleanup on "hbase.tmp.dir".

a) Shutdown AMS.

b) Move "hbase.tmp.dir".

c) Start AMS

Please monitor AMS and check if you are getting same exceptions.

Re: Ambari metrics collector is shutting down abruptly

Did you enable HBase metrics?

Re: Ambari metrics collector is shutting down abruptly

Expert Contributor

I just increased ambari metricsheapsize from 756MB to 4 GB.No issues since then ...

Don't have an account?
Coming from Hortonworks? Activate your account here