Dear Jay , can you help me with my post - https://community.cloudera.com/t5/Support-Questions/how-to-use-Ambari-API-to-add-host-to-new-config-group/td-p/293016  ... View more

Dear Jay   what to say - you are the best ,  well done    ... View more

Dear Jay   regarding this ambari cluster , I noticed that some of the metrics are shown and some of them are not   as the following example          from the logs we can see     [root@master ambari-metrics-collector]# tail -10 hbase-ams-regionserver-master.sys77.com.log 2020-02-21 08:15:13,237 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://master.sys77.com:6188/ws/v1/timeline/metrics 2020-02-21 08:15:13,237 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Container Metrics Uri: http://master.sys77.com:6188/ws/v1/timeline/containermetrics 2020-02-21 08:15:13,240 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 08:15:13,243 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 08:15:13,243 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 08:15:48,203 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Flush of region METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91. due to global heap pressure. Total Memstore size=227.6 M, Region memstore size=68.4 M 2020-02-21 08:15:48,205 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91., current region memstore size 68.37 MB, and 1/1 column families' memstores are being flushed. 2020-02-21 08:15:48,605 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3159, memsize=68.4 M, hasBloomFilter=false, into tmp file hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/.tmp/97bc3e982c0846e2864c2750526cca4f 2020-02-21 08:15:48,648 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/0/97bc3e982c0846e2864c2750526cca4f, entries=267808, sequenceid=3159, filesize=3.9 M 2020-02-21 08:15:48,652 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~68.37 MB/71693616, currentsize=0 B/0 for region METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91. in 447ms, sequenceid=3159, compaction requested=false [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# tail -10 ambari-metrics-collector.log 2020-02-21 08:16:20,918 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Fri Feb 21 08:16:20 UTC 2020 2020-02-21 08:16:21,227 INFO TimelineClusterAggregatorSecond: Saving 8394 metric aggregates. 2020-02-21 08:16:21,372 INFO TimelineMetricHostAggregatorMinute: 5448 row(s) updated in aggregation. 2020-02-21 08:16:21,419 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,440 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,457 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Fri Feb 21 08:10:00 UTC 2020, endTime = Fri Feb 21 08:15:00 UTC 2020 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Fri Feb 21 08:16:21 UTC 2020 2020-02-21 08:16:21,752 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Fri Feb 21 08:16:21 UTC 2020 [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# tail -10 hbase-ams-master-master.sys77.com.log 2020-02-21 07:45:19,565 INFO [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=61300] master.HMaster: Client=ams/null List Table Descriptor for the METRIC_AGGREGATE_DAILY table succeeds 2020-02-21 07:46:09,290 INFO [timeline] availability.MetricSinkWriteShardHostnameHashingStrategy: Calculated collector shard master.sys77.com based on hostname: master.sys77.com 2020-02-21 07:49:59,377 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2020-02-21 07:54:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 2020-02-21 07:59:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0 2020-02-21 08:00:32,031 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hdfsha/user/ams/hbase/MasterProcWALs/state-00000000000000000055.log 2020-02-21 08:00:32,031 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hdfsha/user/ams/hbase/MasterProcWALs/state-00000000000000000056.log] 2020-02-21 08:04:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0 2020-02-21 08:09:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0 2020-02-21 08:14:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0 [root@master ambari-metrics-collector]#     we can see that   2020-02-21 08:15:48,648 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/0/97bc3e982c0846e2864c2750526cca4f, entries=267808, sequenceid=3159, filesize=3.9 M   but we set the - hbase.rootdir=/user/ams/hbase  ... View more

Sorry, I've not come across any scripts yet. For observability the cluster utilisation report is something that you can review to understand how weightage influenced the load. More details are in this link https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/admin_cluster_util_report.html#concept_edr_ntt_2v ... View more

Dear Jay   what to say   excellent answer , you are really one of the best here  ... View more

@mike_bronson7       Kafka broker needs at least the following number of file descriptors to just track log segment files:     (number of partitions)*(partition size / segment size)     You can review the current limits configuration under:     cat /proc/<kafka_pid>/limits     If you want to change them, if you're using ambari console you can go to > Kafka > config > and search for "kafka_user_nofile_limit"   Finally, To see open file descriptors, run:     lsof -p KAFKA_BROKER_PID     ... View more

Jay - can you help me with this post - https://community.cloudera.com/t5/Support-Questions/how-to-recover-bad-namenode-from-good-namenode/td-p/288471   ... View more