Member since
03-14-2016
4721
Posts
1111
Kudos Received
874
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2438 | 04-27-2020 03:48 AM | |
4866 | 04-26-2020 06:18 PM | |
3972 | 04-26-2020 06:05 PM | |
3209 | 04-13-2020 08:53 PM | |
4904 | 03-31-2020 02:10 AM |
02-21-2020
12:23 AM
Dear Jay regarding this ambari cluster , I noticed that some of the metrics are shown and some of them are not as the following example from the logs we can see [root@master ambari-metrics-collector]# tail -10 hbase-ams-regionserver-master.sys77.com.log 2020-02-21 08:15:13,237 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://master.sys77.com:6188/ws/v1/timeline/metrics 2020-02-21 08:15:13,237 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Container Metrics Uri: http://master.sys77.com:6188/ws/v1/timeline/containermetrics 2020-02-21 08:15:13,240 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2020-02-21 08:15:13,243 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-02-21 08:15:13,243 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2020-02-21 08:15:48,203 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Flush of region METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91. due to global heap pressure. Total Memstore size=227.6 M, Region memstore size=68.4 M 2020-02-21 08:15:48,205 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91., current region memstore size 68.37 MB, and 1/1 column families' memstores are being flushed. 2020-02-21 08:15:48,605 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3159, memsize=68.4 M, hasBloomFilter=false, into tmp file hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/.tmp/97bc3e982c0846e2864c2750526cca4f 2020-02-21 08:15:48,648 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/0/97bc3e982c0846e2864c2750526cca4f, entries=267808, sequenceid=3159, filesize=3.9 M 2020-02-21 08:15:48,652 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~68.37 MB/71693616, currentsize=0 B/0 for region METRIC_RECORD,kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1582267405534.b665584cec3e6aa7a266b7fc94ef2e91. in 447ms, sequenceid=3159, compaction requested=false [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# tail -10 ambari-metrics-collector.log 2020-02-21 08:16:20,918 INFO TimelineClusterAggregatorMinute: End aggregation cycle @ Fri Feb 21 08:16:20 UTC 2020 2020-02-21 08:16:21,227 INFO TimelineClusterAggregatorSecond: Saving 8394 metric aggregates. 2020-02-21 08:16:21,372 INFO TimelineMetricHostAggregatorMinute: 5448 row(s) updated in aggregation. 2020-02-21 08:16:21,419 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,440 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,457 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: End Downsampling cycle. 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: Aggregated host metrics for METRIC_RECORD_MINUTE, with startTime = Fri Feb 21 08:10:00 UTC 2020, endTime = Fri Feb 21 08:15:00 UTC 2020 2020-02-21 08:16:21,479 INFO TimelineMetricHostAggregatorMinute: End aggregation cycle @ Fri Feb 21 08:16:21 UTC 2020 2020-02-21 08:16:21,752 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Fri Feb 21 08:16:21 UTC 2020 [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# [root@master ambari-metrics-collector]# tail -10 hbase-ams-master-master.sys77.com.log 2020-02-21 07:45:19,565 INFO [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=61300] master.HMaster: Client=ams/null List Table Descriptor for the METRIC_AGGREGATE_DAILY table succeeds 2020-02-21 07:46:09,290 INFO [timeline] availability.MetricSinkWriteShardHostnameHashingStrategy: Calculated collector shard master.sys77.com based on hostname: master.sys77.com 2020-02-21 07:49:59,377 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2020-02-21 07:54:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 2020-02-21 07:59:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0 2020-02-21 08:00:32,031 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hdfsha/user/ams/hbase/MasterProcWALs/state-00000000000000000055.log 2020-02-21 08:00:32,031 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hdfsha/user/ams/hbase/MasterProcWALs/state-00000000000000000056.log] 2020-02-21 08:04:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0 2020-02-21 08:09:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0 2020-02-21 08:14:59,376 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=159.41 KB, freeSize=150.39 MB, max=150.54 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0 [root@master ambari-metrics-collector]# we can see that 2020-02-21 08:15:48,648 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://hdfsha/user/ams/hbase/data/default/METRIC_RECORD/b665584cec3e6aa7a266b7fc94ef2e91/0/97bc3e982c0846e2864c2750526cca4f, entries=267808, sequenceid=3159, filesize=3.9 M but we set the - hbase.rootdir=/user/ams/hbase
... View more
02-19-2020
07:27 AM
I followed https://community.cloudera.com/t5/Community-Articles/ambari-2-7-3-Ambari-writes-Empty-baseurl-values... and try again, it works. Maybe I didn't hard reload the page and start the create cluster wizard again
... View more
02-17-2020
09:14 PM
it show server error in my browser and i was followed every step in termial what you said . so ,can u tell me why?
... View more
02-17-2020
03:42 AM
Thank you for your collaboration, I found the solution. I'm using the wrong version of Livy.
... View more
02-17-2020
01:16 AM
this suggestion works for me.....
... View more
02-16-2020
10:32 PM
@Kureikana Can you try this Suppose you start your infra solr process as "infra-solr" user then try the following command. Non Kerberos Env: # su - infra-solr
# source /etc/ambari-infra-solr/conf/infra-solr-env.sh
# /usr/lib/ambari-infra-solr/bin/solr start -cloud -noprompt -s /var/lib/ambari-infra-solr/data 2>&1 Kerberos Env: # su - infra-solr
# kinit -kt /etc/security/keytabs/ambari-infra-solr.service.keytab <AMBARI_INFRA_PRINCIPAL>
# source /etc/ambari-infra-solr/conf/infra-solr-env.sh
# /usr/lib/ambari-infra-solr/bin/solr start -cloud -noprompt -s /var/lib/ambari-infra-solr/data -Dsolr.kerberos.name.rules='DEFAULT' 2>&1 .
... View more
02-16-2020
01:04 PM
Hi @AarifAkhter,
As this thread was marked 'Solved' 03-18-2017 you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more targeted answer to your question.
... View more
02-13-2020
02:23 PM
@AarifAkhter We see the error is cause by some DB access/connect Caused by: java.lang.RuntimeException: Error while creating database accessor
at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:120) So few queries: 1. Is this a newly setup ambari server? 2. Do you see the DataBase is running fine and the DB host/port is accessible ? 3. Are you using Embedded Postgres Database? 4. When was last time ambari server running fine? Any recent chances made to ambari server config/host ? I am suspecting that there should be some more detailed log like some more "Caused By" section in your ambari-server.log which is not copied fully in your last update. So can you please recheck your ambari log and let us know what was the first error and if you can share the Full stack trace. some times there are more than on "Caused By" section in a single stack trace.
... View more
02-12-2020
04:26 PM
Hi @lwang @jsensharma Thank you for the useful information that you've provided. After doing some testing. I found out that there an issue with one of the network interfaces on the servers. By testing the jumbo frame connectivity. We remove the defective module and heartbeat lost has been resolved. Thank you for your assistance guys!.
... View more
02-12-2020
12:37 PM
1 Kudo
Your /etc/hosts file does not look correct. According to Learning the Ropes of the HDP Sandbox, the IP address for Docker is 127.0.0.1 and /etc/hosts file should contain something like: 127.0.0.1 sandbox-hdp.hortonworks.com sandbox-hdf.hortonworks.com localhost
... View more