Ambari-metrics collector is stopping constantly

ShaatH — Tue, 14 Jan 2020 07:32:37 GMT

We have an HDP 3.1.0 cluster (single node) without Hbase.

The ambari-metrics-collector is constantly stopping with the following error

2020-01-13 22:09:55,247 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269. is not online on xxxxxxx,61320,1578982133933 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3273) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3250) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2964) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) , details=row 'a�^T}^NK�+`�u�h��^@^Ao�u=0' on table 'METRIC_AGGREGATE_UUID' at region=METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\\ x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269., hostname=xxxxxx,61320,1563230648592, seqNum=103729

I have a couple of questions.

1. Is Hbase a mandatory service for clusters running hdp 3.1.0 +?

2. How do we recreate missing/accidentally deleted znodes like ams-hbase-secure znode in zookeeper? Which component handles this. Many components throw errors like NoNode for xxxxx in zookeeper

3. We tried renaming the znode parent path in config but to no luck.

Any help would be appreciated.

Re: Ambari-metrics collector is stopping constantly

ShaatH — Thu, 16 Jan 2020 10:06:01 GMT

Got the solution.

Follow this guide and things should come back up fine

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

Re: Ambari-metrics collector is stopping constantly

srinivasp — Tue, 11 Feb 2020 10:54:59 GMT

Hi,

i see this is mentioned in above link

"AMS data would be stored in 'hbase.rootdir' identified above. Backup and remove the AMS data.

If the Metrics Service operation mode
1. is 'embedded', then the data is stored in OS files. Use regular OS commands to backup and remove the files in hbase.rootdir"

so we need to remove dir structure or only files inside folders?

please let me know

question Re: Ambari-metrics collector is stopping constantly in Support Questions

Ambari-metrics collector is stopping constantly

Re: Ambari-metrics collector is stopping constantly

Re: Ambari-metrics collector is stopping constantly