Ambari-metrics collector is stopping constantly

[#1]

We have an HDP 3.1.0 cluster (single node) without Hbase.

The ambari-metrics-collector is constantly stopping with the following error

2020-01-13 22:09:55,247 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269. is not online on xxxxxxx,61320,1578982133933
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3273)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3250)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2964)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
, details=row 'a�^T}^NK�+`�u�h��^@^Ao�u=0' on table 'METRIC_AGGREGATE_UUID' at region=METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\\
x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269., hostname=xxxxxx,61320,1563230648592, seqNum=103729

I have a couple of questions.

1. Is Hbase a mandatory service for clusters running hdp 3.1.0 +?

2. How do we recreate missing/accidentally deleted znodes like ams-hbase-secure znode in zookeeper? Which component handles this. Many components throw errors like NoNode for xxxxx in zookeeper

3. We tried renaming the znode parent path in config but to no luck. 

Any help would be appreciated.

[#2 · Accepted solution]

Got the solution.

Follow this guide and things should come back up fine

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

[#3]

Hi,

i see this is mentioned in above link

"AMS data would be stored in 'hbase.rootdir' identified above. Backup and remove the AMS data.

1. If the Metrics Service operation mode
   1. is 'embedded', then the data is stored in OS files. Use regular OS commands to backup and remove the files in hbase.rootdir"

so we need to remove dir structure or only files inside folders?

please let me know