Support Questions
Find answers, ask questions, and share your expertise

Ambari-metrics collector is stopping constantly

Solved Go to solution

Ambari-metrics collector is stopping constantly

Explorer

We have an HDP 3.1.0 cluster (single node) without Hbase.

 

The ambari-metrics-collector is constantly stopping with the following error

 

 

2020-01-13 22:09:55,247 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269. is not online on xxxxxxx,61320,1578982133933
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3273)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3250)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2964)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
, details=row 'a�^T}^NK�+`�u�h��^@^Ao�u=0' on table 'METRIC_AGGREGATE_UUID' at region=METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\\
x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269., hostname=xxxxxx,61320,1563230648592, seqNum=103729

 

 

 

I have a couple of questions.

 

1. Is Hbase a mandatory service for clusters running hdp 3.1.0 +?

2. How do we recreate missing/accidentally deleted znodes like ams-hbase-secure znode in zookeeper? Which component handles this. Many components throw errors like NoNode for xxxxx in zookeeper

3. We tried renaming the znode parent path in config but to no luck. 

 

 

Any help would be appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Ambari-metrics collector is stopping constantly

Explorer

Got the solution.

 

Follow this guide and things should come back up fine

 

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

 

 

View solution in original post

2 REPLIES 2

Re: Ambari-metrics collector is stopping constantly

Explorer

Got the solution.

 

Follow this guide and things should come back up fine

 

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

 

 

View solution in original post

Re: Ambari-metrics collector is stopping constantly

Explorer

Hi,

 

i see this is mentioned in above link

 

"AMS data would be stored in 'hbase.rootdir' identified above. Backup and remove the AMS data.

  1. If the Metrics Service operation mode
    1. is 'embedded', then the data is stored in OS files. Use regular OS commands to backup and remove the files in hbase.rootdir"

so we need to remove dir structure or only files inside folders?

 

please let me know