Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari-metrics collector is stopping constantly

Solved Go to solution

Ambari-metrics collector is stopping constantly

Explorer

We have an HDP 3.1.0 cluster (single node) without Hbase.

 

The ambari-metrics-collector is constantly stopping with the following error

 

 

2020-01-13 22:09:55,247 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269. is not online on xxxxxxx,61320,1578982133933
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3273)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3250)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2964)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
, details=row 'a�^T}^NK�+`�u�h��^@^Ao�u=0' on table 'METRIC_AGGREGATE_UUID' at region=METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\\
x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269., hostname=xxxxxx,61320,1563230648592, seqNum=103729

 

 

 

I have a couple of questions.

 

1. Is Hbase a mandatory service for clusters running hdp 3.1.0 +?

2. How do we recreate missing/accidentally deleted znodes like ams-hbase-secure znode in zookeeper? Which component handles this. Many components throw errors like NoNode for xxxxx in zookeeper

3. We tried renaming the znode parent path in config but to no luck. 

 

 

Any help would be appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Ambari-metrics collector is stopping constantly

Explorer

Got the solution.

 

Follow this guide and things should come back up fine

 

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

 

 

View solution in original post

2 REPLIES 2
Highlighted

Re: Ambari-metrics collector is stopping constantly

Explorer

Got the solution.

 

Follow this guide and things should come back up fine

 

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

 

 

View solution in original post

Highlighted

Re: Ambari-metrics collector is stopping constantly

Explorer

Hi,

 

i see this is mentioned in above link

 

"AMS data would be stored in 'hbase.rootdir' identified above. Backup and remove the AMS data.

  1. If the Metrics Service operation mode
    1. is 'embedded', then the data is stored in OS files. Use regular OS commands to backup and remove the files in hbase.rootdir"

so we need to remove dir structure or only files inside folders?

 

please let me know

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here