Ambari Metrics Collector HBase Corruption

[#1]

We’re running Hortonworks Data Flow (HDF) Nifi actually.

We have recently experienced Ambari Metrics Collector (AMC) HBase corruption. As per the multitude of posted fixes we deleted hbase and hbase-tmp and restarted. Great AMC is working again. But now I’ lost my history. This has happened 3-4 times on different HDF clusters

In addition in this instance Ambari displays what appears to be a medical kit logo suggesting there are still issues.

Is there any good way to recover from this corruption?

Hbase hbc is not of much use although is does show that /hbase does not exist in ZooKeeper even on good clusters.

For the moment I’d like to get the suitcase to go away. Long term does anyone have suggestions on avoiding AMC corruption?

[#2]

@William Sandman

Deleting hbase data is not a solution - you would loose all the history data.

Next time if you see that issue again, better to check the errors and post it in HCC.

"medical kit logo" - can you clarify what do you mean by this?

[#3]

"medical kit logo" = Maintenance Mode Icon....

[#4]

@Steven Matison

May be some one was setting the service to "Maintenance Mode" - Ambari or service can't go to Maintenancemode on its own.

[#5]

@Steven Matison There is nothing like corruption - you will have to review the errors if it happens next time.

[#6]

OK OK... I'm a moron... Medical logo kit... Ambari Maintenance mode. but the corruption question stands.

[#7]

"Hbase hbc is not of much use although is does show that /hbase does not exist in ZooKeeper even on good clusters"

Did you mean `hbase hbck`? You likely need to set the HBASE_CONF_DIR environment variable to the correct path for the Ambari Metrics System's (AMS) HBase instance.

I'd recommend that you reach out to customer support if you're experiencing the same problem repeatedly. It may be a known issue with the version of Ambari in-use.

[#8]

I agree on deleting the hbase data, however it's the only fix I've found so far. No argument from me.

"medical kit logo" - can you clarify what do you mean by this?" --- Turns out it's the maintenance mode indicator. AMC was in maintenance mode. I'd never used Maintenance Mode and was unfamiliar with the indicator. I get the bozo award this week... Sigh.