01-11-2017 03:20 AM
Hi, I am using cloudra for managing my hadoop and hbase, I have recently got one strange issue on hbase and i was not able to resolve that issue. Issue i faced is as follows.
I am runing cloudra hadoop/hbase on a single cluster machine, the machine is having 1 HMaster and 1Region server. Hbase was working fine for the past 6 months from its installion date. Couple of days back the Hmaster went down.When checked the logs file i found ERROR -->
" FATAL org.apache.hadoop.hbase.master.HMaster: Failed to become active master
org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x".
One further analysis . I had checked "Hadoop" directory and logs and found that hadoop data is empty all my datanode and namenode data is gone. I had got below error -->
"/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin$ ./hadoop fs -ls /
Found 2 items
drwxr-xr-x - hdfs supergroup 0 2017-01-06 17:31 /NODATA4U_SECUREYOUR**bleep**".
I did not change any system setting or updates on the machine. but i looks like some how i have lost all my hadoop/hbase data.
Is the above behaviour is due to "cloudra" or what are the steps i need to execute to recover from the error
01-11-2017 07:40 AM - edited 01-11-2017 07:40 AM
I may be wrong but from the look of the only directory available "NODATA4U_SECUREYOUR**bleep**" maybe you could consider that your cluster was "hacked" by someone ?
01-11-2017 09:20 AM
I think this is an issue with Cloudera, someone raised the similar issue (/NODATA4U_SECUREYOUR**bleep**) with CDH 5.9 (Link provided below) and now the same issue raised on CDH 5.8 by a different person.
Cloudera Team, any idea?
01-11-2017 03:30 PM
Thank you for reporting this situation. We take these issues seriously and are looking into it. We will report back when we have more information.
01-12-2017 11:11 PM
Yes, One of my other colleuge is also using cloudera and he is also facing the same problem . we actually dont have any idea for the root cause of the problem. We are using cloudera managed hadoop/hbase in production environment so we are more cautious.
cloudera team have to suggest any work around for this.
01-18-2017 10:52 AM
Please see the latest Cloudera advisory on the topic for more details on the issue.
01-19-2017 02:20 AM
I don't know of any bug in Hadoop that could cause what you are experiencing.
You should really consider my first hypothesis.
You should be able to obtain some information by reading / investigating the logs if still available (logs should help you identify the issue even if it is not a hack).