Hi, I am using cloudra for managing my hadoop and hbase, I have recently got one strange issue on hbase and i was not able to resolve that issue. Issue i faced is as follows.
I am runing cloudra hadoop/hbase on a single cluster machine, the machine is having 1 HMaster and 1Region server. Hbase was working fine for the past 6 months from its installion date. Couple of days back the Hmaster went down.When checked the logs file i found ERROR -->
" FATAL org.apache.hadoop.hbase.master.HMaster: Failed to become active master
org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x".
One further analysis . I had checked "Hadoop" directory and logs and found that hadoop data is empty all my datanode and namenode data is gone. I had got below error -->
"/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin$ ./hadoop fs -ls /
Found 2 items
drwxr-xr-x - hdfs supergroup 0 2017-01-06 17:31 /NODATA4U_SECUREYOUR**bleep**".
I did not change any system setting or updates on the machine. but i looks like some how i have lost all my hadoop/hbase data.
Is the above behaviour is due to "cloudra" or what are the steps i need to execute to recover from the error
I may be wrong but from the look of the only directory available "NODATA4U_SECUREYOUR**bleep**" maybe you could consider that your cluster was "hacked" by someone ?
I think this is an issue with Cloudera, someone raised the similar issue (/NODATA4U_SECUREYOUR**bleep**) with CDH 5.9 (Link provided below) and now the same issue raised on CDH 5.8 by a different person.
Cloudera Team, any idea?
Thank you for reporting this situation. We take these issues seriously and are looking into it. We will report back when we have more information.
Yes, One of my other colleuge is also using cloudera and he is also facing the same problem . we actually dont have any idea for the root cause of the problem. We are using cloudera managed hadoop/hbase in production environment so we are more cautious.
cloudera team have to suggest any work around for this.
Please see the latest Cloudera advisory on the topic for more details on the issue.
I don't know of any bug in Hadoop that could cause what you are experiencing.
You should really consider my first hypothesis.
You should be able to obtain some information by reading / investigating the logs if still available (logs should help you identify the issue even if it is not a hack).
We have just published a new Engineering blog post How to secure ‘Internet exposed’ Apache Hadoop that may be of interest.