Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NO DATA on hdfs directory

NO DATA on hdfs directory

Explorer

Hi, I am using cloudra for managing my hadoop and hbase, I have recently got one strange issue on hbase and i was not able to resolve that issue. Issue i faced is as follows.
I am runing cloudra hadoop/hbase on a single cluster machine, the machine is having 1 HMaster and 1Region server. Hbase was working fine for the past 6 months from its installion date. Couple of days back the Hmaster went down.When checked the logs file i found ERROR -->
" FATAL org.apache.hadoop.hbase.master.HMaster: Failed to become active master
org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x".
One further analysis . I had checked "Hadoop" directory and logs and found that hadoop data is empty all my datanode and namenode data is gone. I had got below error -->
"/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin$ ./hadoop fs -ls /
Found 2 items
drwxr-xr-x - hdfs supergroup 0 2017-01-06 17:31 /NODATA4U_SECUREYOUR**bleep**".

I did not change any system setting or updates on the machine. but i looks like some how i have lost all my hadoop/hbase data.

Is the above behaviour is due to "cloudra" or what are the steps i need to execute to recover from the error

10 REPLIES 10

Re: NO DATA on hdfs directory

Super Collaborator

Hi,

 

I may be wrong but from the look of the only directory available "NODATA4U_SECUREYOUR**bleep**" maybe you could consider that your cluster was "hacked" by someone ?

Re: NO DATA on hdfs directory

Champion

I think this is an issue with Cloudera, someone raised the similar issue (/NODATA4U_SECUREYOUR**bleep**) with CDH 5.9 (Link provided below) and now the same issue raised on CDH 5.8 by a different person.

 

https://community.cloudera.com/t5/Hadoop-101-Training-Quickstart/All-HDFS-directories-auto-deleted-e...

 

Cloudera Team, any idea?

Re: NO DATA on hdfs directory

Community Manager

Thank you for reporting this situation. We take these issues seriously and are looking into it. We will report back when we have more information.



Cy Jervis, Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum

Re: NO DATA on hdfs directory

Explorer

Yes, One of my other colleuge is also using cloudera and he is also facing the same problem . we actually dont have any idea for the root cause of the problem. We are using cloudera managed hadoop/hbase in production environment so we are more cautious. 

 

cloudera team have to suggest any work around for this.

Re: NO DATA on hdfs directory

Community Manager

Please see the latest Cloudera advisory on the topic for more details on the issue.



Cy Jervis, Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum

Re: NO DATA on hdfs directory

Champion

@cjervis: Thank you

 

@MikeRaj: Is your environment kerberized? can you pls confirm... In the other link, she confirmed that its a non-kerberized 

Re: NO DATA on hdfs directory

Explorer

My environment is not kerberized.

Re: NO DATA on hdfs directory

Super Collaborator

I don't know of any bug in Hadoop that could cause what you are experiencing.

You should really consider my first hypothesis.

 

You should be able to obtain some information by reading / investigating the logs if still available (logs should help you identify the issue even if it is not a hack).

 

 

 

 

Highlighted

Re: NO DATA on hdfs directory

Community Manager

We have just published a new Engineering blog post How to secure ‘Internet exposed’ Apache Hadoop that may be of interest.



Cy Jervis, Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum