Reply
Explorer
Posts: 6
Registered: ‎01-11-2017

NO DATA on hdfs directory

Hi, I am using cloudra for managing my hadoop and hbase, I have recently got one strange issue on hbase and i was not able to resolve that issue. Issue i faced is as follows.
I am runing cloudra hadoop/hbase on a single cluster machine, the machine is having 1 HMaster and 1Region server. Hbase was working fine for the past 6 months from its installion date. Couple of days back the Hmaster went down.When checked the logs file i found ERROR -->
" FATAL org.apache.hadoop.hbase.master.HMaster: Failed to become active master
org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x".
One further analysis . I had checked "Hadoop" directory and logs and found that hadoop data is empty all my datanode and namenode data is gone. I had got below error -->
"/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/bin$ ./hadoop fs -ls /
Found 2 items
drwxr-xr-x - hdfs supergroup 0 2017-01-06 17:31 /NODATA4U_SECUREYOUR**bleep**".

I did not change any system setting or updates on the machine. but i looks like some how i have lost all my hadoop/hbase data.

Is the above behaviour is due to "cloudra" or what are the steps i need to execute to recover from the error

Posts: 173
Topics: 8
Kudos: 19
Solutions: 19
Registered: ‎07-16-2015

Re: NO DATA on hdfs directory

[ Edited ]

Hi,

 

I may be wrong but from the look of the only directory available "NODATA4U_SECUREYOUR**bleep**" maybe you could consider that your cluster was "hacked" by someone ?

Posts: 388
Topics: 11
Kudos: 60
Solutions: 34
Registered: ‎09-02-2016

Re: NO DATA on hdfs directory

I think this is an issue with Cloudera, someone raised the similar issue (/NODATA4U_SECUREYOUR**bleep**) with CDH 5.9 (Link provided below) and now the same issue raised on CDH 5.8 by a different person.

 

https://community.cloudera.com/t5/Hadoop-101-Training-Quickstart/All-HDFS-directories-auto-deleted-e...

 

Cloudera Team, any idea?

Posts: 604
Kudos: 68
Solutions: 36
Registered: ‎04-06-2015

Re: NO DATA on hdfs directory

Thank you for reporting this situation. We take these issues seriously and are looking into it. We will report back when we have more information.

 




Cy Jervis, Community Manager - I'm not an expert but will supply relevant content from time to time. :)

Learn more about the Cloudera Community:


Terms of Service


Community Guidelines


How to use the forum

Explorer
Posts: 6
Registered: ‎01-11-2017

Re: NO DATA on hdfs directory

Hi mathieu,

 

My cluster was not hacked. i am working in a secured environment.

Thank for you.

Explorer
Posts: 6
Registered: ‎01-11-2017

Re: NO DATA on hdfs directory

Yes, One of my other colleuge is also using cloudera and he is also facing the same problem . we actually dont have any idea for the root cause of the problem. We are using cloudera managed hadoop/hbase in production environment so we are more cautious. 

 

cloudera team have to suggest any work around for this.

Highlighted
Posts: 604
Kudos: 68
Solutions: 36
Registered: ‎04-06-2015

Re: NO DATA on hdfs directory

Please see the latest Cloudera advisory on the topic for more details on the issue.

 




Cy Jervis, Community Manager - I'm not an expert but will supply relevant content from time to time. :)

Learn more about the Cloudera Community:


Terms of Service


Community Guidelines


How to use the forum

Posts: 388
Topics: 11
Kudos: 60
Solutions: 34
Registered: ‎09-02-2016

Re: NO DATA on hdfs directory

@cjervis: Thank you

 

@MikeRaj: Is your environment kerberized? can you pls confirm... In the other link, she confirmed that its a non-kerberized 

Explorer
Posts: 6
Registered: ‎01-11-2017

Re: NO DATA on hdfs directory

My environment is not kerberized.

Posts: 173
Topics: 8
Kudos: 19
Solutions: 19
Registered: ‎07-16-2015

Re: NO DATA on hdfs directory

I don't know of any bug in Hadoop that could cause what you are experiencing.

You should really consider my first hypothesis.

 

You should be able to obtain some information by reading / investigating the logs if still available (logs should help you identify the issue even if it is not a hack).

 

 

 

 

Announcements