Reply
Expert Contributor
Posts: 109
Registered: ‎05-19-2016

Does hard restart of agent delete data?

[ Edited ]

Does hard restart of agent delete data?

 

I did a hard restart of my agent and now I do not see data in hadoop fs -ls /user/hue or hadoop fs -ls /user/hive where did it go? I also do not see my other users but only hue and hive. What do I do? Where did it go?

 

 I don't think data in hdfs should go anywhere with that. 

 

If I query my tables in hive, I keep getting 

 

The operation has no results. Help please!

 

 

I can't believe data could be deleted like that? How do I get that back?

 

The imports still seem to complete as oozie jobs did not throw the errors but I do not find the data anywhere

 

Cloudera Employee
Posts: 44
Registered: ‎08-13-2014

Re: Does hard restart of agent delete data?

I saw your post on StackOverflow, here's the answer I posted there for reference...

 

Doing a hard restart of the Cloudera Manager agent will not cause data loss, but will cause all of the Hadoop daemons to be restarted. A normal restart of the agent does not do this, so a hard restart is useful if you need to force a stop of all the running processes.

 

If you are seeing no data in HDFS following a restart check the status of the HDFS service in Cloudera Manager. It will tell you how much capacity is used in HDFS, the number of files and other metrics. If you're seeing no data it could be that your DataNodes have not been started. Check to see if this is the case and if your NameNode is still in safe mode.

Expert Contributor
Posts: 109
Registered: ‎05-19-2016

Re: Does hard restart of agent delete data?

@Jim Halfpenny : Thank you for your response. Both my data nodes have been restarted and are in good health. I had restarted cloudera-scm-server as well. Does that make any difference ?

 

Basically I restarted everything, all services. I am still sure HDFS has data since this is what I see in configured capacity for HDFS. 540.7 GiB/3.0 TiB. we have only two data nodes and had this much data only but hadoop fs -ls does not show any data or any directories. select * from hive tables returned no data. Could you please tell what is the possibility in this case?

Cloudera Employee
Posts: 44
Registered: ‎08-13-2014

Re: Does hard restart of agent delete data?

Do you see an error when you try to run hadoop fs -ls and if so what does it say? Are you able to read the files as well as view the directory listing with ls?

 

Regards,

Jim

Expert Contributor
Posts: 109
Registered: ‎05-19-2016

Re: Does hard restart of agent delete data?

Hey,

 

   In hadoop fs -ls , I get only hue and hive user and there is no data in there really that used to be there. I had created more users as well in the user directory. Those are not here either. 

 

 I have tried running hadoop fs -put source destination, I still see nothing in HDFS. It's like I am not able to see the HDFS that is actually there. I am sure data is still there but hdfs command is not listing it. I have tried using hue filebrowser as well. It's just hive and hue users that I see and there is not data in there

Expert Contributor
Posts: 109
Registered: ‎05-19-2016

Re: Does hard restart of agent delete data?

Cloudera Employee
Posts: 44
Registered: ‎08-13-2014

Re: Does hard restart of agent delete data?

Your hive data is likely to be in /user/hive/warehouse, from what you're saying this directory is no longer there, is that correct? If so then it's possible it has been deleted, but it's no likely to be due to the hard restart of the agent.

Does this cluster have Cloudera Navigator installed? If so you can use that to look at the audit events and see if and when the data was deleted. If not, check to see if you have HDFS audit logging enabled, this too will tell you if the data was deleted.

Were there any changes on the cluster prior to the loss of your data? I'm interested as to why you needed a agent hard restart, was this in response to some problem you had on your cluster?

Regards,
Jim
Expert Contributor
Posts: 109
Registered: ‎05-19-2016

Re: Does hard restart of agent delete data?

That's correct.  warehouse in hive is not there.

 I had to do a hard restart since the server was abruptly shut off and restarting the services was getting problematic as it was complaining of different agent id's and supervisord process. I thought doing a hard restart should reove stale configuratins.

 

I executed these commands

 

cloudera-scm-agent next_start_clean

 

 and hard_resttart both on agent and server

 

 

. I had two datanodes on different servers. Both of them have been formatted. HDFS space on the server looks like has been formatted. This is what I see in report:

 

 

Name: 10.10.10.9:50010 (warehouse.swtched.com)
Hostname: warehouse.swtched.com
Rack: /default
Decommission Status : Normal
Configured Capacity: 2372610195456 (2.16 TB)
DFS Used: 867430400 (827.25 MB)
Non DFS Used: 512833609728 (477.61 GB)
DFS Remaining: 1858909155328 (1.69 TB)
DFS Used%: 0.04%
DFS Remaining%: 78.35%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 2
Last contact: Mon Jan 09 11:12:43 IST 2017

 

 

But how come all the data nodes have been formatted?

Cloudera Employee
Posts: 44
Registered: ‎08-13-2014

Re: Does hard restart of agent delete data?

Hi,

When the server was shut off is it possible that the disks containing the data did not get mounted? This might explain why the data is not appearing, if the disks are not mounted then the blocks will not be available. Other than that I would recommend that you check you logs files to see if this provides any clues as to the cause of the problem.

 

Regards,

Jim

New Contributor
Posts: 3
Registered: ‎07-24-2017

Re: Does hard restart of agent delete data?

Hey,

 

It looks like i am facing the same problem i.e i restarted my cluster and i cannot find my hive warehouse directory anymore.I can see that the meta data is still present but the actual table data isnt present. Any info on what caused the problem and a solution if you found one?

Announcements