01-07-2017 12:28 AM - edited 01-07-2017 01:02 AM
Does hard restart of agent delete data?
I did a hard restart of my agent and now I do not see data in hadoop fs -ls /user/hue or hadoop fs -ls /user/hive where did it go? I also do not see my other users but only hue and hive. What do I do? Where did it go?
I don't think data in hdfs should go anywhere with that.
If I query my tables in hive, I keep getting
The operation has no results. Help please!
I can't believe data could be deleted like that? How do I get that back?
The imports still seem to complete as oozie jobs did not throw the errors but I do not find the data anywhere
01-07-2017 07:17 AM
I saw your post on StackOverflow, here's the answer I posted there for reference...
Doing a hard restart of the Cloudera Manager agent will not cause data loss, but will cause all of the Hadoop daemons to be restarted. A normal restart of the agent does not do this, so a hard restart is useful if you need to force a stop of all the running processes.
If you are seeing no data in HDFS following a restart check the status of the HDFS service in Cloudera Manager. It will tell you how much capacity is used in HDFS, the number of files and other metrics. If you're seeing no data it could be that your DataNodes have not been started. Check to see if this is the case and if your NameNode is still in safe mode.
01-07-2017 07:25 AM
@Jim Halfpenny : Thank you for your response. Both my data nodes have been restarted and are in good health. I had restarted cloudera-scm-server as well. Does that make any difference ?
Basically I restarted everything, all services. I am still sure HDFS has data since this is what I see in configured capacity for HDFS. 540.7 GiB/3.0 TiB. we have only two data nodes and had this much data only but hadoop fs -ls does not show any data or any directories. select * from hive tables returned no data. Could you please tell what is the possibility in this case?
01-07-2017 08:54 AM
Do you see an error when you try to run hadoop fs -ls and if so what does it say? Are you able to read the files as well as view the directory listing with ls?
01-07-2017 10:16 AM
In hadoop fs -ls , I get only hue and hive user and there is no data in there really that used to be there. I had created more users as well in the user directory. Those are not here either.
I have tried running hadoop fs -put source destination, I still see nothing in HDFS. It's like I am not able to see the HDFS that is actually there. I am sure data is still there but hdfs command is not listing it. I have tried using hue filebrowser as well. It's just hive and hue users that I see and there is not data in there
01-08-2017 02:57 AM
01-08-2017 09:44 PM
That's correct. warehouse in hive is not there.
I had to do a hard restart since the server was abruptly shut off and restarting the services was getting problematic as it was complaining of different agent id's and supervisord process. I thought doing a hard restart should reove stale configuratins.
I executed these commands
and hard_resttart both on agent and server
. I had two datanodes on different servers. Both of them have been formatted. HDFS space on the server looks like has been formatted. This is what I see in report:
Name: 10.10.10.9:50010 (warehouse.swtched.com)
Decommission Status : Normal
Configured Capacity: 2372610195456 (2.16 TB)
DFS Used: 867430400 (827.25 MB)
Non DFS Used: 512833609728 (477.61 GB)
DFS Remaining: 1858909155328 (1.69 TB)
DFS Used%: 0.04%
DFS Remaining%: 78.35%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Last contact: Mon Jan 09 11:12:43 IST 2017
But how come all the data nodes have been formatted?
01-11-2017 05:57 AM
When the server was shut off is it possible that the disks containing the data did not get mounted? This might explain why the data is not appearing, if the disks are not mounted then the blocks will not be available. Other than that I would recommend that you check you logs files to see if this provides any clues as to the cause of the problem.
07-24-2017 01:37 PM
It looks like i am facing the same problem i.e i restarted my cluster and i cannot find my hive warehouse directory anymore.I can see that the meta data is still present but the actual table data isnt present. Any info on what caused the problem and a solution if you found one?