Created 07-16-2018 07:22 AM
Hi All,
When i am ruining hive quries some times i am facing error: Insufficient space for shared memory file . After few tries it throws
Exception in thread "main" java.io.IOException: No space left on device
Help Appreciated .
Created 07-16-2018 09:42 AM
Hi Satish,
You are running out of disk space on which ever node this job runs on. You can check from the ResourceManager UI to track it down. Check your nodes disk space & ensure that the temporary locations for hive, (& potentially YARN) have enough space to process the spill.
Dave
Created 07-16-2018 09:49 AM
I think when you are running the hive query it's generating some temporary files in HDFS.
Can you check the remaining space in hdfs
$ hdfs dfsadmin -report
When you delete files in hadoop it drops the files to the /.Trash but does not remove them. If you are sure of the files you want to delete its always a good idea to use -skipTrash option
$ hdfs dfs -rm -skipTrash /path/to/file
This will skip the trash
However, if you do not add the -skipTrash flag, files are stored in a trash folder, which by default is:
/user/hdfs/.Trash
Empty the HDFS Trash by running
$ hdfs dfs -expunge
HTH