Insufficient space for shared memory file

[#1]

Hi All,

When i am ruining hive quries some times i am facing error: Insufficient space for shared memory file . After few tries it throws

Exception in thread "main" java.io.IOException: No space left on device

Help Appreciated .

[#2]

Hi Satish,

You are running out of disk space on which ever node this job runs on. You can check from the ResourceManager UI to track it down. Check your nodes disk space & ensure that the temporary locations for hive, (& potentially YARN) have enough space to process the spill.

Dave

[#3]

@Satish Anjaneyappa

I think when you are running the hive query it's generating some temporary files in HDFS.

Can you check the remaining space in hdfs

$ hdfs dfsadmin -report

When you delete files in hadoop it drops the files to the /.Trash but does not remove them. If you are sure of the files you want to delete its always a good idea to use -skipTrash option

$ hdfs dfs -rm -skipTrash  /path/to/file

This will skip the trash

However, if you do not add the -skipTrash flag, files are stored in a trash folder, which by default is:

/user/hdfs/.Trash

Empty the HDFS Trash by running

$ hdfs dfs -expunge

HTH