Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Insufficient space for shared memory file

Insufficient space for shared memory file

New Contributor

Hi All,

When i am ruining hive quries some times i am facing error: Insufficient space for shared memory file . After few tries it throws

Exception in thread "main" java.io.IOException: No space left on device

Help Appreciated .

2 REPLIES 2

Re: Insufficient space for shared memory file

New Contributor

Hi Satish,

You are running out of disk space on which ever node this job runs on. You can check from the ResourceManager UI to track it down. Check your nodes disk space & ensure that the temporary locations for hive, (& potentially YARN) have enough space to process the spill.

Dave

Re: Insufficient space for shared memory file

Mentor

@Satish Anjaneyappa

I think when you are running the hive query it's generating some temporary files in HDFS.

Can you check the remaining space in hdfs

$ hdfs dfsadmin -report

When you delete files in hadoop it drops the files to the /.Trash but does not remove them. If you are sure of the files you want to delete its always a good idea to use -skipTrash option

$ hdfs dfs -rm -skipTrash  /path/to/file

This will skip the trash

However, if you do not add the -skipTrash flag, files are stored in a trash folder, which by default is:

/user/hdfs/.Trash

Empty the HDFS Trash by running

$ hdfs dfs -expunge

HTH