Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Getting Error : Spilling has been disabled due to no usable scratch space

avatar

Hello,

 

 

I am using : impalad version 2.7.0-cdh5.9.0  and for few days my impala query is getting failed with the error :

java.sql.SQLException:

Spilling has been disabled due to no usable scratch space. Please specify a usable scratch space location via the --scratch_dirs impalad flag.

 

the weired thing is that i have done all the possible  solutions as below,  but still why i am getting the error :

 

1. have created /tmp//tmp/impala-scratch directory on all the impalad nodes.

2. have created /tmp/impala-scratch on hdfs also

3. have created a folder /data/impala-scratch and startimg impala-server like below :

./impala-server start --scratch_dirs="/data/impala-scratch"

 

and scratch folders have enough space and write permission. kindly help to resolve this problem , this problem is in PROD env.

1 ACCEPTED SOLUTION

avatar

If you're starting Impala from the command line like that you can configure  flags and environment variables with the /etc/default/impala -  https://www.cloudera.com/documentation/enterprise/latest/topics/impala_processes.html#starting_via_c... .  The relevant variable in that file is IMPALA_SERVER_ARGS.

 

(If anyone else reads this, if you're using Cloudera Manager you can configure the scratch directories through the UI. You probably won't have to since CM does a pretty good job of autoconfiguring scratch directories).

View solution in original post

5 REPLIES 5

avatar

The Impala daemon wasn't able to set up the scratch directories during startup. The reason will be logged in one of the impalad*.WARNING logs, probably one of the first messages in there.

avatar

I cant see any error related to scratch dir setup.i can see only :

Could not remove and recreate directory /tmp/impala-scratch: cannot use it for scratch. Error was: Encountered error removing directory '/tmp/impala-scratch': Operation not permitted.

 

looks there is no impact of provided path while starting. can you tell if my command is correct or do i need to use other command for that ?

avatar

The user that Impala is running under needs to be able to remove and recreate the scratch directory at startup (i.e. /tmp/impala-scratch). This is done to ensure that the directory is free of old files and that Impala has ownership of the directory.

 

Based on the log message that user doesn't have the required permissions to do that.

 

I suspect if you just delete that directory and let Impala create it at startup, that will solve your problem.

avatar

my point is why impala is not trying to use the directory which i am providing like this : 

./impala-server start --scratch_dirs="/data/impala-scratch"

 

why its still expecting /tmp/impala-scratch" ?

 

How to resolve it ?

avatar

If you're starting Impala from the command line like that you can configure  flags and environment variables with the /etc/default/impala -  https://www.cloudera.com/documentation/enterprise/latest/topics/impala_processes.html#starting_via_c... .  The relevant variable in that file is IMPALA_SERVER_ARGS.

 

(If anyone else reads this, if you're using Cloudera Manager you can configure the scratch directories through the UI. You probably won't have to since CM does a pretty good job of autoconfiguring scratch directories).