Getting Error : Spilling has been disabled due to no usable scratch space

[#1]

Hello,

I am using : impalad version 2.7.0-cdh5.9.0  and for few days my impala query is getting failed with the error :

java.sql.SQLException:

Spilling has been disabled due to no usable scratch space. Please specify a usable scratch space location via the --scratch_dirs impalad flag.

the weired thing is that i have done all the possible  solutions as below,  but still why i am getting the error :

1. have created /tmp//tmp/impala-scratch directory on all the impalad nodes.

2. have created /tmp/impala-scratch on hdfs also

3. have created a folder /data/impala-scratch and startimg impala-server like below :

./impala-server start --scratch_dirs="/data/impala-scratch"

and scratch folders have enough space and write permission. kindly help to resolve this problem , this problem is in PROD env.

[#2]

The Impala daemon wasn't able to set up the scratch directories during startup. The reason will be logged in one of the impalad*.WARNING logs, probably one of the first messages in there.

[#3]

I cant see any error related to scratch dir setup.i can see only :

Could not remove and recreate directory /tmp/impala-scratch: cannot use it for scratch. Error was: Encountered error removing directory '/tmp/impala-scratch': Operation not permitted.

looks there is no impact of provided path while starting. can you tell if my command is correct or do i need to use other command for that ?

[#4]

The user that Impala is running under needs to be able to remove and recreate the scratch directory at startup (i.e. /tmp/impala-scratch). This is done to ensure that the directory is free of old files and that Impala has ownership of the directory.

Based on the log message that user doesn't have the required permissions to do that.

I suspect if you just delete that directory and let Impala create it at startup, that will solve your problem.

[#5]

my point is why impala is not trying to use the directory which i am providing like this : 

./impala-server start --scratch_dirs="/data/impala-scratch"

why its still expecting /tmp/impala-scratch" ?

How to resolve it ?

[#6 · Accepted solution]

If you're starting Impala from the command line like that you can configure  flags and environment variables with the /etc/default/impala -  https://www.cloudera.com/documentation/enterprise/latest/topics/impala_processes.html#starting_via_c... .  The relevant variable in that file is IMPALA_SERVER_ARGS.

(If anyone else reads this, if you're using Cloudera Manager you can configure the scratch directories through the UI. You probably won't have to since CM does a pretty good job of autoconfiguring scratch directories).