question Re: Getting Error : Spilling has been disabled due to no usable scratch space in Archives of Support Questions (Read Only)

Getting Error : Spilling has been disabled due to no usable scratch space

hrishi1dypim — Fri, 16 Sep 2022 12:15:34 GMT

Hello,

I am using : impalad version 2.7.0-cdh5.9.0 and for few days my impala query is getting failed with the error :

java.sql.SQLException:

Spilling has been disabled due to no usable scratch space. Please specify a usable scratch space location via the --scratch_dirs impalad flag.

the weired thing is that i have done all the possible solutions as below, but still why i am getting the error :

1. have created /tmp//tmp/impala-scratch directory on all the impalad nodes.

2. have created /tmp/impala-scratch on hdfs also

3. have created a folder /data/impala-scratch and startimg impala-server like below :

./impala-server start --scratch_dirs="/data/impala-scratch"

and scratch folders have enough space and write permission. kindly help to resolve this problem , this problem is in PROD env.

Re: Getting Error : Spilling has been disabled due to no usable scratch space

Tim Armstrong — Tue, 19 Sep 2017 00:15:46 GMT

The Impala daemon wasn't able to set up the scratch directories during startup. The reason will be logged in one of the impalad*.WARNING logs, probably one of the first messages in there.

Re: Getting Error : Spilling has been disabled due to no usable scratch space

hrishi1dypim — Tue, 19 Sep 2017 07:48:12 GMT

I cant see any error related to scratch dir setup.i can see only :

Could not remove and recreate directory /tmp/impala-scratch: cannot use it for scratch. Error was: Encountered error removing directory '/tmp/impala-scratch': Operation not permitted.

looks there is no impact of provided path while starting. can you tell if my command is correct or do i need to use other command for that ?

Re: Getting Error : Spilling has been disabled due to no usable scratch space

Tim Armstrong — Tue, 19 Sep 2017 16:25:58 GMT

The user that Impala is running under needs to be able to remove and recreate the scratch directory at startup (i.e. /tmp/impala-scratch). This is done to ensure that the directory is free of old files and that Impala has ownership of the directory.

Based on the log message that user doesn't have the required permissions to do that.

I suspect if you just delete that directory and let Impala create it at startup, that will solve your problem.

Re: Getting Error : Spilling has been disabled due to no usable scratch space

hrishi1dypim — Wed, 20 Sep 2017 06:41:35 GMT

my point is why impala is not trying to use the directory which i am providing like this :

./impala-server start --scratch_dirs="/data/impala-scratch"

why its still expecting /tmp/impala-scratch" ?

How to resolve it ?

Re: Getting Error : Spilling has been disabled due to no usable scratch space

Tim Armstrong — Wed, 20 Sep 2017 07:19:02 GMT

If you're starting Impala from the command line like that you can configure flags and environment variables with the /etc/default/impala - https://www.cloudera.com/documentation/enterprise/latest/topics/impala_processes.html#starting_via_cmdline . The relevant variable in that file is IMPALA_SERVER_ARGS.

(If anyone else reads this, if you're using Cloudera Manager you can configure the scratch directories through the UI. You probably won't have to since CM does a pretty good job of autoconfiguring scratch directories).