Could not create files in any configured scratch directories (--scratch_dirs=) on backend

[#1]

Hi All,

When I try to run a huge query at Impala that does a lot of aggregation & sorts, it throws up with the following error. The error is vague and of not much help. Our cluster has 32 impala daemons and all the nodes have been configured with 8 disks of 4TB each. We defined scratch directories on all the disks/nodes in Cloudera Manager. I tried looking at the logs as indicated in the error message but I couldn't find anything useful. 

Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'data405.prod.com:22000'. See logs for previous errors that may have prevented creating or writing scratch files.

Version of impala: 2.12.0-cdh5.15.0

Any pointers would be greatly helpful.

Thanks,
Ravi

[#2 · Accepted solution]

I'll assume that you have some directories configured and passed in via the --scratch_dirs argument (you can check the debug page on port 25000 or the impalad.INFO log to confirm the flag value).

Then what likely happened is that there's some reason the directories weren't usable. Any errors that prevent using the directories are logged at startup.

[#3]

Tim,

Thanks for the suggestion. I checked the startup logs for the impalad that is complaining and looks like there are some weird permission issues which we fixed and things are looking good. However there is one thing that is puzzling me. 

We have impala user allocated to impala_prod LDAP group, however, there is also a local user impala allocated to local group impala on the node running impalad. When we removed scratch dirs and fired up impalad, looks like scratch dirs are created with the user from local and group from LDAP. Can this possible?

- Ravi

[#4]

I unfortunately don't know too many of the details of LDAP. Impala doesn't do anything sophisticated to create the directories - it just mkdir() with S_IRWXU|S_IRWXG|S_IRWXO to create the impala-scratch subdirectory and any missing parent directories.