question Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend in Support Questions

Could not create files in any configured scratch directories (--scratch_dirs=) on backend

munna143 — Fri, 16 Sep 2022 14:08:19 GMT

Hi All,

When I try to run a huge query at Impala that does a lot of aggregation & sorts, it throws up with the following error. The error is vague and of not much help. Our cluster has 32 impala daemons and all the nodes have been configured with 8 disks of 4TB each. We defined scratch directories on all the disks/nodes in Cloudera Manager. I tried looking at the logs as indicated in the error message but I couldn't find anything useful.

Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'data405.prod.com:22000'. See logs for previous errors that may have prevented creating or writing scratch files.

Version of impala: 2.12.0-cdh5.15.0

Any pointers would be greatly helpful.

Thanks,
Ravi

Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Tim Armstrong — Fri, 08 Feb 2019 16:51:18 GMT

I'll assume that you have some directories configured and passed in via the --scratch_dirs argument (you can check the debug page on port 25000 or the impalad.INFO log to confirm the flag value).

Then what likely happened is that there's some reason the directories weren't usable. Any errors that prevent using the directories are logged at startup.

Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend

munna143 — Fri, 08 Feb 2019 21:29:18 GMT

Tim,

Thanks for the suggestion. I checked the startup logs for the impalad that is complaining and looks like there are some weird permission issues which we fixed and things are looking good. However there is one thing that is puzzling me.

We have impala user allocated to impala_prod LDAP group, however, there is also a local user impala allocated to local group impala on the node running impalad. When we removed scratch dirs and fired up impalad, looks like scratch dirs are created with the user from local and group from LDAP. Can this possible?

- Ravi

Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Tim Armstrong — Fri, 08 Feb 2019 23:59:40 GMT

I unfortunately don't know too many of the details of LDAP. Impala doesn't do anything sophisticated to create the directories - it just mkdir() with S_IRWXU|S_IRWXG|S_IRWXO to create the impala-scratch subdirectory and any missing parent directories.