Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Solved Go to solution

Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Explorer

Hi All,

 

When I try to run a huge query at Impala that does a lot of aggregation & sorts, it throws up with the following error. The error is vague and of not much help. Our cluster has 32 impala daemons and all the nodes have been configured with 8 disks of 4TB each. We defined scratch directories on all the disks/nodes in Cloudera Manager. I tried looking at the logs as indicated in the error message but I couldn't find anything useful. 

 

Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'data405.prod.com:22000'. See logs for previous errors that may have prevented creating or writing scratch files.

 

Version of impala: 2.12.0-cdh5.15.0

 

 

Any pointers would be greatly helpful.

 

Thanks,
Ravi

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Master Collaborator

I'll assume that you have some directories configured and passed in via the --scratch_dirs argument (you can check the debug page on port 25000 or the impalad.INFO log to confirm the flag value).

 

Then what likely happened is that there's some reason the directories weren't usable. Any errors that prevent using the directories are logged at startup.

3 REPLIES 3
Highlighted

Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Master Collaborator

I'll assume that you have some directories configured and passed in via the --scratch_dirs argument (you can check the debug page on port 25000 or the impalad.INFO log to confirm the flag value).

 

Then what likely happened is that there's some reason the directories weren't usable. Any errors that prevent using the directories are logged at startup.

Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Explorer

Tim,

 

Thanks for the suggestion. I checked the startup logs for the impalad that is complaining and looks like there are some weird permission issues which we fixed and things are looking good. However there is one thing that is puzzling me. 

 

We have impala user allocated to impala_prod LDAP group, however, there is also a local user impala allocated to local group impala on the node running impalad. When we removed scratch dirs and fired up impalad, looks like scratch dirs are created with the user from local and group from LDAP. Can this possible?

 

- Ravi

Re: Could not create files in any configured scratch directories (--scratch_dirs=) on backend

Master Collaborator

I unfortunately don't know too many of the details of LDAP. Impala doesn't do anything sophisticated to create the directories - it just mkdir() with S_IRWXU|S_IRWXG|S_IRWXO to create the impala-scratch subdirectory and any missing parent directories.