Skip to main content
JoaquinS
JoaquinS
Gold ⭐️
January 9, 2019
Solved

Scratch file generation information

Hello,

 

A simple question:

 

How can I know which queries generate Scratch files?

I'm inspecting the Impalad logs and I couldn't find any information about the scratch file generation.

 

Regards,

Silva

Tim Armstrong
Tim ArmstrongAnswer
Platinum ⭐️
January 10, 2019
The Cloudera Manager queries page has the bytes spilled to disk as one of the metrics it tracks per query. Also in CM, there's a "Cluster utilization report" that has some aggregate information about how much data is spilled to disk over longer time windows. Also, if you're looking at the scratch files themselves the query ID is embedded in the file name (although that's an implementation detail and could change in the future).
View in thread

4 replies

Tim Armstrong
Platinum ⭐️
Tim Armstrong
Platinum ⭐️
January 9, 2019
They're used for spill to disk - see https://www.cloudera.com/documentation/enterprise/latest/topics/impala_scalability.html#spill_to_disk
Translate
JoaquinS
JoaquinSAuthor
Gold ⭐️
January 10, 2019

But I need to know which specific queries spills into disk, generating the scratch files. Is possible to have that kind of information?.

Translate
Tim Armstrong
Platinum ⭐️
Tim ArmstrongAnswer
Platinum ⭐️
January 10, 2019
The Cloudera Manager queries page has the bytes spilled to disk as one of the metrics it tracks per query. Also in CM, there's a "Cluster utilization report" that has some aggregate information about how much data is spilled to disk over longer time windows. Also, if you're looking at the scratch files themselves the query ID is embedded in the file name (although that's an implementation detail and could change in the future).
Translate
Terms & ConditionsAccessibility statement