Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

impala fails to keep up with performance under high load

impala fails to keep up with performance under high load

Contributor

We've few hundred users of our system and we see serious performance degradation when concurrent queries exceeds just mere 10+. I think it might be some configuration we're missing here either in config or the way we're managing the cluster.

 

AverageScannerThreadConcurrency affecting query performance seriously. The same query when run under bit of load ( just couple of other big queries are running ).

 

Same query that scan approximately 800 G of data runs fast without load vs super slow ( 20 min ) under load.

 

AverageScannerThreadConcurrency: 28.664101859697162 (fast)

AverageScannerThreadConcurrency: 1.5204863450230135 (slow)

 

Any suggestions on how can we we infludence scanner thread concurrency to improve HDFS scan ?

Don't have an account?
Coming from Hortonworks? Activate your account here