Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala concurrency test

Impala concurrency test

New Contributor

We are trying to integrate our web application with Impala as the back end DB, however we run into issues with concurrent access.

 

Currently we assigned 120G memory to Impala on our 7 nodes cluster, the concurrent query speed drops fast when increasing the number concurrent users. However we see the peak memory never exceeded 60G on each node. the CPU is only used  less than 30%, and the network throuput is only used 300M/sec eventhough our network speed is 4G/sec.  The max concurrent queries we can send is 64. Our web application requires over hundred concurrent queries.

 

We wonder if anyone can advise on any other configuration we should change to remove the bottleneck on the concurrency barrier?

 

3 REPLIES 3

Re: Impala concurrency test

Master Collaborator

Are you sending all queries to the same coordinator? Probably want to load balance them.

 

You can increase the max number of concurrent connections by changing the --fe_service_threads startup option.

 

Let us know how your experience goes!

Re: Impala concurrency test

New Contributor

Thanks Alex.

 

Both suggestions have help the concurrency performance. However our CPU usage is still below 50%. How can we fully use our CPU. I have read a concurrency test from Cloudera, there the CPU usage is almost 90% for the test.

Here is the link; http://blog.cloudera.com/blog/2015/09/how-impala-scales-for-business-intelligence-new-test-results/

Judy

 

 

Re: Impala concurrency test

New Contributor

You could look into MT_DOP query option to maximize your CPU usage.

https://www.cloudera.com/documentation/enterprise/latest/topics/impala_mt_dop.html

 

Cheers,

Lucas