Reply
New Contributor
Posts: 2
Registered: ‎11-16-2017

Impala concurrency test

We are trying to integrate our web application with Impala as the back end DB, however we run into issues with concurrent access.

 

Currently we assigned 120G memory to Impala on our 7 nodes cluster, the concurrent query speed drops fast when increasing the number concurrent users. However we see the peak memory never exceeded 60G on each node. the CPU is only used  less than 30%, and the network throuput is only used 300M/sec eventhough our network speed is 4G/sec.  The max concurrent queries we can send is 64. Our web application requires over hundred concurrent queries.

 

We wonder if anyone can advise on any other configuration we should change to remove the bottleneck on the concurrency barrier?

 

Cloudera Employee
Posts: 290
Registered: ‎10-16-2013

Re: Impala concurrency test

Are you sending all queries to the same coordinator? Probably want to load balance them.

 

You can increase the max number of concurrent connections by changing the --fe_service_threads startup option.

 

Let us know how your experience goes!

Highlighted
New Contributor
Posts: 2
Registered: ‎11-16-2017

Re: Impala concurrency test

Thanks Alex.

 

Both suggestions have help the concurrency performance. However our CPU usage is still below 50%. How can we fully use our CPU. I have read a concurrency test from Cloudera, there the CPU usage is almost 90% for the test.

Here is the link; http://blog.cloudera.com/blog/2015/09/how-impala-scales-for-business-intelligence-new-test-results/

Judy

 

 

Announcements