11-16-2017 12:14 PM
We are trying to integrate our web application with Impala as the back end DB, however we run into issues with concurrent access.
Currently we assigned 120G memory to Impala on our 7 nodes cluster, the concurrent query speed drops fast when increasing the number concurrent users. However we see the peak memory never exceeded 60G on each node. the CPU is only used less than 30%, and the network throuput is only used 300M/sec eventhough our network speed is 4G/sec. The max concurrent queries we can send is 64. Our web application requires over hundred concurrent queries.
We wonder if anyone can advise on any other configuration we should change to remove the bottleneck on the concurrency barrier?
11-17-2017 09:17 PM
Are you sending all queries to the same coordinator? Probably want to load balance them.
You can increase the max number of concurrent connections by changing the --fe_service_threads startup option.
Let us know how your experience goes!
11-28-2017 02:54 PM
Both suggestions have help the concurrency performance. However our CPU usage is still below 50%. How can we fully use our CPU. I have read a concurrency test from Cloudera, there the CPU usage is almost 90% for the test.