Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

control cpu usage by impala

control cpu usage by impala

New Contributor

How do we control the number of cores used by impala. Currently we are running yarn and impala work load but impala queries are cpu intensive and ends up using most of the available cpu cores on the system.

3 REPLIES 3

Re: control cpu usage by impala

New Contributor

Also we have set --num-scanner-threads to 52 assuming 1 thread per core. We have 28 core cpu with hyper threading enabled.

Re: control cpu usage by impala

Champion

@kratka

 

There are different methos, below is the default settings,

if you go to CM-> host -> select a host -> resources (menu)
It will show you how many resources (cpu, mem, etc) has to be allocated for Yarn, Impala, hdfs, etc per node

 

You can control them using the below,
a. CM -> Yarn -> Config -> click on "Nodemanager" (left) and "Resource Management" (left) -> Consider cpu or mem as needed
b. CM -> Impala -> Config -> Click on "Imapal Daemon" (left) and "Resource Management" (left) -> consider only cpu.shares, mem_limit as needed.

 

If no luck then you can use dynamic resource pooling and create different job queue for MR & Impala

Re: control cpu usage by impala

New Contributor

assigning resources under resources tab will be per host. We need to limit cpu at impala cluster level.

And we are not using static pools we are using dynamic pools cpu.shares will not be helpful i believe.