Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Improving performmance - CDH5

Improving performmance - CDH5

Explorer

Hi,

 

I'm doing a check list performance on my client's cluster and I'm following this link:

 

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_admin_performance.h...

 

However, the sectioni Improving Performance in Shuffle Handler and IFile Reader tells:

 

  • To enable this feature for YARN, set the mapreduce.shuffle.manage.os.cache property to true (default). To further tune performance, adjust the value of the mapreduce.shuffle.readahead.bytes property. The default value is 4MB.
  • To enable this feature for MRv1, set the mapred.tasktracker.shuffle.fadvise property to true (default). To further tune performance, adjust the value of the mapred.tasktracker.shuffle.readahead.bytes property. The default value is 4MB.

Once that I have CM UI to administer the cluster, I'm not able to find these options on input box search on the top bar navigation.

 

I'm wondering whether these options is still valid.

 

Should I configure it in *-site.xml conf files?

 

Thanks

 

Fabricio

 

 

1 REPLY 1

Re: Improving performmance - CDH5

Super Collaborator

Yes the setting is still valid for YARN in the current release.

Since it is not something that is normally is set we do not have it in the CM UI. You can set it using the following steps:

 

  1. Navigate to the YARN configuration page (CM -> Clusters -> YARN -> Configuration)
  2. Naviate to NodeManager Default Group -> Advanced -> NodeManager Advanced Configuration Snippet (Safety Valve) for mapred-site.xml
  3. In the field marked Value set the property with the required size.
  4. Save changes and you will need to restart the NM's

Example of the property:

 

<property>
<name>mapreduce.shuffle.readahead.bytes</name>
<value>12345678</value>
</property>
 
The shuffle service runs as part of the NM and this is a setting for the shuffle handler service.
 
Wilfred