- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
GC duration concerning for jobtracker in CDH5.9 cluster
- Labels:
-
Cloudera Manager
-
MapReduce
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
We are getting warning (concerning) messages for mapreduce1 jobtracker frequently on our Cloudera CDH 5.9 cluster. Currently java heap size of jobtracker is 4.11GB and it's 14-node cluster.
Below is the warning message that appears regarding the garbage collection.
Test of whether this role spends too much time in garbage collection.
Concerning:Average time spent in garbage collection was 20.6 seconds(34.28%)per minute over the previous 5 minute(s).Warning threshold:30.00%
Created ‎11-13-2017 07:58 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Try the below, it may help you
Cm -> Yarn -> Configuration -> "Java Heap Size of NodeManager in Bytes" Get the current value like 1GB or 2GB, etc... Increase one extra GB, ex: if it is 1GB, increase it to 2GB
(or)
Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins
restart yarn as needed
Created ‎11-13-2017 07:58 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Try the below, it may help you
Cm -> Yarn -> Configuration -> "Java Heap Size of NodeManager in Bytes" Get the current value like 1GB or 2GB, etc... Increase one extra GB, ex: if it is 1GB, increase it to 2GB
(or)
Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins
restart yarn as needed
Created ‎11-13-2017 08:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎11-14-2017 12:40 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
> Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins
I'd not recommend doing this, as it only hides the problem - You should be concerned if there's excessive GC due to heap pressure, and the default alert window and thresholds covers it well. Only increase it if you're absolutely sure the GC spikes are temporary and workload-driven but non-concerning, which doesn't seem determined yet.
Also, if you want to check the heap usage before increasing the heap by any value, you may see it in the heap charts shown on the JobTracker instance page of CM.
Created ‎11-14-2017 01:41 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for inputs. I checked heap charts on jobtracker instance and it's hitting the maximum heap value frequently and then reducing to little less value.
Also, there has not been any change/increase in load.
I checked the jobtracker logs but couldn't find any pauses as logging is not enabled for GC.
Can you please let me know what are the history retention configurations of jobtracker?
Can you please suggest me how to identify the reason behind the GC taking significant time?
Thanks,
Priya