Reply
New Contributor
Posts: 3
Registered: ‎02-13-2015
Accepted Solution

How to fix (workaround) TaskTracker shutdown due to Jetty bug?

I've been searching for a while and I don't find any information on how to fix or workaround the problem of TaskTracker shutting down due to the Jetty CPU usage bug.

 

We have Cloudera distribution cdh4.6.0 (which uses jetty-6.1.26.cloudera.2.jar and jetty-util-6.1.26.cloudera.2.jar).

 

TaskTracker shuts down after just a couple of minutes online in some of the nodes of our cluster. The following is written to the log file:

 

2015-02-13 15:33:05,223 FATAL org.apache.hadoop.mapred.JettyBugMonitor: ************************************************************
Jetty CPU usage: 1573657853.3%. This is greater than the fatal threshold mapred.tasktracker.jetty.cpu.threshold.fatal. Aborting JVM.
************************************************************
2015-02-13 15:33:05,459 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:

 

 

How can we fix this problem, i.e. prevent TaskTracker from shutting down or getting into a bad state?

Is there a newer Jetty version we can use that does not cause the problem?

 

Thank you.

Eduardo.

Cloudera Employee
Posts: 578
Registered: ‎01-20-2014

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

Maybe you're running into issues described here:
http://bugs.centos.org/view.php?id=6421

Try powering down your affected nodes, then starting them after a few minutes

Regards,
Gautam Gopalakrishnan
Highlighted
Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

Hi eafonsof

 

Updating your kernel version will resolve the issue most likely.

 

I updated mine to kernel 3.18.x and resolved the issue.

Announcements