Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to fix (workaround) TaskTracker shutdown due to Jetty bug?

avatar
New Contributor

I've been searching for a while and I don't find any information on how to fix or workaround the problem of TaskTracker shutting down due to the Jetty CPU usage bug.

 

We have Cloudera distribution cdh4.6.0 (which uses jetty-6.1.26.cloudera.2.jar and jetty-util-6.1.26.cloudera.2.jar).

 

TaskTracker shuts down after just a couple of minutes online in some of the nodes of our cluster. The following is written to the log file:

 

2015-02-13 15:33:05,223 FATAL org.apache.hadoop.mapred.JettyBugMonitor: ************************************************************
Jetty CPU usage: 1573657853.3%. This is greater than the fatal threshold mapred.tasktracker.jetty.cpu.threshold.fatal. Aborting JVM.
************************************************************
2015-02-13 15:33:05,459 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:

 

 

How can we fix this problem, i.e. prevent TaskTracker from shutting down or getting into a bad state?

Is there a newer Jetty version we can use that does not cause the problem?

 

Thank you.

Eduardo.

1 ACCEPTED SOLUTION

avatar
Maybe you're running into issues described here:
http://bugs.centos.org/view.php?id=6421

Try powering down your affected nodes, then starting them after a few minutes

Regards,
Gautam Gopalakrishnan

View solution in original post

2 REPLIES 2

avatar
Maybe you're running into issues described here:
http://bugs.centos.org/view.php?id=6421

Try powering down your affected nodes, then starting them after a few minutes

Regards,
Gautam Gopalakrishnan

avatar
Expert Contributor

Hi eafonsof

 

Updating your kernel version will resolve the issue most likely.

 

I updated mine to kernel 3.18.x and resolved the issue.