Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to fix (workaround) TaskTracker shutdown due to Jetty bug?

SOLVED Go to solution

How to fix (workaround) TaskTracker shutdown due to Jetty bug?

New Contributor

I've been searching for a while and I don't find any information on how to fix or workaround the problem of TaskTracker shutting down due to the Jetty CPU usage bug.

 

We have Cloudera distribution cdh4.6.0 (which uses jetty-6.1.26.cloudera.2.jar and jetty-util-6.1.26.cloudera.2.jar).

 

TaskTracker shuts down after just a couple of minutes online in some of the nodes of our cluster. The following is written to the log file:

 

2015-02-13 15:33:05,223 FATAL org.apache.hadoop.mapred.JettyBugMonitor: ************************************************************
Jetty CPU usage: 1573657853.3%. This is greater than the fatal threshold mapred.tasktracker.jetty.cpu.threshold.fatal. Aborting JVM.
************************************************************
2015-02-13 15:33:05,459 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:

 

 

How can we fix this problem, i.e. prevent TaskTracker from shutting down or getting into a bad state?

Is there a newer Jetty version we can use that does not cause the problem?

 

Thank you.

Eduardo.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

Maybe you're running into issues described here:
http://bugs.centos.org/view.php?id=6421

Try powering down your affected nodes, then starting them after a few minutes

Regards,
Gautam Gopalakrishnan
2 REPLIES 2

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

Maybe you're running into issues described here:
http://bugs.centos.org/view.php?id=6421

Try powering down your affected nodes, then starting them after a few minutes

Regards,
Gautam Gopalakrishnan
Highlighted

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

Expert Contributor

Hi eafonsof

 

Updating your kernel version will resolve the issue most likely.

 

I updated mine to kernel 3.18.x and resolved the issue.