How to fix (workaround) TaskTracker shutdown due to Jetty bug?

eafonsof — Fri, 16 Sep 2022 09:21:38 GMT

I've been searching for a while and I don't find any information on how to fix or workaround the problem of TaskTracker shutting down due to the Jetty CPU usage bug.

We have Cloudera distribution cdh4.6.0 (which uses jetty-6.1.26.cloudera.2.jar and jetty-util-6.1.26.cloudera.2.jar).

TaskTracker shuts down after just a couple of minutes online in some of the nodes of our cluster. The following is written to the log file:

2015-02-13 15:33:05,223 FATAL org.apache.hadoop.mapred.JettyBugMonitor: ************************************************************
Jetty CPU usage: 1573657853.3%. This is greater than the fatal threshold mapred.tasktracker.jetty.cpu.threshold.fatal. Aborting JVM.
************************************************************
2015-02-13 15:33:05,459 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:

How can we fix this problem, i.e. prevent TaskTracker from shutting down or getting into a bad state?

Is there a newer Jetty version we can use that does not cause the problem?

Thank you.

Eduardo.

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

GautamG — Tue, 17 Feb 2015 04:09:44 GMT

Maybe you're running into issues described here:
http://bugs.centos.org/view.php?id=6421

Try powering down your affected nodes, then starting them after a few minutes

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

nauseous — Thu, 02 Apr 2015 17:47:39 GMT

Hi eafonsof

Updating your kernel version will resolve the issue most likely.

I updated mine to kernel 3.18.x and resolved the issue.

question Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug? in Archives of Support Questions (Read Only)

How to fix (workaround) TaskTracker shutdown due to Jetty bug?

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?

Re: How to fix (workaround) TaskTracker shutdown due to Jetty bug?