Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

nodemanager outofmemory and crash

Highlighted

nodemanager outofmemory and crash

New Contributor

env:

hadoop2.3cdh5.0.2

ibm jdk 1.7.0 (sr2)

powerlinux

symptom:

when I start nodemanager, after some hours (<24), it will crash, I paster the nodemanager process log:

2015-12-05 16:17:59,916 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_7582_01_000747
java.lang.IllegalArgumentException: disparate values
	at sun.misc.FDBigInt.quoRemIteration(FloatingDecimal.java:2939)
	at sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:922)
	at sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)
	at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)
	at java.util.Formatter.format(Formatter.java:2500)
	at java.util.Formatter.format(Formatter.java:2435)
	at java.lang.String.format(String.java:2153)
	at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)
	at org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(StringUtils.java:758)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:487)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)
2015-12-05 16:18:00,017 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_7582_01_000355
java.lang.IllegalArgumentException: disparate values
	at sun.misc.FDBigInt.quoRemIteration(FloatingDecimal.java:2939)
	at sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:922)
	at sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)
	at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)
	at java.util.Formatter.format(Formatter.java:2500)
	at java.util.Formatter.format(Formatter.java:2435)
	at java.lang.String.format(String.java:2153)
	at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)
	at org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(StringUtils.java:758)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:487)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)
2015-12-05 16:18:03,082 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_7582_01_000077
java.lang.ArrayIndexOutOfBoundsException
	at sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:848)
	at sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)
	at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)
	at java.util.Formatter.format(Formatter.java:2500)
	at java.util.Formatter.format(Formatter.java:2435)
	at java.lang.String.format(String.java:2153)
	at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)
	at org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(StringUtils.java:758)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:487)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)
2015-12-05 16:18:03,147 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_7582_01_000300
java.lang.ArrayIndexOutOfBoundsException
	at sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:848)
	at sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)
	at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)
	at java.util.Formatter.format(Formatter.java:2500)
	at java.util.Formatter.format(Formatter.java:2435)
	at java.lang.String.format(String.java:2153)
	at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)
	at org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(StringUtils.java:758)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:487)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)
2015-12-05 16:18:03,352 FATAL org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Container Monitor,5,main] threw an Error.
  Shutting down now...
java.lang.OutOfMemoryError: Java heapspace
	at java.util.HashMap.inflateTable(HashMap.java:342)
	at java.util.HashMap.<init>(HashMap.java:322)
	at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(ProcfsBasedProcessTree.java:154)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:390)
2015-12-05 16:18:03,354 INFO org.apache.hadoop.util.ExitUtil: Halt with status -1 Message: HaltException

please help me ,we even replace all the memory for this array.

or not, we even decide to replace the machine 's mainboard.

I want to find solution from software side.

 

 

1 REPLY 1

Re: nodemanager outofmemory and crash

Contributor

Just some observations from another user:

  • you're using quite old CDH version (there is 5.0.6 version already, if you need to stick with 5.0 branch; or you can use 5.5.1)
  • also it is recommended to use Oracle Java: CDH 5.0.x requirements (another JVM works fine, we're using OpenJDK for example, but Cloudera is throughfully tested and supported with the versions from Oracle)

Btw. some HW can detect failures in runtime (/var/log/mcelog, /var/log/kern.log, ...). Of course you should check it "separately" from the Hadoop, when in doubt (memtest, badblocks, ...).

 

Don't have an account?
Coming from Hortonworks? Activate your account here