We are facing Mapreduce2 > History Server intermitent shutdown.
Once we start the process it works for some time and then if throws the below log:
I saw in ambari History Server heap size = 900Mb, increase it to 1000Mb but still the same.
Please let me know any comment or if any detail is required to troubleshoot this.
2017-10-05 18:57:01,506 ERROR mortbay.log (Slf4jLog.java:warn(87)) - handle failed
2017-10-05 19:06:26,909 ERROR mortbay.log (Slf4jLog.java:warn(87)) - Error for /jmx
2017-10-05 19:00:19,549 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:createPassword(385)) - Creating password for identifier: owner=srvc_egi_hdev, renewer=yarn, realUser=oozie/hc4t02046.itcs.hpecorp.net@EAHPEDEV.ITCS.HPECORP.NET, issueDate=1507230017382, maxDate=1507834817382, sequenceNumber=1183674, masterKeyId=322, currentKey: 322
2017-10-05 18:57:26,058 WARN server.AuthenticationFilter (AuthenticationFilter.java:doFilter(530)) - AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
2017-10-05 19:16:11,695 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:createPassword(385)) - Creating password for identifier: owner=srvc_egi_hdev, renewer=yarn, realUser=oozie/hc4t02046.itcs.hpecorp.net@EAHPEDEV.ITCS.HPECORP.NET, issueDate=1507230971695, maxDate=1507835771695, sequenceNumber=1183675, masterKeyId=322, currentKey: 322
2017-10-05 19:16:11,694 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[Timer-1,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
2017-10-05 19:15:00,875 FATAL ipc.Server (Server.java:doRunLoop(870)) - Bug in read selector!
java.lang.OutOfMemoryError: Java heap space
2017-10-05 19:14:47,223 ERROR mortbay.log (Slf4jLog.java:warn(87)) - /jmx?qry=java.lang:type=OperatingSystem
java.lang.OutOfMemoryError: Java heap space
2017-10-05 19:16:11,698 WARN server.AuthenticationFilter (AuthenticationFilter.java:doFilter(530)) - AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
2017-10-05 19:16:11,698 WARN server.AuthenticationFilter (AuthenticationFilter.java:doFilter(530)) - AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
2017-10-05 19:14:42,796 ERROR mortbay.log (Slf4jLog.java:warn(87)) - Error for /jobhistory
java.lang.OutOfMemoryError: Java heap space
2017-10-05 19:10:41,467 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[IPC Client (137225802) connection to hc4t02044.itcs.hpecorp.net/16.202.10.51:8020 from jhs/hc4t02252.itcs.hpecorp.net@EAHPEDEV.ITCS.HPECORP.NET,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
2017-10-05 19:16:11,703 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
2017-10-05 19:09:22,111 ERROR mortbay.log (Slf4jLog.java:warn(87)) - handle failed
java.lang.OutOfMemoryError: Java heap space
2017-10-05 19:08:14,649 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[666280300@qtp-444127949-5,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
<br>
