Support Questions
Find answers, ask questions, and share your expertise

Yarn Node Managers shutting down

Expert Contributor

Hi All,

I've HDP 2.5.x, and i'm facing issue with the Node Managers going down.

Initially, the error was - /tmp/keystore.jks was not available.. so i generated the keystore.jks file again.

Now, i'm getting the following error in starting Yarn Node Managers

any ideas what needs to be done ?

------------------------------------------------

java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect 
at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:504)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:178)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:457)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)Caused by: java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)at java.security.KeyStore.load(KeyStore.java:1445)at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:173)at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)at org.apache.hadoop.mapred.ShuffleHandler$HttpPipelineFactory.<init>(ShuffleHandler.java:757)at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:502)... 11 moreCaused by: java.security.UnrecoverableKeyException: Password verification failedat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)... 19 more2017-07-12 20:05:18,245 INFOtimeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:run(416)) - Closing HadoopTimelineMetricSink. Flushing metrics to collector...2017-07-12 20:05:18,249 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NodeManager at nwk2-bdp-hadoop-08.gdcs-qa.apple.com/10.78.130.135

************************************************************/

1 ACCEPTED SOLUTION

Expert Contributor

i changed the password in Ambari for the truststore & keystore, and it is working fine now.

View solution in original post

1 REPLY 1

Expert Contributor

i changed the password in Ambari for the truststore & keystore, and it is working fine now.

; ;