Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Node Managers shutting down

Solved Go to solution
Highlighted

Yarn Node Managers shutting down

Expert Contributor

Hi All,

I've HDP 2.5.x, and i'm facing issue with the Node Managers going down.

Initially, the error was - /tmp/keystore.jks was not available.. so i generated the keystore.jks file again.

Now, i'm getting the following error in starting Yarn Node Managers

any ideas what needs to be done ?

------------------------------------------------

java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect 
at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:504)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:178)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:457)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)Caused by: java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)at java.security.KeyStore.load(KeyStore.java:1445)at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:173)at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)at org.apache.hadoop.mapred.ShuffleHandler$HttpPipelineFactory.<init>(ShuffleHandler.java:757)at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:502)... 11 moreCaused by: java.security.UnrecoverableKeyException: Password verification failedat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)... 19 more2017-07-12 20:05:18,245 INFOtimeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:run(416)) - Closing HadoopTimelineMetricSink. Flushing metrics to collector...2017-07-12 20:05:18,249 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NodeManager at nwk2-bdp-hadoop-08.gdcs-qa.apple.com/10.78.130.135

************************************************************/

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Yarn Node Managers shutting down

Expert Contributor

i changed the password in Ambari for the truststore & keystore, and it is working fine now.

1 REPLY 1

Re: Yarn Node Managers shutting down

Expert Contributor

i changed the password in Ambari for the truststore & keystore, and it is working fine now.

Don't have an account?
Coming from Hortonworks? Activate your account here