Yarn Node Managers shutting down

karan_alang1 — Thu, 13 Jul 2017 03:12:16 GMT

Hi All,

I've HDP 2.5.x, and i'm facing issue with the Node Managers going down.

Initially, the error was - /tmp/keystore.jks was not available.. so i generated the keystore.jks file again.

Now, i'm getting the following error in starting Yarn Node Managers

any ideas what needs to be done ?

------------------------------------------------

java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect 
at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:504)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:178)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:457)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)Caused by: java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)at java.security.KeyStore.load(KeyStore.java:1445)at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:173)at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)at org.apache.hadoop.mapred.ShuffleHandler$HttpPipelineFactory.<init>(ShuffleHandler.java:757)at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:502)... 11 moreCaused by: java.security.UnrecoverableKeyException: Password verification failedat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)... 19 more2017-07-12 20:05:18,245 INFOtimeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:run(416)) - Closing HadoopTimelineMetricSink. Flushing metrics to collector...2017-07-12 20:05:18,249 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NodeManager at nwk2-bdp-hadoop-08.gdcs-qa.apple.com/10.78.130.135

************************************************************/

Re: Yarn Node Managers shutting down

karan_alang1 — Fri, 14 Jul 2017 02:20:55 GMT

i changed the password in Ambari for the truststore & keystore, and it is working fine now.

question Yarn Node Managers shutting down in Archives of Support Questions (Read Only)

Yarn Node Managers shutting down

Re: Yarn Node Managers shutting down