Hi All,
I've HDP 2.5.x, and i'm facing issue with the Node Managers going down.
Initially, the error was - /tmp/keystore.jks was not available.. so i generated the keystore.jks file again.
Now, i'm getting the following error in starting Yarn Node Managers
any ideas what needs to be done ?
------------------------------------------------
java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect
at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:504)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:178)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:457)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)Caused by: java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)at java.security.KeyStore.load(KeyStore.java:1445)at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:173)at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)at org.apache.hadoop.mapred.ShuffleHandler$HttpPipelineFactory.<init>(ShuffleHandler.java:757)at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:502)... 11 moreCaused by: java.security.UnrecoverableKeyException: Password verification failedat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)... 19 more2017-07-12 20:05:18,245 INFOtimeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:run(416)) - Closing HadoopTimelineMetricSink. Flushing metrics to collector...2017-07-12 20:05:18,249 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NodeManager at nwk2-bdp-hadoop-08.gdcs-qa.apple.com/10.78.130.135
************************************************************/
