HDP 2.6.3 Install & Upgrade Issue

MindGlass — Wed, 08 Nov 2017 08:30:21 GMT

I'm upgrading hdp 2.6.2.14 to 2.6.3.0, but I'm stuck in this issue and cannot finalize upgrade procedure.

JAVA ENV : jdk 1.7

To my way of thinking, 'java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0' this issue is different version between jdk7 and jdk8. Maybe YarnShuffleService.class file is compiled by jdk7 compiler by apache's some developer.

I think, this issue should be fixed ASAP. I know jdk7 is deprecated in this HDP 2.6.3.0 version and it will be removed future version.

Nodemanager Log.

2017-11-08 09:17:51,530 INFOyarn.YarnShuffleService (YarnShuffleService.java:serviceInit(156)) - Started YARN shuffle service for Spark on port 7337. Authentication is not enabled.Registered executor file is /data/log/hadoop/hadoop-yarn/nodemanager/recovery-state/nm-aux-services/spark_shuffle/registeredExecutors.ldb

2017-11-08 09:17:51,532 INFOutil.ApplicationClassLoader (ApplicationClassLoader.java:<init>(105)) - classpath: [file:/usr/hdp/2.6.3.0-235/spark2/aux/spark-2.2.0.2.6.3.0-235-yarn-shuffle.jar]

2017-11-08 09:17:51,532 INFOutil.ApplicationClassLoader (ApplicationClassLoader.java:<init>(106)) - system classes: [java., javax.accessibility., javax.activation., javax.activity., javax.annotation., javax.annotation.processing., javax.crypto., javax.imageio., javax.jws., javax.lang.model., -javax.management.j2ee., javax.management., javax.naming., javax.net., javax.print., javax.rmi., javax.script., -javax.security.auth.message., javax.security.auth., javax.security.cert., javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools., javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., org.xml.sax., org.apache.commons.logging., org.apache.log4j., -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml, yarn-default.xml]

2017-11-08 09:17:51,534 FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager

java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:184)

at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:274)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:546)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)

2017-11-08 09:17:51,539 INFOimpl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping NodeManager metrics system...

2017-11-08 09:17:51,539 INFOimpl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.

2017-11-08 09:17:51,540 INFOimpl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NodeManager metrics system stopped.

2017-11-08 09:17:51,541 INFOimpl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown complete.

2017-11-08 09:17:51,545 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:

Re: HDP 2.6.3 Install & Upgrade Issue

jsensharma — Sun, 18 Aug 2019 08:41:42 GMT

@Peter Kim

ERROR:

java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0

From HDP 2.6.3 onwards JDK 1.8 Must to be used as codes are compiled agains this JDK for few components as JDK 1.7 is already EOL (End Of Life Few year back).

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_support-matrices/content/ch_matrices-hdp.html

So you should upgrade the JDK to 1.8 and then upgrade HDP. So please Install JDK 1.8 and set the JAVA_HOME accordingly. You can also setup JDK 1.8 via ambari using:

# ambari-server setup –j /usr/lib/jvm18

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/setup_options.html

Re: HDP 2.6.3 Install & Upgrade Issue

MindGlass — Wed, 08 Nov 2017 12:10:01 GMT

I didn't notice that actually. That because a little different version of HDP 2.6.2 to HDP 2.6.3..

And as I know, JDK 1.8 minimum requirement will be adopted HDP 3.0+....

question HDP 2.6.3 Install & Upgrade Issue in Archives of Support Questions (Read Only)

HDP 2.6.3 Install & Upgrade Issue

Re: HDP 2.6.3 Install & Upgrade Issue

Re: HDP 2.6.3 Install & Upgrade Issue