Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP 2.6.3 Install & Upgrade Issue

avatar
Rising Star

I'm upgrading hdp 2.6.2.14 to 2.6.3.0, but I'm stuck in this issue and cannot finalize upgrade procedure.

JAVA ENV : jdk 1.7

To my way of thinking, 'java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0' this issue is different version between jdk7 and jdk8. Maybe YarnShuffleService.class file is compiled by jdk7 compiler by apache's some developer.

I think, this issue should be fixed ASAP. I know jdk7 is deprecated in this HDP 2.6.3.0 version and it will be removed future version.

Nodemanager Log.

2017-11-08 09:17:51,530 INFOyarn.YarnShuffleService (YarnShuffleService.java:serviceInit(156)) - Started YARN shuffle service for Spark on port 7337. Authentication is not enabled.Registered executor file is /data/log/hadoop/hadoop-yarn/nodemanager/recovery-state/nm-aux-services/spark_shuffle/registeredExecutors.ldb

2017-11-08 09:17:51,532 INFOutil.ApplicationClassLoader (ApplicationClassLoader.java:<init>(105)) - classpath: [file:/usr/hdp/2.6.3.0-235/spark2/aux/spark-2.2.0.2.6.3.0-235-yarn-shuffle.jar]

2017-11-08 09:17:51,532 INFOutil.ApplicationClassLoader (ApplicationClassLoader.java:<init>(106)) - system classes: [java., javax.accessibility., javax.activation., javax.activity., javax.annotation., javax.annotation.processing., javax.crypto., javax.imageio., javax.jws., javax.lang.model., -javax.management.j2ee., javax.management., javax.naming., javax.net., javax.print., javax.rmi., javax.script., -javax.security.auth.message., javax.security.auth., javax.security.cert., javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools., javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., org.xml.sax., org.apache.commons.logging., org.apache.log4j., -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml, yarn-default.xml]

2017-11-08 09:17:51,534 FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager

java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:184)

at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:274)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:546)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)

2017-11-08 09:17:51,539 INFOimpl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping NodeManager metrics system...

2017-11-08 09:17:51,539 INFOimpl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.

2017-11-08 09:17:51,540 INFOimpl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NodeManager metrics system stopped.

2017-11-08 09:17:51,541 INFOimpl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown complete.

2017-11-08 09:17:51,545 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Peter Kim

ERROR:

java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0

.

From HDP 2.6.3 onwards JDK 1.8 Must to be used as codes are compiled agains this JDK for few components as JDK 1.7 is already EOL (End Of Life Few year back).

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_support-matrices/content/ch_matrices-hdp...

42496-jdk-support.png

.

So you should upgrade the JDK to 1.8 and then upgrade HDP. So please Install JDK 1.8 and set the JAVA_HOME accordingly. You can also setup JDK 1.8 via ambari using:

# ambari-server setup –j /usr/lib/jvm18

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/setup_option...

.

View solution in original post

2 REPLIES 2

avatar
Master Mentor

@Peter Kim

ERROR:

java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0

.

From HDP 2.6.3 onwards JDK 1.8 Must to be used as codes are compiled agains this JDK for few components as JDK 1.7 is already EOL (End Of Life Few year back).

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_support-matrices/content/ch_matrices-hdp...

42496-jdk-support.png

.

So you should upgrade the JDK to 1.8 and then upgrade HDP. So please Install JDK 1.8 and set the JAVA_HOME accordingly. You can also setup JDK 1.8 via ambari using:

# ambari-server setup –j /usr/lib/jvm18

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/setup_option...

.

avatar
Rising Star

I didn't notice that actually. That because a little different version of HDP 2.6.2 to HDP 2.6.3..

And as I know, JDK 1.8 minimum requirement will be adopted HDP 3.0+....