Hi,

I am upgrading from hdp2.6.3 to hdp3.1.4. Post upgrade am getting the following error when running one of my jobs

Caused by: java.lang.ClassNotFoundException: org.apache.hbase.thirdparty.com.google.common.cache.CacheLoader
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357

I see that the jar containing this class is hbase-shaded-miscellaneous-2.2.0.jar and is present in 

/usr/hdp/3.1.4.0-315/hbase/lib/hbase-shaded-miscellaneous-2.2.0.jar

/usr/hdp/3.1.4.0-315/spark2/jars/hbase-shaded-miscellaneous-2.2.0.jar

I can see that the spark2-hdp-yarn-archive.tar.gz created in /hadoop/yarn/local/filecache/0/11837/spark2-hdp-yarn-archive.tar.gz does not contain this jar file.

When I add the jar explicity to my spark-submit command by appending it to --jars option and adding it to conf as below , I dont see this error ,

--conf "spark.driver.extraClassPath=/usr/hdp/3.1.4.0-315/spark2/jars/hbase-protocol-shaded-2.0.2.3.1.4.0-315.jar

Progressing in above manner I then get another ClassNotfound for another shaded class in hbase-protocol-shaded-2.0.2.3.1.4.0-315.jar. 

Can I get some pointers on what the issue may be.

Other jars like hbase-client-2.0.2.3.1.4.0-315.jar etc are not giving this issue.

Thanks,

Kevin Mathrani