Thanks Wilfred.   This issue is from http://community.cloudera.com/t5/Batch-Processing-and-Workflow/how-to-add-external-guava-16-0-1-jar-in-CDH-oozie-classpath/m-p/37803#U37803   Does that mean there is no solution to let CDH5.5.0  Hue/Oozie support spark action(Spark 1.5.0) which will write data into Cassandra 2.1.11? ... View more

We generated shadow (gradle) to shadow guava-16.0.1 for sparktest.jar, and put the shadow (gradle) sparktest.jar into oozie's share lib classpath, then run the shawsaprktest.jar in CDH oozie, it throws following exception: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.jets3t.service.ServiceException: Request Error: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.ServiceException: Request Error: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:478) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleException(Jets3tNativeFileSystemStore.java:411) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at org.apache.hadoop.fs.s3native.$Proxy48.retrieveMetadata(Unknown Source) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:468) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272) at org.apache.hadoop.fs.Globber.glob(Globber.java:151) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921) at org.apache.spark.rdd.RDD.count(RDD.scala:1121) at com.gridx.spark.MeterReadingLoader$.load(MeterReadingLoader.scala:120) at com.gridx.spark.MeterReadingLoader$.main(MeterReadingLoader.scala:101) at com.gridx.spark.MeterReadingLoader.main(MeterReadingLoader.scala)     According to http://quabr.com/20613854/exception-when-getting-a-list-of-buckets-on-s3-using-jets3t, after changing httpclient from 4.2.5 jar  to 4.2 jar in hdfs oozie shared lib, it throws following exception: JA008: File does not exist: hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark/httpclient-4.2.5.jar ... View more

Thanks very much Wilfred for your helpful suggestion! We tried shadow before in gradle, at that time it still had the guava Nosuchmethod issue. Let's try shadow again with adding guava-16.0.1.jar in oozie sharelib to see whether it can work. ... View more

Agree, andreF. Thanks for your resposen.   I package guava 16.0.1 into my jar, it still get the same problem, I guess it has guava version confliction, since there are serveral different guava in CDH's classpath.txt.   I really confused about this.  ... View more

Hi Wilfred,     I have the similar issue as andreF's, we have serval differnt guava in /etc/spark/conf/classpath.txt, do you know how to fix the issue?   /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/guava-11.0.2.jar /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/guava-11.0.jar /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/guava-14.0.1.jar /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/guava-16.0.1.jar   Our app needs to use guava-16.0.1.jar, so I add guava-16.0.1.jar into /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/, and add "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars/guava-16.0.1.jar" into /etc/spark/conf/classpath.txt.   However, it doesn't work, spark action in oozie still can not find guava-16.0.1.jar. How does classpath.txt work? Do you know how to manage or modify the classpath.txt manually? Thanks!       ... View more

Hi andreF,   I have the similar issue, did you fix the issue? ... View more

 Hi Harsh,   Actually I'm still a little confued about the 4 ways mentioned in "One Last Thing" in http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/ I tried all the ways, but all didn't work.(I'm using CDH Hue and oozie workflow) Following were what I tried with the 4 ways:   For way 1: It recommended "oozie.libpath=/path/to/jars,another/path/to/jars" I add oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark or oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark/guava-16.0.1.jar and oozie.use.system.libpath=true is by default. Both don't work.   For way 2: I added guava-16.0.1.jar into “lib” next to current workspace workflow.xml in HDFS, it doesn't work.   For way 3: I can not find any <archive> tag in a Spark action with the path to a single jar, so I have no way to try way3.   For way 4: I added guava-16.0.1.jar to the ShareLib (e.g. hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark) and set oozie.use.system.libpath=true in job.properties, it still doesn't work.   Could you please give any suggestion? Thanks very much for any of your help! I appreciated! ... View more

Dear sonups3, I met the same issue, can you explain detailed what you did to fix? Thanks! ... View more

Dear, I met the same issue, can you explain detailed what you did to fix? Thanks! ... View more

Hi jdb, how did you add external jar into oozie, and what configuration you changed to make oozie classpath pick up the new jar? Thanks! ... View more