11-05-2014 03:11 PM
I have checked out some message boards and found we should add jars to the class path if we get errors of this nature and I have attempted to do that but I'm not having any luck.
below are my attempts to add the jar to the classpath which I think were succesfull
[hdfs@cloudera01 root]$ ls
[hdfs@cloudera01 root]$ pwd
[hdfs@cloudera01 root]$ export SPARK_SUBMIT_CLASSPATH=./commons-codec-1.4.jar:$SPARK_HOME/assembly/lib/*:./root/zookeeper-3.4.5-cdh5.1.3.jar
[hdfs@cloudera01 root]$ echo $SPARK_SUBMIT_CLASSPATH
Does anyone have any ideas or notice if my commands are off?
Links I used to try these operations are below, thanks for any thoughts/ideas
I believe that in the Cloudera article it is saying to include the app you are running with the missing dependency included within that jar in the classpath rather than just the missing jar but I have been having a difficult time gettting Maven to doexactly what I would like it to do. I can get it to include the whole dependency list of my app but then Hue/something-with-my-setup won't let me upload the jar because I belive there is an issue with uploading files greater than 64MB just going off Google search results. I don't think I would need to include my app with the dependency for it to work though if that was the root of the problem.
11-05-2014 11:20 PM
Yes, in general you need to include JARs with your app that your app uses. This is true in Java in general. Are you certain you are using Zookeeper in your app? SPARK_CLASSPATH is an old mechanism. See Spark 1.0.0 docs, or better, update to CDH 5.2 / Spark 1.1 (although that's not the problem here). I don't think this is the best place for Maven help and am not sure what you are referring to regarding dependencies and a 64MB limit.
11-06-2014 02:02 PM
Including the all the jars worked like a charm. Regarding the "64MB limit", I wasn't able to upload the uber-jar to HDFS via Hue (Error:Undefined) and from some searches I saw people claiming that they thought it was a size issue.