Reply
Explorer
Posts: 6
Registered: ‎06-16-2017

HUE can't find jar on HDFS

As I can see on github it's a well known error. So, everytime I try to submit jar (Spark job) and same jar is on HDFS I'm getting this error:

Warning: Skip remote jar hdfs://bigdata1.int.ch:8020/user/hue/spark-streaming.jar.
java.lang.ClassNotFoundException: com.test.parquetToHadoop
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:702)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:169)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:167)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.Exception: spark-submit exited with code 101}.
Warning: Skip remote jar hdfs://bigdata1.int.ch:8020/user/hue/spark-streaming.jar.
java.lang.ClassNotFoundException: com.test.parquetToHadoop
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:702)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:169)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:167)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(SparkYarnApp.scala:238) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(SparkYarnApp.scala:235) scala.Option.foreach(Option.scala:236) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:235) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)

Is there some workaround? Is there a way to connect Hue with UNIX local file system and how, because  I haven't found it? Also I read some of people are doing that, but how?

Any suggestions?

Ty

Announcements