Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

i am not able to run hivecontext on spark

i am not able to run hivecontext on spark

New Contributor

i have installed spark and hive on the cluster and i am trying to process the file from hdfs

i have added these jars

spark-assembly-1.2.0-cdh5.3.0-hadoop2.5.0-cdh5.3.0.jar

when i try to proces local file system i have added these jars

spark-assembly-1.2.0-hadoop1.0.4

spark-core_2.10-0.9.0-incubating

and it works fine

 

i am getting this error

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hiv                                                                                        e/conf/HiveConf
        at org.apache.spark.sql.hive.api.java.JavaHiveContext.<init>(JavaHiveCon                                                                                        text.scala:30)
        at max.main(max.java:29)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.                                                                                        java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces                                                                                        sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveCon                                                                                        f
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 9 more

i have added hive config in classpath

2 REPLIES 2

Re: i am not able to run hivecontext on spark

Master Collaborator

Please read other related posts in this forum first for steps you should take to resolve this.

Spark SQL undefined function collect_list under cdh 5.7.2

New Contributor

While doing oozie workflow and submitting spark jar getting below exception. But this collect_list method/function is  available in following packageorg.apache.spark.sql.functions.collect_list under the jars/spark-assembly-1.6.0-cdh5.7.2-hadoop2.6.0-cdh5.7.2.jar

 

Seems to spark.sql.functions in not picking above mentioned jar

 

 

Exception in thread "main" org.apache.spark.sql.AnalysisException: undefined function collect_list;

        at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry$$anonfun$2.apply(FunctionRegistry.scala:65)

        at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry$$anonfun$2.apply(FunctionRegistry.scala:65)

 

 

Regards,

Rama