Cloudera Community
JPMChaseBigData
user rank
Contributor

Re: spark sql is not working on CDH5.3

by Contributor in Support Questions
‎03-25-2015 10:00 AM
‎03-25-2015 10:00 AM
I agree. What is the best solution for this? ... View more

Re: spark sql is not working on CDH5.3

by Contributor in Support Questions
‎03-24-2015 11:32 AM
1 Kudo
‎03-24-2015 11:32 AM
1 Kudo
Hi,   There is bug in the classpath.   You need add a line int the compute-classpath.sh CLASSPATH="$CLASSPATH:/opt/cloudera/parcels/CDH/lib/hive/lib/*" .  Then it will work without any issues.   Regards, Venkat     ... View more

Re: spark sql is not working on CDH5.3

by Contributor in Support Questions
‎02-23-2015 12:03 PM
‎02-23-2015 12:03 PM
my bad. I fixed the typo. But still no luck. Thanks. ... View more

Re: spark sql is not working on CDH5.3

by Contributor in Support Questions
‎02-23-2015 11:26 AM
‎02-23-2015 11:26 AM
Thanks for quick reply. I have added the path too.  But still not working. correct me if I'm wrong or missing something.   #!/bin/bash export JAVA_HOME=/usr/java/jdk1.7.0_55 SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark/ SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/spark/lib/*.jar   JARS="" for j in `ls /opt/cloudera/parcels/CDH/lib/hadoop/client/*.jar` do JARS=$JARS:$j JARS1=$j,$JARS1 done CLI=/opt/cloudera/parcels/CDH/lib/hive/lib/hive-cli-0.13.1-cdh5.3.1.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/hive-common-0.13.1-cdh5.3.1.jar:=/opt/cloudera/parcels/CDH/lib/hive/lib/hive-jdbc-0.13.1-cdh5.3.1.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec-0.13.1-cdh5.3.1.jar $SPARK_CLASSPATH:$JARS:$CLI $SPARK_HOME/bin/spark-sql --master local ... View more

spark sql is not working on CDH5.3

by Contributor in Support Questions
‎02-23-2015 10:48 AM
‎02-23-2015 10:48 AM
We upgraded to CDH5.3. spark-sql is not working. Could any bodyk please provide us am I missing any steps we need to follow to spark-sql to work. I'm getting the below error.   Steps we did: 1.copy the hive-site.xml to /etc/spark/conf 2.Try to start the thrifiserver but getting the error.   /opt/cloudera/parcels/CDH/lib/spark/sbin/start-thriftserver.sh starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /var/log/spark/spark-root-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-rkqcrl-odshaun01.out failed to launch org.apache.spark.sql.hive.thriftserver.HiveThriftServer2:         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)         ... 18 more   bash-4.1$ ./spark-sql Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/cli/CliDriver         at java.lang.ClassLoader.defineClass1(Native Method)         at java.lang.ClassLoader.defineClass(ClassLoader.java:800)         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)         at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)         at java.net.URLClassLoader.access$100(URLClassLoader.java:71)         at java.net.URLClassLoader$1.run(URLClassLoader.java:361)         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)         at java.security.AccessController.doPrivileged(Native Method)         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)         at java.lang.ClassLoader.loadClass(ClassLoader.java:412)         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)         at java.lang.Class.forName0(Native Method)         at java.lang.Class.forName(Class.java:270)         at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.cli.CliDriver         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)         at java.security.AccessController.doPrivileged(Native Method)         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)         ... 18 more   Regards, Venky ... View more
Contact Me
Online
Offline
Last Visited
‎03-24-2018 01:10 PM
Community Statistics
Member Since ‎02-11-2015 09:45 AM
Last Visited ‎03-24-2018 01:10 PM
Posts 10
Kudos received 1