Member since
08-18-2016
8
Posts
6
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5029 | 12-05-2016 06:18 AM | |
1500 | 11-03-2016 08:30 PM |
12-05-2016
06:22 AM
1 Kudo
While running the Spark 2.0 example in this tutorial in the Sandbox, I am getting this error. I have followed the steps given and I am not able to find the issue. Has any faced this issue? ---------------------------------------------------------------------------------------------------------------------------------------- [root@sandbox ~]# export SPARK_HOME=/usr/hdp/current/spark-client [root@sandbox ~]# export SPARK_MAJOR_VERSION=2
[root@sandbox ~]# cd /usr/hdp/current/spark2-client [root@sandbox spark2-client]# su spark [spark@sandbox spark2-client]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples*.jar 10
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession$
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession$
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
12-05-2016
06:18 AM
I was able to find the issue. I had not gone to the correct folder location "" before running the spark-submit. Before running "spark-submit" we should navigate to the right folder "$cd /usr/hdp/current/spark-client".
... View more
12-05-2016
06:02 AM
When I submit a Spark program like this, after su Spark, I get a "bash: ./bin/spark-submit: Permission denied" error. What is the password for "spark" user? How can I cross thi step? Please help. [spark@sandbox root]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples*.jar 10
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
11-03-2016
10:16 PM
Throughout the tutorial they instruct the user to copy paste the commands, but the pyspark commands work only when it is typed manually. I hope the admins fix this.
... View more
11-03-2016
08:30 PM
3 Kudos
@Michael Young You are right. The root cause was that the default Docker maximum container size was 10 GB. The HDP Sandbox size is about 13.5 GB. So there was not enough space to create the image. I made the changes below and it worked. 1. Modify the docker config in /etc/sysconfig/docker-storage to add the line: DOCKER_STORAGE_OPTIONS= "--storage-opt dm.basesize=20G" 2. Restart Docker. 3. Load the Hortonworks Sandbox Docker image again.
... View more
11-03-2016
07:14 PM
2 Kudos
I tried to install the HDP Sandbox on Docker in AWS (Amazon Linux). When loading the Docker image, I am seeing this error. I edited the "/etc/sysconfig/docker" to provide the "-g" option to increase the storage space (free space is about 500 GB), but I am still getting this error. Please help. # docker load <HDP_2.5_docker.tar
b1b065555b8a: Loading layer [==================================================>] 202.2 MB/202.2 MB
3901568415a3: Loading layer [=====================================> ] 10.37 GB/13.85 GB
ApplyLayer exit status 1 stdout: stderr: write /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/jre/lib/amd64/server/libjvm.so: no space left on device
... View more
Labels:
- Labels:
-
Docker
09-23-2016
07:05 AM
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)