Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created on 08-04-2016 01:53 PM - edited 09-16-2022 03:32 AM
Hi All,
My CDH 5.5 was running all fine but now when i typed spark-shell command ,i see below issue of sqlcontext,can anyone suggest what should i do to remove this issue,spark-shell was running perfectly till this issue.
Please note i restarted Cloudera manager from admin console
16/08/04 13:38:51 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
at org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
16/08/04 13:38:51 INFO SparkContext: Successfully stopped SparkContext
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /user/cloudera/.sparkStaging/application_1470339377450_0002. Name node is in safe mode.
The reported blocks 919 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 921.
The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1416)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNames
<console>:10: error: not found: value sqlContext
import sqlContext.implicits._
^
<console>:10: error: not found: value sqlContext
import sqlContext.sql
Created 08-04-2016 07:42 PM
Created 08-04-2016 07:42 PM
got solved by below
sudo -u hdfs hdfs dfsadmin -safemode leave
Created 08-16-2016 07:58 AM
i am using cloudera vm 5.7 when i type spark-shell i am getting following error.
<console>:16: error: not found: value sqlContext
import sqlContext.implicits._
^
<console>:16: error: not found: value sqlContext
import sqlContext.sql
Created 08-16-2016 08:09 AM
5.5 or 5.7? the title and text disagree. 5.5 would have Spark 1.4, and I am not sure whether SQLContext was exposed as sqlContext by the shell by default like that. It should be in Spark 1.6 (= CDH 5.6+)