Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

not able to import import spark.implicits._

avatar

I am using spark shell , my os is centos6 when i am trying to import spark.implicits._ , i get below error:

<console>:30: error: not found: value spark
         import spark.implicits._
                ^
1 ACCEPTED SOLUTION

avatar

I am able to import import spark.implicits._ , earlier i was using spark1 but launching spark2 solved the problem.

View solution in original post

3 REPLIES 3

avatar

also I get below error too :

scala> val spark = SparkSession.builder().enableHiveSupport().getOrCreate()
<console>:30: error: not found: value SparkSession
         val spark = SparkSession.builder().enableHiveSupport().getOrCreate(

avatar

I am able to import import spark.implicits._ , earlier i was using spark1 but launching spark2 solved the problem.

avatar
Master Collaborator

Hi,

 

There is no package called spark.implicits. 

Spark 1.x:

If you are using spark1.x version you will create sqlContext. By using sqlContext  you can call sqlContext.implicits

Example:

val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
sqlContext.implicits

Spark 2.x:

If you are using spark2.x version you will create session object. By using session you can call spark.implicits.

Example:

val spark: SparkSession = SparkSession.builder.appName(appName).config("spark.master", "local[*]").getOrCreate
spark.implicits

Note: If you are created session object using different name then you need to call with that reference name. 

For example,

val rangaSpark: SparkSession = SparkSession.builder.appName(appName).config("spark.master", "local[*]").getOrCreate
rangaSpark.implicits