Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

not able to import import spark.implicits._

Solved Go to solution

not able to import import spark.implicits._

I am using spark shell , my os is centos6 when i am trying to import spark.implicits._ , i get below error:

<console>:30: error: not found: value spark
         import spark.implicits._
                ^
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: not able to import import spark.implicits._

I am able to import import spark.implicits._ , earlier i was using spark1 but launching spark2 solved the problem.

View solution in original post

3 REPLIES 3
Highlighted

Re: not able to import import spark.implicits._

also I get below error too :

scala> val spark = SparkSession.builder().enableHiveSupport().getOrCreate()
<console>:30: error: not found: value SparkSession
         val spark = SparkSession.builder().enableHiveSupport().getOrCreate(
Highlighted

Re: not able to import import spark.implicits._

I am able to import import spark.implicits._ , earlier i was using spark1 but launching spark2 solved the problem.

View solution in original post

Highlighted

Re: not able to import import spark.implicits._

Cloudera Employee

Hi,

 

There is no package called spark.implicits. 

Spark 1.x:

If you are using spark1.x version you will create sqlContext. By using sqlContext  you can call sqlContext.implicits

Example:

val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
sqlContext.implicits

Spark 2.x:

If you are using spark2.x version you will create session object. By using session you can call spark.implicits.

Example:

val spark: SparkSession = SparkSession.builder.appName(appName).config("spark.master", "local[*]").getOrCreate
spark.implicits

Note: If you are created session object using different name then you need to call with that reference name. 

For example,

val rangaSpark: SparkSession = SparkSession.builder.appName(appName).config("spark.master", "local[*]").getOrCreate
rangaSpark.implicits 
Don't have an account?
Coming from Hortonworks? Activate your account here