question How to use sqlContext? Getting an error! in Support Questions

How to use sqlContext? Getting an error!

sudhir_reddy — Wed, 04 Jul 2018 17:59:32 GMT

sqlcontext.png Please find snapshot attached for the issue.

Re: How to use sqlContext? Getting an error!

jsensharma — Wed, 04 Jul 2018 18:05:27 GMT

Are you using Spark2?

Have you set the Env variable as following before opening the shell?

# export SPARK_MAJOR_VERSION=2
# spark-shell

Re: How to use sqlContext? Getting an error!

sudhir_reddy — Wed, 04 Jul 2018 18:12:48 GMT

Yes, using spark2. It is by default using Spark2. What else should I try? @Jay Kumar SenSharma

Re: How to use sqlContext? Getting an error!

jsensharma — Wed, 04 Jul 2018 18:14:56 GMT

@sudhir reddy

Can you please share the complete output along with the command that you ran?

Re: How to use sqlContext? Getting an error!

sudhir_reddy — Wed, 04 Jul 2018 18:18:26 GMT

val rd = sqlContext.read.json("/user/root/Example2.JSON"); //Used this command.

<console>:23: error: not found: value sqlContext val rd = sqlContext.read.json("/user/root/Example2.JSON"); ^ //Got this output @Jay Kumar SenSharm

Re: How to use sqlContext? Getting an error!

jsensharma — Wed, 04 Jul 2018 18:27:56 GMT

@sudhir reddy

How are you logging in to the Shell? Which command are you using to login in to the spark shell ? Which mode?

For example when you run "spark-shell" then you will see lots of output ... we need that complete output.

Example

# spark-shell <br>

As much information you can provide about your issue ... there are possibilities that your issue will get resolved faster. Else there will be more interactions to know which HDP version are you using ? What is the logging happening? ...etc

Re: How to use sqlContext? Getting an error!

sandyy006 — Wed, 04 Jul 2018 19:16:30 GMT

@sudhir reddy

SqlContext is not available by default in Spark2 shell.

Create a sqlContext using below statement after launching the spark-shell and then you can read the json using this sqlContext.

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

Let me know if this helps.

Re: How to use sqlContext? Getting an error!

A_Abessi — Wed, 04 Jul 2018 19:25:12 GMT

val sc: SparkContext // An existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val rd = sqlContext.read.json("user/root/Example2.JSON")

Re: How to use sqlContext? Getting an error!

sudhir_reddy — Wed, 04 Jul 2018 20:08:00 GMT

Thanks! It worked.

Re: How to use sqlContext? Getting an error!

sudhir_reddy — Wed, 04 Jul 2018 20:09:27 GMT

Thank you!

Re: How to use sqlContext? Getting an error!

A_Abessi — Thu, 05 Jul 2018 14:36:17 GMT

Hope this helps.