question Re: Converting JSON to Rdd in Archives of Support Questions (Read Only)

Converting JSON to Rdd

amehta7 — Fri, 24 Jun 2016 05:17:03 GMT

I am getting a json response, and in my sparkSQL data source, i need to read the data and infer schema for the json and convert in to rdd<ROW>. Is there any class to do that in spark?

Thanks

Re: Converting JSON to Rdd

rgelhausen — Fri, 24 Jun 2016 05:38:36 GMT

http://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets

Re: Converting JSON to Rdd

amehta7 — Fri, 24 Jun 2016 05:47:21 GMT

I dont want to read from files. I have json data in a variable coming from http response in my code.

Re: Converting JSON to Rdd

mqureshi — Fri, 24 Jun 2016 05:58:43 GMT

@Akash Mehta Can you do something like this?

dataframe = sqlContext.read.format(“json”).load(your json here)

Re: Converting JSON to Rdd

amehta7 — Fri, 24 Jun 2016 06:04:19 GMT

But "your json here" takes a path and i am having the json from an httpresponse (converted to string).

I need to read from that and infer the schema and convert to rdd<ROW>

Re: Converting JSON to Rdd

clukasik — Fri, 24 Jun 2016 06:32:18 GMT

val dataframe = sqlContext.read.json(<a RDD[String] where each line is JSON object>)

See https://spark.apache.org/docs/1.6.0/api/java/org/apache/spark/sql/DataFrameReader.html#json(org.apache.spark.rdd.RDD)

Re: Converting JSON to Rdd

mqureshi — Fri, 24 Jun 2016 06:51:29 GMT

load will infer schema and convert to a row. Question is whether it will take an http url. Can you try?

Re: Converting JSON to Rdd

amehta7 — Fri, 24 Jun 2016 08:14:36 GMT

This will output a dataframe and i need RDD[Row]

Re: Converting JSON to Rdd

clukasik — Fri, 24 Jun 2016 19:16:18 GMT

Will this work?

https://spark.apache.org/docs/1.6.0/api/java/org/apache/spark/sql/DataFrame.html#rdd()

Re: Converting JSON to Rdd

amehta7 — Sat, 25 Jun 2016 02:00:13 GMT

Yes yes load will do that but load requires an input path and i have my json stored in a string variable.

Re: Converting JSON to Rdd

mqureshi — Sun, 26 Jun 2016 08:54:41 GMT

@Akash Mehta

So, even following wont work for you? If not, I think currently there is no other way given we have looked at all other possible options.

//a DataFrame can be created for a JSON dataset represented by
// an RDD[String] storing one JSON object per string.

val anotherPeopleRDD = sc.parallelize(
  """{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}""" :: Nil)
val anotherPeople = sqlContext.read.json(anotherPeopleRDD)