question Re: Spark 1.6 - Dataframe read json throws org.apache.spark.sql.DataFrame = [_corrupt_record: string] in Support Questions

Spark 1.6 - Dataframe read json throws org.apache.spark.sql.DataFrame = [_corrupt_record: string]

rmy1712 — Tue, 14 Nov 2017 01:40:29 GMT

I am using Spark 1.6.

I get the following error when I try to read a json file as per documentation of Spark 1.6

scala> val colors = sqlContext.read.json("C:/Downloads/colors.json");
colors: org.apache.spark.sql.DataFrame = [_corrupt_record: string]

1. The Spark 1.6 works fine and I have been able to read other text files.

2. Attached the json file as text(since upload of json is blocked), I have also validated the json using jsoneditoronline to ensure the json file is well formulated

colors.txt

Re: Spark 1.6 - Dataframe read json throws org.apache.spark.sql.DataFrame = [_corrupt_record: string]

dineshc — Tue, 14 Nov 2017 01:45:46 GMT

This is usually a problem with multiline json files.

use the following in your json file:

{"colors":[{"color":"black","category":"hue","type":"primary","code":{"rgba":[255,255,255,1],"hex":"#000"}},{"color":"white","category":"value","code":{"rgba":[0,0,0,1],"hex":"#FFF"}},{"color":"red","category":"hue","type":"primary","code":{"rgba":[255,0,0,1],"hex":"#FF0"}},{"color":"blue","category":"hue","type":"primary","code":{"rgba":[0,0,255,1],"hex":"#00F"}},{"color":"yellow","category":"hue","type":"primary","code":{"rgba":[255,255,0,1],"hex":"#FF0"}},{"color":"green","category":"hue","type":"secondary","code":{"rgba":[0,255,0,1],"hex":"#0F0"}}]}

Re: Spark 1.6 - Dataframe read json throws org.apache.spark.sql.DataFrame = [_corrupt_record: string]

rmy1712 — Tue, 14 Nov 2017 01:53:41 GMT

@Dinesh Chitlangia

That helped. Thank you..!!