Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark 1.6 - Dataframe read json throws org.apache.spark.sql.DataFrame = [_corrupt_record: string]

avatar
Rising Star

I am using Spark 1.6.

I get the following error when I try to read a json file as per documentation of Spark 1.6

scala> val colors = sqlContext.read.json("C:/Downloads/colors.json");
colors: org.apache.spark.sql.DataFrame = [_corrupt_record: string]

1. The Spark 1.6 works fine and I have been able to read other text files.

2. Attached the json file as text(since upload of json is blocked), I have also validated the json using jsoneditoronline to ensure the json file is well formulated

colors.txt

1 ACCEPTED SOLUTION

avatar

This is usually a problem with multiline json files.

use the following in your json file:

{"colors":[{"color":"black","category":"hue","type":"primary","code":{"rgba":[255,255,255,1],"hex":"#000"}},{"color":"white","category":"value","code":{"rgba":[0,0,0,1],"hex":"#FFF"}},{"color":"red","category":"hue","type":"primary","code":{"rgba":[255,0,0,1],"hex":"#FF0"}},{"color":"blue","category":"hue","type":"primary","code":{"rgba":[0,0,255,1],"hex":"#00F"}},{"color":"yellow","category":"hue","type":"primary","code":{"rgba":[255,255,0,1],"hex":"#FF0"}},{"color":"green","category":"hue","type":"secondary","code":{"rgba":[0,255,0,1],"hex":"#0F0"}}]}

View solution in original post

2 REPLIES 2

avatar

This is usually a problem with multiline json files.

use the following in your json file:

{"colors":[{"color":"black","category":"hue","type":"primary","code":{"rgba":[255,255,255,1],"hex":"#000"}},{"color":"white","category":"value","code":{"rgba":[0,0,0,1],"hex":"#FFF"}},{"color":"red","category":"hue","type":"primary","code":{"rgba":[255,0,0,1],"hex":"#FF0"}},{"color":"blue","category":"hue","type":"primary","code":{"rgba":[0,0,255,1],"hex":"#00F"}},{"color":"yellow","category":"hue","type":"primary","code":{"rgba":[255,255,0,1],"hex":"#FF0"}},{"color":"green","category":"hue","type":"secondary","code":{"rgba":[0,255,0,1],"hex":"#0F0"}}]}

avatar
Rising Star

@Dinesh Chitlangia

That helped. Thank you..!!