Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

Spark 1.6 - Dataframe read json throws org.apache.spark.sql.DataFrame = [_corrupt_record: string]

avatar
Rising Star

I am using Spark 1.6.

I get the following error when I try to read a json file as per documentation of Spark 1.6

scala> val colors = sqlContext.read.json("C:/Downloads/colors.json");
colors: org.apache.spark.sql.DataFrame = [_corrupt_record: string]

1. The Spark 1.6 works fine and I have been able to read other text files.

2. Attached the json file as text(since upload of json is blocked), I have also validated the json using jsoneditoronline to ensure the json file is well formulated

colors.txt

1 ACCEPTED SOLUTION

avatar

This is usually a problem with multiline json files.

use the following in your json file:

{"colors":[{"color":"black","category":"hue","type":"primary","code":{"rgba":[255,255,255,1],"hex":"#000"}},{"color":"white","category":"value","code":{"rgba":[0,0,0,1],"hex":"#FFF"}},{"color":"red","category":"hue","type":"primary","code":{"rgba":[255,0,0,1],"hex":"#FF0"}},{"color":"blue","category":"hue","type":"primary","code":{"rgba":[0,0,255,1],"hex":"#00F"}},{"color":"yellow","category":"hue","type":"primary","code":{"rgba":[255,255,0,1],"hex":"#FF0"}},{"color":"green","category":"hue","type":"secondary","code":{"rgba":[0,255,0,1],"hex":"#0F0"}}]}

View solution in original post

2 REPLIES 2

avatar

This is usually a problem with multiline json files.

use the following in your json file:

{"colors":[{"color":"black","category":"hue","type":"primary","code":{"rgba":[255,255,255,1],"hex":"#000"}},{"color":"white","category":"value","code":{"rgba":[0,0,0,1],"hex":"#FFF"}},{"color":"red","category":"hue","type":"primary","code":{"rgba":[255,0,0,1],"hex":"#FF0"}},{"color":"blue","category":"hue","type":"primary","code":{"rgba":[0,0,255,1],"hex":"#00F"}},{"color":"yellow","category":"hue","type":"primary","code":{"rgba":[255,255,0,1],"hex":"#FF0"}},{"color":"green","category":"hue","type":"secondary","code":{"rgba":[0,255,0,1],"hex":"#0F0"}}]}

avatar
Rising Star

@Dinesh Chitlangia

That helped. Thank you..!!