Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to specify only particular fields using read.schema in JSON : SPARK Scala

How to specify only particular fields using read.schema in JSON : SPARK Scala

Hello,

i want to read json file with scala. That is head off my file.

Array[Array[String]] = Array(Array(PlayLog(PlayLogHeader(2, LeChat-PPR, None, ad6bf296-98ea-426f-bf62-a53c0d3b4fee, 2017-03-20T00:00:00.000+01:00), PlayLogBody(147323158, Some(fr-cpl-lx0001.jcd.priv), dms-player, 2017-03-20T00:00:00.000+01:00, None, None, Some(10), 104168052, Some(94014659), None, List(), List(), List()))), Array(PlayLog(PlayLogHeader(2, LeChat-PPR, None, daba4e38-390e-4a17-b2d2-5143a22bfa26, 2017-03-20T00:00:00.000+01:00), PlayLogBody(35776754, Some(fr-lab-lx0014.jcd.priv), dms-player, 2017-03-20T00:00:00.000+01:00, None, None, Some(10), 102675128, Some(50732058), None, List(), List(), List()))))

i start by this code but it's don't match

val head = StructType(
                Array(
                    StructField("version",DoubleType,true), 
                    StructField("makerId",StringType,true), 
                    StructField("envId",StringType,true), 
                    StructField("id",StringType,true), 
                    StructField("creationTime",StringType,true)
                )
            )
val host = StructType(
                Array(
                    StructField("string",StringType,true)
                )
            )


val duration = StructType(
                Array(
                    StructField("int",LongType,true) 
                )
            )


val frame = StructType(
                Array(
                    StructField("frameid",LongType,true) 
                )
            )


			
val bod = StructType(
                Array(
                    StructField("playerid",StringType,true), 
					StructField("hostname",
						ArrayType(host,true),
								true),
                    StructField("playertype",StringType,true), 
                    StructField("timestamp",StringType,true), 
                    StructField("campaignid",StringType,true), 
                    StructField("duration",
						ArrayType(host,true),
						true), 
					StructField("creativeid",StringType,true),
					StructField("frameid",
						ArrayType(frame,true),
						true),
					StructField("status",StringType,true),
					StructField("media",StringType,true),
					StructField("condition",StringType,true),
					StructField("context",StringType,true)
                )
            )
			


val schema = StructType(Array(


    StructField("PlayLogHeader",
        ArrayType(head,true),
                true),
	StructField("PlayLogBody",
        ArrayType(bod,true),
                true)
				
    )
)



Don't have an account?
Coming from Hortonworks? Activate your account here