Hello Team, I am working the tutorial on RDD. I am having some difficulties understanding some commands. Can you please advise what steps 3-8 do? . Encode the Schema in a string val schemaString = "name age" 4. Generate the schema based on the string of schema val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, nullable = true))
val schema = StructType(fields) 5. Convert records of the RDD (people) to Rows val rowRDD = peopleRDD.map(_.split(",")).map(attributes => Row(attributes(0), attributes(1).trim)) 6. Apply the schema to the RDD val peopleDF = spark.createDataFrame(rowRDD, schema) 6. Creates a temporary view using the DataFrame peopleDF.createOrReplaceTempView("people") 7. SQL can be run over a temporary view created using DataFrames val results = spark.sql("SELECT name FROM people") 8.The results of SQL queries are DataFrames and support all the normal RDD operations. The columns of a row in the result can be accessed by field index or by field name results.map(attributes => "Name: " + attributes(0)).show() https://www.cloudera.com/tutorials/dataframe-and-dataset-examples-in-spark-repl.html Programmatically Specifying Schema What does the code below do? val ds = Seq(1, 2, 3).toDS() val ds = Seq(Person("Andy", 32)).toDS() Section Section DataSet API is clear. If we need to map the JSON file to a class we use the as(class name). So to map a file to a class we use the ".as[Classname]"? what does this command do? val ds = Seq(1, 2, 3).toDS() Thanks, Roshan
... View more
