I am trying to convert a text file to DataFrame. I found using following method instead of case class. But where is the data type for each field is defined if we go by this method. val people = sc.textFile("file:/home/edureka/dmishra/people.txt")
val schemaString = "name age"
import org.apache.spark.sql.Row;
import org.apache.spark.sql.types.{StructType,StructField,StringType};
val schema =
StructType(
schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
val rowRDD = people.map(_.split(",")).map(p => Row(p(0), p(1).trim))
val peopleDataFrame = sqlContext.createDataFrame(rowRDD, schema)
peopleDataFrame.registerTempTable("people")
val results = sqlContext.sql("select name,age from people")
val r = results.map(t => "Name: " + t(0) + "Age : " + t(1)).collect().foreach(println scala> results.dtypes.foreach(println) (name,StringType) (age,StringType) Where is the data type assigned for data frame. How to define age as integer data type in this case or if there is a date field, where to define it. Thanks
... View more
