@Amal Babu this is my take, I'm sure there's better ways %spark
import sqlContext.implicits._
val data = sc.wholeTextFiles("hdfs://sandbox.hortonworks.com:8020/user/guest/")
val dataDF = data.toDF()
dataDF.select("_1").show()
result import sqlContext.implicits._
data: org.apache.spark.rdd.RDD[(String, String)] = hdfs://sandbox.hortonworks.com:8020/user/guest/ MapPartitionsRDD[64] at wholeTextFiles at <console>:68
dataDF: org.apache.spark.sql.DataFrame = [_1: string, _2: string]
+--------------------+
| _1|
+--------------------+
|hdfs://sandbox.ho...|
|hdfs://sandbox.ho...|
|hdfs://sandbox.ho...|
+--------------------+
as long as you use wholeTextFiles you should be able to maintain filenames. From the documentation SparkContext.wholeTextFiles lets you read a directory containing multiple small text files, and returns each of them as (filename, content) pairs. This is in contrast with textFile , which would return one record per line in each file.
... View more
