Group is used to collect data having the same key. It is not mandatory to have an aggregation to be performed along with group. For a better understanding, let us consider a file with ID,Name and Age as below 1,John,23 2,James,24 3,Alice,30 4,Bob,23 5,Bill,24 If we have the below script applied on the file, loading the file and grouping it by age, we get all the data associated to one age into one single group. details = LOAD 'file' USING PigStorage(',') as (id:int, name:chararray, age:int); grouped_data = GROUP details by age; dump grouped_data; Output being (23,{(1,John,23),(4,Bob,23)}) (24,{(2,James,24),(5,Bill,24)}) (30,{(3,Alice,30)}) Further more, if you describe the schema of the grouped data, you would see as below describe grouped_data; grouped_data: {group: int,details: {(id: int,name: chararray,age: int)}} You can explore more here ... View more

Under the assumption that the file is Text and each line represent one record, you could read the file line by line and map each line to a Row. Then you can create a data frame form the RDD[Row] something like sqlContext.createDataFrame(sc.textFile("<file path>").map { x => getRow(x) }, schema) I have the below basic definition for creating the Row from your line using substring. But you can use your own implementation. def getRow(x : String) : Row={ val columnArray = new Array[String](4) columnArray(0)=x.substring(0,3) columnArray(1)=x.substring(3,13) columnArray(2)=x.substring(13,18) columnArray(3)=x.substring(18,22) Row.fromSeq(columnArray) } If the records are not delimited by a new line, you may need to use a FixedLengthInputFormat and read the record one at a time and apply the similar logic as above. The fixedlengthinputformat.record.length in that case will be your total length, 22 in this example. Instead of textFile, you may need to read as sc.newAPIHadoopRDD ... View more