Can you please try removing the master("local[*]") from the spark code and pass it as a parameter in spark submit -- master yarn --deploy-mode cluster.. It should work.. ... View more

@Pavan Kumar KondaIt depends on lot of constraints like compression, serialization and whether the storage format is splittable etc. I think ORC is just for hive. Avro Avro is language neutral data serialization Writables has the drawback that they do not provide language portability. Avro formatted data can be described through language independent schema. Hence Avro formatted data can be shared across applications using different languages. Avro stores the schema in header of file so data is self-describing. Avro formatted files are splittable and compressible and hence it’s a good candidate for data storage in Hadoop ecosystem. Schema Evolution – Schema used to read a Avro file need not be same as schema which was used to write the files. This makes it possible to add new fields. Parquet Parquet is a columnar format. Columnar formats works well where only few columns are required in query/ analysis. Only required columns would be fetched / read, it reduces the disk I/O. Parquet is well suited for data warehouse kind of solutions where aggregations are required on certain column over a huge set of data. Parquet provides very good compression upto 75% when used with compression formats like snappy. Parquet can be read and write using Avro API and Avro Schema. It also provides predicate pushdown, thus reducing further disk I/O cost. ... View more

Hi Tibor thanks for your reply. I have looked at the above link. I have a dataset of structure i.e. .gz file which iam reading in spark abcdefghij; abc=1234 xyz=987 abn=567 ubg=345 after pivot abcdefghij abn ubg abc xyz abcdefghij 567 987 1234 987 and so on. All the above columns are string columns and abn values are duplicated. So as they are strings and iam just looking to split the data thats why i dont need aggregation, i just need pivot. Any ideas to acheive this in spark scala? thank u ... View more

@Bernhard Walter thanks for the reply. ... View more