Can you please try removing the master("local[*]") from the spark code and pass it as a parameter in spark submit -- master yarn --deploy-mode cluster.. It should work.. ... View more

@bkosaraju thanks for your reply. So you are suggesting for Hive with LLAP or postgres DB(any RDBMS) in cluster? If you suggest Hive with LLAP: We are using HDP-2.5 how can i configure Hive with LLAP? Ya in the batch processing i ll be required to insert and update records. we are using hive1.2 and hope it supports ACID and updates? Do you think it will help to create indexes on these hive tables? https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Indexing ... View more

@Josh Elser @Harald Berghoff If the jira patch is pushed then i believe we can use this cloudera connector and donot need to rewrite hbase salting and spark partitioner from the below post is it? We can just use the connector api instead isnt it? http://www.opencore.com/blog/2016/10/efficient-bulk-load-of-hbase-using-spark/ Also we have another connector from hortonworks not sure if this supports bulk loading the previous one is from cloudera and rdd version. however the below one supports dataframe version of spark https://github.com/hortonworks-spark/shc It would be good if this connector supports dataframe bulk loading? It supports the standard write not sure abt bulk load though https://issues.apache.org/jira/browse/HBASE-15336 ... View more

@Pavan Kumar KondaIt depends on lot of constraints like compression, serialization and whether the storage format is splittable etc. I think ORC is just for hive. Avro Avro is language neutral data serialization Writables has the drawback that they do not provide language portability. Avro formatted data can be described through language independent schema. Hence Avro formatted data can be shared across applications using different languages. Avro stores the schema in header of file so data is self-describing. Avro formatted files are splittable and compressible and hence it’s a good candidate for data storage in Hadoop ecosystem. Schema Evolution – Schema used to read a Avro file need not be same as schema which was used to write the files. This makes it possible to add new fields. Parquet Parquet is a columnar format. Columnar formats works well where only few columns are required in query/ analysis. Only required columns would be fetched / read, it reduces the disk I/O. Parquet is well suited for data warehouse kind of solutions where aggregations are required on certain column over a huge set of data. Parquet provides very good compression upto 75% when used with compression formats like snappy. Parquet can be read and write using Avro API and Avro Schema. It also provides predicate pushdown, thus reducing further disk I/O cost. ... View more

@Harald Berghoff & @Josh Elser thanks for your reply. OK will try to use HFiles from according to the link which i knew exists. However in that link it says about Spark-hbase connector to support bulk loading in future. Now the jira says its resolved then does it mean now we can use the spark-hbase connector? https://issues.apache.org/jira/browse/HBASE-14150 ... View more

@mqureshi yes i tried making it transient as well no luck and inside .map map(path => sc.textFile(path)) its not reading the contents of the file i thnk its just returning the string which is strange if i do sc.textfile outside map function it returns the data. Above is the whole code nothing else to it. Looks simple.. probably rather than calling sc.textfile inside map may be i need to fetch data using s3 api as shown in this link. http://michaelryanbell.com/processing-whole-files-spark-s3.html but not sure i might still get task serailization error???? any ideas to better implement this requirement ... View more

Hi Tibor thanks for your reply. I have looked at the above link. I have a dataset of structure i.e. .gz file which iam reading in spark abcdefghij; abc=1234 xyz=987 abn=567 ubg=345 after pivot abcdefghij abn ubg abc xyz abcdefghij 567 987 1234 987 and so on. All the above columns are string columns and abn values are duplicated. So as they are strings and iam just looking to split the data thats why i dont need aggregation, i just need pivot. Any ideas to acheive this in spark scala? thank u ... View more

@Matt Burgess currently we are using hive 0.13 version with cdh 5.3.1 and also we have nifi 0.7. Its running a hiveserver2 service. Currently we cannot change hive version on the cdh server. Is there any other quicker alternative to extract data from hive? Thank you.. ... View more