Hi Binu, Thank you for your advice. I've done some experiments based on Hortonwork's Hive Benchmark to compare the performance of Hive and Spark to analyse S3 data. I assume that both the two methods need to load S3 data into HDFS and create hive tables pointing to the HDFS data. The reason I also create hive tables for Spark is that I want to use HiveQL and I don't want to write too many codes for registering temp tables for Spark. I observed the following things: for tpcds_10GB, load S3 text table into HDFS took 233 seconds, which is acceptable create orc_table and table analysing took very long time (more than one hour, so I terminate it manually), which is unacceptable. execute query12.sql, hive text table (17.64 secs), hive orc table(6.013 secs), spark (45 secs). There are also some example that spark outperform hive (i.e. query15.sql) . My questions: As table analysing for orc table taking very long time, is there a way to avoid re-analysing tables when load S3 to HDFS? If there is no way to avoid the long-time table optimisation operation, I might not be able to use the Hive method because in my project there are many tables and all of them are very large. Should I alway use HiveContext rather than SQLContex? Because I find when I use SQLContext class, some of hive script can't execute. Looking forward to your reply! Thank you very much! ... View more

Thanks for your reply! The key requirements are listed as follows: one PDF document is very small, less than 1MB, but the number is huge. Total size will be 10 TB or more. they are archive files. Once they are indexed, update will not happen usually. They are just used for querying and downloading. Any suggestions for this scenario? Thanks ... View more

I solved the problem by removing "-D fs.s3a.fast.upload=true" but still don't know the reason. Do some one know why "fs.s3a.fast.upload=true" will cause java heap space problem? Thanks ... View more

Thanks Stevel. My project will have lots of tables and some of them will have partitions. In addition, I need to do table analyse to improve query performance. My question is that "if I save these data to S3 and load them back to HDFS next time when a new cluster starts, is there a way to avoid repairing partitions and doing table analyse again?" Because I find that table analyse usually will take lots of time. Another question is "Can I use HDCloud if my S3 data locates in a region that doesn't support HDCloud? If I can use HDCloud, what's about the performance and price?" ... View more

Thank you very much! ... View more

Thank you very much for your valuable suggestion. I am going to try SparkSQL. Actually, I am now facing a couple of problems of letting Hadoop tools (sqoop, hive) to work with S3 data. Hopefully, I can get some further guides from you. My project is to integrate data from different resources (i.e. S3 and databases), so as to analyse the data. My customer wants to store everything in S3, and use Hadoop as ephemeral cluster. I am now trying to figure out the following things : 1) how to transfer data from database to S3, 2) and how to analyse data already in S3. I've already done some experiments as listed below: 1) using sqoop to transfer data from database to S3 directly (but failed). My question is "Is sqoop a good choice in this scenario?" 2) query data and write into a hive table pointing to S3. (but very slow when there are many partitions). My test was based on the hive benchmark data "tpcds_bin_partitioned_orc_10”. I guess it is because there are many sub-directories and many small files in each directory, leading to write data to S3 very costly. I am going to try SparkSQL according to your suggestion. But I am still curious about which way is better to use Hive with S3 data in terms of performance, loading S3 data to HDFS for Hive analyse and then saving results back to S3, or doing analyse directly on hive table point to S3 data. Thank you very much for your time. ... View more