spark-submit provides the --files tag to upload files to the execution directories. If you have small files that do not change. Alternatively as the others have suggested put it in HDFS ... View more

In general when you use date organized tables partitioning makes sense. If you for example are sure that data has been inserted in the last days you can use this to restrict the amount of data that need to be read. For example by partitioning by day and adding a where condition at the end that only reads the last two days or so. I assume you already use ORC? That data format has automatic indexing which should skip the majority of data blocks of your data set assuming you could give a range the timestamp can be in. ( Like where timestamp > one week ago ) . Since Hive 1.2.0 you also have the ability to read footers first which would make even better use of this feature. ( PEr default tasks are started but will close immediately, with the below setting the files are opened for a quick peek into the footer before tasks are even assigned ) hive.exec.orc.split.strategy=ETL Finally Hive can use statistics for query acceleration but you need to be sure to have updated statistics so autogather needs to be true. It can also lead to wrong results if you sideload data etc. hive.compute.query.using.stats=true hive.stats.autogather=true So in summary use partitioning, ORC, and have a look at the parameters below and try them out. ... View more

I think you are putting it up wrongly, you should give us your requirements ( sub 1s, sub 5 second queries, data volumes queried, data volumes total , number requests per sec/min , reading only?, daily updates or lots of writers? ) and we can give an answer based on that. Benchmarks always change very much depending on requirements. As far as I am concerned they are pretty much meaningless for a specific usecase. Hive Hive is a analytical aggregation tool and it is hard to get queries below 5s, you can tune it for up to hundred parallel queries but it will not work for sub second queries. However we will soon have LLAP which is a persistent process that should bring down the minimum time significantly ( to perhaps 1-2s). The good thing is after that it scales to huge data volumes with very predictable behavior. So if you need to potentially aggregate billions of rows this would be the way to go. Phoenix/HBase Phoenix/HBase is much better for sub second queries that scan a small amount of data ( up to millions of rows ). In these cases you will also get a higher concurrency so it would be a good fit for a webservice, however you need to be careful in the data modeling. And it will not scale as predictably as Hive. Also the only tool that I am aware of where you could have small updates as well. Presto Don't know it too much but as I understood it its main advantage is that you can run probabilistic queries. That provide approximate results very very quickly. Someone else might chime in. Drill Similar to Presto I don't know too much about it, seems to be quite impressive for a variety of usecases but you don't have the wide support of Hive and Phoenix Kylin OLAP engine for VERY fast queries of aggregated data that can be precomputed into a cube. Druid Forgot one, Druid sounds really interesting, they have an OLAP engine based on inverted indexes which might work great to aggregate large but not huge subsets of data. ... View more