But will the total files too many? i had smaller sized files before with 512M, was slow as well. we only have 5 nodes now so too many files will not help?   Thanks Shannon ... View more

Thanks Lars, trying to understand how it appies here, should i try to increate the blozks size to 4G, and keep each file under 4G (3-4G)?   Shannon ... View more

I ran compute stats. I didnot do any compression, as i read it will slow down.   Shannon ... View more

Took a quick look at bucketing, i dont have one column  that is used the most, and this table will join other tables later.   As far as controlling the number of files, since i have an idea of the data size, i can controll the file size when doing insert -select i can set hive.merge.size.per.task and hive.merge.smallfiles.avgsize.   My questions, 1, in this case, how big can i set the block size, 1G, 2G, will it hurt if set to big? 2, how big should i use for the file, right now i set to 4G, should i increase? 3, i have 5 nodes, with that for each partition it is better to have at least 5 files?   Thanks Shannon ... View more

Java Heap Size of Catalog Server in Bytes - i set to 4G   Java Heap Size of Impala Llama ApplicationMaster in Bytes  - does this have anything to do with impala query? ... View more

Thanks, i will take a look at bucketing.   Do you mean run compute stats, yes i did that. ... View more

When i run a simple select with where id ='xxx', i see there are 1111 jobs/tasks(?), it is bcoz my block size is 512 and total is 500G, so 1000+ blocks to scan? If so i should increase my block size? ... View more

Also block size is 512M ... View more