Member since
06-03-2016
2
Posts
1
Kudos Received
0
Solutions
06-30-2016
09:02 AM
2 Kudos
if you are using mysql database as metastore then you can use following query to get partition locations directly from metastore. select TBLS.TBL_NAME,PARTITIONS.PART_NAME,SDS.LOCATIONfrom SDS,TBLS,PARTITIONSwhere PARTITIONS.SD_ID = SDS.SD_IDand TBLS.TBL_ID=PARTITIONS.TBL_IDorder by 1,2;
... View more
06-06-2016
09:49 AM
If you say that increasing the heap doesn't help are we talking about decent sizes like 8GB+? Also did you increase the java opts AND the container size? set hive.tez.java.opts="-Xmx3400m"; set hive.tez.container.size =
4096; If yes then you most likely have a different problem like for example loading data into a partitioned table. ORC writers keep one buffer open for every output file. So if you load badly to a partitioned table they will keep a lot of memory open. There are ways around it like optimized sorted load or the distribute by keyword. http://www.slideshare.net/BenjaminLeonhardi/hive-loading-data If however you use significantly less than 4-8GB for the task then you should increase that.
... View more