Running in HDP 2.4
hive> SELECT COUNT(*) from tweets;
I get
Error: Failure while running task:java.lang.IllegalArgumentException: tez.runtime.io.sort.mb 859 should be larger than 0 and should be less than the available task memory (MB):789
I have tried setting tez.runtime.io.sort.mb to 789 and also to 600.
I have tried running hive thus:
hive --hiveconf hive.tez.container.size=1024
Always the same error.
My config is:
Using cores=16 memory=64GB disks=4 hbase=True
Profile: cores=16 memory=49152MB reserved=16GB usableMem=48GB disks=4
Num Container=8
Container Ram=6144MB
Used Ram=48GB
Unused Ram=16GB
***** mapred-site.xml *****
mapreduce.map.memory.mb=6144
mapreduce.map.java.opts=-Xmx4096m
mapreduce.reduce.memory.mb=6144
mapreduce.reduce.java.opts=-Xmx4096m
mapreduce.task.io.sort.mb=1792
***** yarn-site.xml *****
yarn.scheduler.minimum-allocation-mb=6144
yarn.scheduler.maximum-allocation-mb=49152
yarn.nodemanager.resource.memory-mb=49152
yarn.app.mapreduce.am.resource.mb=6144
yarn.app.mapreduce.am.command-opts=-Xmx4096m
***** tez-site.xml *****
tez.am.resource.memory.mb=6144
tez.am.java.opts=-Xmx4096m
***** hive-site.xml *****
hive.tez.container.size=6144
hive.tez.java.opts=-Xmx4096m
hive.auto.convert.join.noconditionaltask.size=1342177000
Any help is much appreciated. TIA!!
