question Re: Hive tez memory error in Support Questions

Hive tez memory error

ed_day — Thu, 12 Jan 2017 04:05:00 GMT

Running in HDP 2.4

hive> SELECT COUNT(*) from tweets;

I get

Error: Failure while running task:java.lang.IllegalArgumentException: tez.runtime.io.sort.mb 859 should be larger than 0 and should be less than the available task memory (MB):789

I have tried setting tez.runtime.io.sort.mb to 789 and also to 600.

I have tried running hive thus:

hive --hiveconf hive.tez.container.size=1024

Always the same error.

My config is:

 Using cores=16 memory=64GB disks=4 hbase=True
 Profile: cores=16 memory=49152MB reserved=16GB usableMem=48GB disks=4
 Num Container=8
 Container Ram=6144MB
 Used Ram=48GB
 Unused Ram=16GB
 ***** mapred-site.xml *****
 mapreduce.map.memory.mb=6144
 mapreduce.map.java.opts=-Xmx4096m
 mapreduce.reduce.memory.mb=6144
 mapreduce.reduce.java.opts=-Xmx4096m
 mapreduce.task.io.sort.mb=1792
 ***** yarn-site.xml *****
 yarn.scheduler.minimum-allocation-mb=6144
 yarn.scheduler.maximum-allocation-mb=49152
 yarn.nodemanager.resource.memory-mb=49152
 yarn.app.mapreduce.am.resource.mb=6144
 yarn.app.mapreduce.am.command-opts=-Xmx4096m
 ***** tez-site.xml *****
 tez.am.resource.memory.mb=6144
 tez.am.java.opts=-Xmx4096m
 ***** hive-site.xml *****
 hive.tez.container.size=6144
 hive.tez.java.opts=-Xmx4096m
 hive.auto.convert.join.noconditionaltask.size=1342177000

Any help is much appreciated. TIA!!

Re: Hive tez memory error

ssubhas — Thu, 12 Jan 2017 04:27:50 GMT

@ed day

Could you check the value for property 'tez.task.resource.memory.mb'?

And try increasing the value of the property tez.task.resource.memory.mb=2048 and check.

Re: Hive tez memory error

srai1 — Thu, 12 Jan 2017 05:37:11 GMT

@ed day You can also look at (tez-config) tez.task.launch.cmd-opts via Ambari to see the amount of heap set for -Xmx and also ensure it matches the value in hive.tez.java.opts (hive config)

Re: Hive tez memory error

ed_day — Thu, 12 Jan 2017 18:11:11 GMT

Thanks Sindhu. I upped it from 1024 to 2048 but still same error.

Re: Hive tez memory error

ed_day — Thu, 12 Jan 2017 18:27:28 GMT

Thanks srai. There is no -Xmx in either of these. For me, tez.task.launch.cmd-opts is:

-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC

hive.tez.java.opts is:

-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps

Re: Hive tez memory error

ed_day — Thu, 12 Jan 2017 18:58:12 GMT

I've tried the following:

Upped hive.tez.container.size to 2048

tez.am.resource.memory.mb = 2048

hive.tez.container.size=4096

tez.runtime.io.sort.mb = 409

Re: Hive tez memory error

ed_day — Thu, 12 Jan 2017 19:21:09 GMT

tez.runtime.io.sort.mb to 1024

Changed: HADOOP_USER_NAME=hdfs hive --hiveconf hive.tez.container.size=2048

Now a different error. So that is good...ish.

Re: Hive tez memory error

rbalamohan — Fri, 13 Jan 2017 00:02:06 GMT

You can remove "hive.tez.java.opts=-Xmx4096m". Tez automatically takes up 80% of the container size allocated to it. As per your hive-site.xml and yarn-site.xml, "hive.tez.container.size=6144", "yarn.scheduler.minimum-allocation-mb=6144". So ~4915MB should be automatically be assigned without specifying any Xmx value in hive.tez.java.opts.
Remove "--hiveconf hive.tez.container.size=2048" from hive cli command. By specifying 2048, it would end up under utilizing the memory.
After incorporating #1, with the current config in hive-site, yarn-site you have posted, can you run "hive" cli without specifying any option and run the query?
For simple like select count(*) from table, it should not launch tez job if it had enough information about the rows in metastore. Run "analyze table tweets compute statistics" and re-run this select count(*) statement. It should fetch information from metastore directly as opposed to launching the tez job.

Re: Hive tez memory error

ahoneycutt — Thu, 25 May 2017 22:25:04 GMT

This ultimately lead me to the solution. I started by removing hive.tez.java.opts -Xmx200m which gave me a slightly higher value in my error message increased from 192 MB to the 245 MB show below but my sort size was still to big.

tez.runtime.io.sort.mb 3244 should be larger than 0 and should be less than the available task memory (MB):245

Removing it here allows me to leave the sort size and task completes as expected.

tez.task.launch.cmd-opts -Xmx256m{{heap_dump_opts}} CHANGED TO RECOMMENDED shown next. -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC{{heap_dump_opts}}

Re: Hive tez memory error

ahoneycutt — Thu, 25 May 2017 22:31:01 GMT

Thanks, I didn't see this until after the struggle but this worked for me.

Re: Hive tez memory error

mqadri — Thu, 13 Jul 2017 02:28:41 GMT

@Srinivasarao Daruna, can you give details of the HW and Spark config

Re: Hive tez memory error

ed_day — Thu, 13 Jul 2017 15:26:55 GMT

Hi Muji, the above solved my problem. Thanks anyway.