- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Hive query OutOfMemoryError: Java heap space
- Labels:
-
Apache Hive
-
Apache Tez
Created 05-25-2016 01:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am issuing a command that is executing about 1500 xpaths on a single XML file (it is about 10MB in size).
I am getting the error in the title. I have tried increasing just about every configuration setting I know related to Hive/Tez's java heap space.
Nothing seems to work. I restart the server after every configuration change.
I also went and changed hive-env.sh to -Xmx8g and it still doesn't seem to fix the issue. I ran -verbose:gc and see that the gc stops at ~1000MB. Why wouldn't that go on up to 8G if I changed -Xmx to be 8g?
Is there anyway to tell if it is the client breaking and needing more heap or the map jobs?
Created 05-25-2016 07:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Kevin Vasko Hi Kevin, can you export the below in hive-env.sh file from Ambari then restart the affected components :
---
export HADOOP_CLIENT_OPTS="-Xmx6144m"
---
Then run the below command on the node where HS2 is running as hive user, to check the heap size (MaxHeapSize):
# jmap -heap <PID-of-HS2>
Thanks !
Created 11-28-2017 05:05 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have faced same issue .Please increase the memory before running the hive query .But if you are not able to do grep on xml then you have to split your file on the basis of tags by using gawk.
set mapreduce.map.memory.mb=9000;
set mapreduce.map.java.opts=-Xmx7200m;
set mapreduce.reduce.memory.mb=9000;
set mapreduce.reduce.java.opts=-Xmx7200m;

- « Previous
-
- 1
- 2
- Next »