Support Questions
Find answers, ask questions, and share your expertise

Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

I have a 35 node Hadoop(HDP2.2 ) cluster with 24 data nodes. I have dedicated Hive server( 0.14) with a memory of 68 GB, and 40GB heap space. The PermGen space for Name node has been configured as 512 MB.

Below are the mapreduce configurations:

Default virtual memory for job's map-task: 5120MB
Default virtual memory for job's reduce-task: 5120MB
Map-side sort buffer memory: 1024 MB

Occasionally, after executing Hive queries, I get the below error in Hive server logs:

2016-04-24 22:12:41,529 ERROR [HiveServer2-Background-Pool: Thread-212461]: ql.Driver (SessionState.java:printError(833)) - FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask. PermGen space
2016-04-24 22:12:41,529 ERROR [HiveServer2-Background-Pool: Thread-212461]: operation.Operation (SQLOperation.java:run(199)) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask. PermGen space
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:314)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:146)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)

Any suggestions to resolve this issue is greatly appreciated.

Thanks

8 REPLIES 8

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

What value you are using for the "XX:MaxPermSize=512m" following JVM parameter (128m/ 256m or 512m)?

You will need to increase the value a bit compared to your default "MaxPermSize" value

As the error seems to be coming while running the hive client hence you should check if the hive component is having enough PermSize defined.

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

Could you please clarify which parameter you are referring to?

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

Both PermGen size and Max PermGen size is set to 512MB. I have not set PermGen specifically for Hive client, so as per my understanding, the Hadoop settings are used by default.

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

One comment 40GB is a TON. I am pretty sure this will mostly result in Garbage Collection outages. I had an environment where we ran continuous 25 queries per second which is pretty much the limit and we could do it with 8 GB of RAM. Now I have seen Memory leaks in Hiveserver Especially older versions so keep an eye out for that.

Back then I used the following settings for hiveserver. Seemed to work fine. However in Java8 you might want G1GC.

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -XX:NewRatio=12 -Xmx40960m -Xms40960m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit -XX:+UseParNewGC -XX:+PrintGCTimeStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:/tmp/gc_$SERVICE.log"

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

Thanks for the response. The Hive heap space was set to 40GB, as lesser value was throwing OOM error . I'm aware of the memory leak error in the Hive version used, but I would like to understand the cause of PermGen space error in Hive server or how I could fine tune the configurations to avoid it.

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

Just saying perhaps you should open a support ticket and get the memory leak fixed. Anyhow you can try the parameters I added on top to configure your hiveserver. Add them to the hive-env.

Surrounded by

if [ "$SERVICE" = "hiveserver2" ]; then

fi

PermGen issues are weird because they are normally related to classloader issues. And Hive should not load new classes every time it runs a query. So you should check how big you permgen is and perhaps increase it with the value Joy gave. If that doesn;t help in the long run you might have another memory leak which also should be a support ticket.

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

Thank you, I'm planning to upgrade my cluster to use a higher version of Hive. I have one more query, would Hive queries or UDF's cause PermGen error. I had noticed below statement in error logs relating to Hive UDF:

Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask.PermGen space

Re: Getting java.lang.OutOfMemoryError: PermGen space issue in Hive server

hmmm that is possible. I think I read about an error in a Hive version where each register call of the jar added the class definitions to the permgen space. ( i.e. every time the script was called it would increase the permgen space a bit )

There are a couple similar jiras out there:

https://issues.apache.org/jira/browse/HIVE-11408