I have a 35 node Hadoop(HDP2.2 ) cluster with 24 data nodes. I have dedicated Hive server( 0.14) with a memory of 68 GB, and 40GB heap space. The PermGen space for Name node has been configured as 512 MB.
Below are the mapreduce configurations:
Default virtual memory for job's map-task: 5120MB Default virtual memory for job's reduce-task: 5120MB Map-side sort buffer memory: 1024 MB
Occasionally, after executing Hive queries, I get the below error in Hive server logs:
2016-04-24 22:12:41,529 ERROR [HiveServer2-Background-Pool: Thread-212461]: ql.Driver (SessionState.java:printError(833)) - FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask. PermGen space 2016-04-24 22:12:41,529 ERROR [HiveServer2-Background-Pool: Thread-212461]: operation.Operation (SQLOperation.java:run(199)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask. PermGen space at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:314) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:146) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415)
Any suggestions to resolve this issue is greatly appreciated.
What value you are using for the "XX:MaxPermSize=512m" following JVM parameter (128m/ 256m or 512m)?
You will need to increase the value a bit compared to your default "MaxPermSize" value
As the error seems to be coming while running the hive client hence you should check if the hive component is having enough PermSize defined.
Both PermGen size and Max PermGen size is set to 512MB. I have not set PermGen specifically for Hive client, so as per my understanding, the Hadoop settings are used by default.
One comment 40GB is a TON. I am pretty sure this will mostly result in Garbage Collection outages. I had an environment where we ran continuous 25 queries per second which is pretty much the limit and we could do it with 8 GB of RAM. Now I have seen Memory leaks in Hiveserver Especially older versions so keep an eye out for that.
Back then I used the following settings for hiveserver. Seemed to work fine. However in Java8 you might want G1GC.
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -XX:NewRatio=12 -Xmx40960m -Xms40960m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit -XX:+UseParNewGC -XX:+PrintGCTimeStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:/tmp/gc_$SERVICE.log"
Thanks for the response. The Hive heap space was set to 40GB, as lesser value was throwing OOM error . I'm aware of the memory leak error in the Hive version used, but I would like to understand the cause of PermGen space error in Hive server or how I could fine tune the configurations to avoid it.
Just saying perhaps you should open a support ticket and get the memory leak fixed. Anyhow you can try the parameters I added on top to configure your hiveserver. Add them to the hive-env.
if [ "$SERVICE" = "hiveserver2" ]; then
PermGen issues are weird because they are normally related to classloader issues. And Hive should not load new classes every time it runs a query. So you should check how big you permgen is and perhaps increase it with the value Joy gave. If that doesn;t help in the long run you might have another memory leak which also should be a support ticket.
Thank you, I'm planning to upgrade my cluster to use a higher version of Hive. I have one more query, would Hive queries or UDF's cause PermGen error. I had noticed below statement in error logs relating to Hive UDF:
Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask.PermGen space
hmmm that is possible. I think I read about an error in a Hive version where each register call of the jar added the class definitions to the permgen space. ( i.e. every time the script was called it would increase the permgen space a bit )
There are a couple similar jiras out there: