HiveServer2 is going OutOfMemory very frequently

[#1]

SYMPTOM:

hiveserver2 is failing with OutOfMemoryError very frequently, user increased HiveServer2 heap size to 16GB but the still faces the same issue.

we enabled heap dump using -XX:+HeapDumpOnOutOfMemoryError to see what object are causing the hiveserver2 heap to grow such big.

ROOT CAUSE:

After initial analysis we notice that some of the connection objects are set with the fetchSize of 50M,we enquire user about that which reveals that there are some connection strings where the fetchSize was set as 50M.

with the fetchSize setting in place HiveServer2 is taking a lot of heap space while fetching the query result and going into out of memory.

WORKAROUND:

NA

RESOLUTION:

Ask user to remove the fetchSize setting from connection string. there is an improvement in place in the hive community tracked as https://issues.apache.org/jira/browse/HIVE-14901 to put a 'guardrail' against someone using too high a value for fetchSize.

[#2]

got the same issues multiple times on our HDP 2.3 cluster.