Community Articles

Find and share helpful community-sourced technical articles.
Labels (1)
avatar
Expert Contributor

By Default, HiveServer2 and HiveMetastore does not have configuration for HeapDump on OOM. Production clusters have OOM and since the HeapDump on OOM is not configured, root cause analysis of the issue is obstructed.

Navigate as below in Ambari:

Ambari UI > Hive > Configs > Advanced hive-env > hive-env template

Add following :

if [ "$SERVICE" = "metastore" ]; then
  export HADOOP_HEAPSIZE={{hive_metastore_heapsize}} # Setting for HiveMetastore
else
  export HADOOP_HEAPSIZE={{hive_heapsize}} # Setting for HiveServer2 and Client
fi

export HADOOP_CLIENT_OPTS="-Xmx${HADOOP_HEAPSIZE}m -Xloggc:/var/log/hive/gc.log-$SERVICE-`date +'%Y%m%d%H%M'`
-XX:ErrorFile=/var/log/hive/hive-metastore-error.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/hive/
$HADOOP_CLIENT_OPTS"

if [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_CLIENT_OPTS="-Xmx${HADOOP_HEAPSIZE}m -Xloggc:/var/log/hive/gc.log-$SERVICE-`date +'%Y%m%d%H%M'`
-XX:ErrorFile=/var/log/hive/hive-server2-error.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/hive/  -XX:+PrintGCDateStamps $HADOOP_CLIENT_OPTS"
fi
1,896 Views
0 Kudos
Comments

The script seems wrong.
With this configuration, the lines for "hiveserver2" will never be valid. The settings in the previous lines "export HADOOP_CLIENT_OPTS=" will overwirte them all with the "$HADOOP_CLIENT_OPTS" at the end. The name of ErrorFile will become "hive-metastore-error.log-`date +'%Y%m%d%H%M'" for example.
The script should be:

if [ "$SERVICE" = "metastore" ]; then
....
elif [ "$SERVICE" = "hiveserver2" ]; then
....
fi