Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Reboot HiveServer2 after Detected pause in JVM or host machine

Reboot HiveServer2 after Detected pause in JVM or host machine

Contributor

In moment shutdown HiveServer2 see a lot of GC warning

[root@serv02 hive]$ cat hiveserver2.log | grep GC | grep WARN | tail -20
2018-10-16 05:31:21,969 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 25250ms
2018-10-16 05:31:51,089 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 28618ms
2018-10-16 05:32:21,611 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 30019ms
2018-10-16 05:33:01,439 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 39326ms
2018-10-16 05:33:27,008 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 25067ms
2018-10-16 05:33:49,902 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 22391ms
2018-10-16 05:34:17,273 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 24213ms
2018-10-16 05:35:57,881 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 25266ms
2018-10-16 06:44:20,963 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 11909ms
2018-10-16 06:46:13,689 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 46503ms
2018-10-16 06:46:42,536 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 28346ms
2018-10-16 06:47:17,036 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 33999ms
2018-10-16 06:47:32,386 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 14844ms
2018-10-16 06:49:50,693 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 14302ms
2018-10-16 06:50:19,092 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 15315ms
2018-10-16 06:52:04,384 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 14102ms
2018-10-16 06:52:22,888 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 15489ms
2018-10-16 07:35:50,690 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 17582ms
2018-10-16 07:36:10,431 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 19238ms
2018-10-16 07:54:04,241 WARN  [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@cae2a97]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 22707ms
		

Hive_env_template


      export HADOOP_USER_CLASSPATH_FIRST=true  #this prevents old metrics libs from mapreduce lib from bringing in old jar deps overriding HIVE_LIB
      if [ "$SERVICE" = "cli" ]; then
      if [ -z "$DEBUG" ]; then
      export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseNUMA -XX:+UseParallelGC -XX:-UseGCOverheadLimit"
      else
      export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
      fi
      fi


      # The heap size of the jvm stared by hive shell script can be controlled via:


      if [ "$SERVICE" = "metastore" ] || [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_HEAPSIZE=53960
fi


      export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS  -Xmx${HADOOP_HEAPSIZE}m"


      # Larger heap size may be required when running queries over large number of files or partitions.
      # By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be
      # appropriate for hive server (hwi etc).




      # Set HADOOP_HOME to point to a specific hadoop install directory
      HADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}


      export HIVE_HOME=${HIVE_HOME:-{{hive_home_dir}}}


      # Hive Configuration Directory can be controlled by:
      export HIVE_CONF_DIR=${HIVE_CONF_DIR:-{{hive_config_dir}}}


      # Folder containing extra libraries required for hive compilation/execution can be controlled by:
      if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
      if [ -f "${HIVE_AUX_JARS_PATH}" ]; then
      export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
      elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
      export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar
      fi
      elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
      export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar
      fi


      export METASTORE_PORT={{hive_metastore_port}}


      {% if sqla_db_used or lib_dir_available %}
      export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:{{jdbc_libs_dir}}"
      export JAVA_LIBRARY_PATH="$JAVA_LIBRARY_PATH:{{jdbc_libs_dir}}"
      {% endif %}
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS{{heap_dump_opts}}"