HIVESERVER2 doesn't work CDP 7.0.3 Fresh install

[#1]

Hi guys

I just install a fresh cluster with CDP 7.0.3 (7.0.3-1.cdh7.0.3.p0.1635019) in "Trial period" to practice and probe the environment...

The cluster has 3 masters and 5 workers and only has HDFS (no HA mode), Hive, Impala, Zookeeper, Spark and Yarn services installed with integrated postgre database, no ldap, no tls...

All works fine except HIVESERVER2 that doesn't start... i try to debug the error but i can't

The messages from /var/log/hive/hadoop-cmf-hive-HIVESERVER2-hostname.log

2020-03-13 13:15:54,948 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:15:55,681 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist
2020-03-13 13:16:02,898 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:16:03,546 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist
2020-03-13 13:16:11,600 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:16:12,226 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist
2020-03-13 13:16:21,776 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:16:22,400 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist

And this message from ClouderaManager UI:

stderr

SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
2020-03-13 12:54:09: Starting HiveServer2
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported!
	at org.apache.hadoop.hive.conf.HiveConf.validateExecutionEngine(HiveConf.java:5594)
	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5578)
	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5434)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:100)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:84)
	at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

stdout

Fri Mar 13 12:54:06 UTC 2020
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
using /usr/java/jdk1.8.0_232-cloudera as JAVA_HOME
using 7 as CDH_VERSION
using /opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/lib/hive as HIVE_HOME
using /var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2 as HIVE_CONF_DIR
using /opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/lib/hadoop as HADOOP_HOME
using /var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2/yarn-conf as HADOOP_CONF_DIR
CONF_DIR=/var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2
CMF_CONF_DIR=
using /etc/spark/conf as SPARK_CONF_DIR
WARN: spark-defaults.conf already exists in /var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2. Skipping symlink creation.

I try to search the messages on Internet but i can't find something to help me.

Can help me? 

So many thanks.

[#2 · Accepted solution]

Hi Sergiete

The error is described here:

Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported!

mr execution engine stands for "MapReduce Execution Engine".

In Hive3 there is only Hive on Tez or Hive on Spark as an execution engine available:

https://docs.cloudera.com/runtime/7.0.3/hive-introduction/topics/hive_whats_new_in_this_release_hive...

You need to change your execution engine!

[#3]

Hi again...

I continue researching... i was tryed to find this config into the service configuration on cloudera manager -> hive -> configuration but i can't find:

hive.query.redaction.rules

After that i tryed to add another instance of hiveserver2 because i only have one.. the new instance present the same behaviour.

Finally i went to the /var/run/cloudera-scm-agent/process/ directory and found this in the hive-site.xml:

[user@host 1546333361-hive-HIVESERVER2]# cat hive-site.xml | grep redaction
    <name>hive.query.redaction.rules</name>
    <value>/var/run/cloudera-scm-agent/process/1546333361-hive-HIVESERVER2/redaction-rules.json</value>

And the file:

[user@host 1546333361-hive-HIVESERVER2]# cat redaction-rules.json
{
  "version": 1,
  "rules": [
    {
      "description": "Redact passwords from json files",
      "trigger": "password",
      "search": "\"password\"[ ]*:[ ]*\"[^\"]+\"",
      "caseSensitive": false,
      "replace": "\"password\": \"LOG-REDACTED\""
    },
    {
      "description": "Redact password= and password:",
      "trigger": "password",
      "search": "password[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "password=LOG-REDACTED"
    },
    {
      "description": "Redact passwd= and passwd:",
      "trigger": "passwd",
      "search": "passwd[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "passwd=LOG-REDACTED"
    },
    {
      "description": "Redact pass= and pass:",
      "trigger": "pass",
      "search": "pass[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "pass=LOG-REDACTED"
    },
    {
      "description": "Redact PASSWORD,",
      "trigger": "PASSWORD,",
      "search": "PASSWORD,[^\"\\\\]+",
      "caseSensitive": false,
      "replace": "PASSWORD, LOG-REDACTED"
    },
    {
      "description": "Redact secret= and secret:",
      "trigger": "secret",
      "search": "secret[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "secret=LOG-REDACTED"
    },
    {
      "description": "Credit Card numbers (with separator)",
      "search": "\\d{4}[^\\w:]\\d{4}[^\\w:]\\d{4}[^\\w:]\\d{4}",
      "caseSensitive": true,
      "replace": "XXXX-XXXX-XXXX-XXXX"
    },
    {
      "description": "Social Security numbers (with separator)",
      "search": "\\d{3}[^\\w:]\\d{2}[^\\w:]\\d{4}",
      "caseSensitive": true,
      "replace": "XXX-XX-XXXX"
    }
  ]
}

Any suggest?

[#4 · Accepted solution]

Hi Sergiete

The error is described here:

Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported!

mr execution engine stands for "MapReduce Execution Engine".

In Hive3 there is only Hive on Tez or Hive on Spark as an execution engine available:

https://docs.cloudera.com/runtime/7.0.3/hive-introduction/topics/hive_whats_new_in_this_release_hive...

You need to change your execution engine!

[#5]

Hi..

So many thanks Silvan... i think that i found the "issue" is in the traslation to Spanish of the interface:

In English the interface inform about hiveserver2 on Hive and Hive on Tez

But in Spanish not...

Any way i'll read the refered documentation... like i said is my first CDP... i intalled so many CDH in the pass... but is my first time with CDP.

Regards.

[#6]

Hello again Silvan

Like you said the problem was the engine, after install Hive on Tez, the issue persist, finally i was changed the engine from MR to Spark and HIVESERVER2 of Hive start propertly

So many thanks for your help!

[#7]

@silvan 

I did a fresh install of Tez followed by HMS followed by Hive on Tez.

But Hive on TEZ not coming up and throws below error, Kindly suggest?

- Is there any configuration property needs to updated in Tez or HMS or at some place?

Error:

org.apache.hive.service.ServiceException: org.apache.hive.service.ServiceException: Unable to setup tez session pool
at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:690) ~[hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
Caused by: org.apache.hive.service.ServiceException: Unable to setup tez session pool

2020-04-16 15:24:13,671 ERROR org.apache.hive.service.server.HiveServer2: [main]: Error starting Tez sessions:
org.apache.hive.service.ServiceException: Unable to setup tez session pool
at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:793) ~[hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763) ~[hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1587049942856_0006 failed 2 times due to AM Container for appattempt_1587049942856_0006_000002 exited with exitCode: 1

- VM

[#8 · Accepted solution]

Hello @Sergiete ,

thank you for raising our attention that there was an improvement point in Cloudera CDP documentation.

We have added clarification in our CDP documentation about "Installing Hive on Tez" [1] and that the Spark execution engine is not supported, as it's been replaced by Tez [2].

Based on our updated documentation [1] the correct order of installing the Hive service is:

1. Install the Hive service, designated Hive on Tez in CDP.
   HiveServer is installed automatically during this process.
2. Install HMS, which is designated Hive.

Best regards:

Ferenc

[1] https://docs.cloudera.com/runtime/7.0.3/hive-introduction/topics/hive_installing_on_tez.html

[2] https://docs.cloudera.com/runtime/7.0.3/hive-introduction/topics/hive-unsupported.html