Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HIVESERVER2 doesn't work CDP 7.0.3 Fresh install

avatar
Explorer

Hi guys

 

I just install a fresh cluster with CDP 7.0.3 (7.0.3-1.cdh7.0.3.p0.1635019) in "Trial period" to practice and probe the environment...

 

The cluster has 3 masters and 5 workers and only has HDFS (no HA mode), Hive, Impala, Zookeeper, Spark and Yarn services installed with integrated postgre database, no ldap, no tls...

 

All works fine except HIVESERVER2 that doesn't start... i try to debug the error but i can't

 

The messages from /var/log/hive/hadoop-cmf-hive-HIVESERVER2-hostname.log

 

 

2020-03-13 13:15:54,948 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:15:55,681 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist
2020-03-13 13:16:02,898 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:16:03,546 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist
2020-03-13 13:16:11,600 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:16:12,226 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist
2020-03-13 13:16:21,776 INFO  org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml
2020-03-13 13:16:22,400 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist

 

 

And this message from ClouderaManager UI:

 

stderr

 

SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
2020-03-13 12:54:09: Starting HiveServer2
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported!
	at org.apache.hadoop.hive.conf.HiveConf.validateExecutionEngine(HiveConf.java:5594)
	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5578)
	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5434)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:100)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:84)
	at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

 

stdout

 

Fri Mar 13 12:54:06 UTC 2020
JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
using /usr/java/jdk1.8.0_232-cloudera as JAVA_HOME
using 7 as CDH_VERSION
using /opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/lib/hive as HIVE_HOME
using /var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2 as HIVE_CONF_DIR
using /opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/lib/hadoop as HADOOP_HOME
using /var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2/yarn-conf as HADOOP_CONF_DIR
CONF_DIR=/var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2
CMF_CONF_DIR=
using /etc/spark/conf as SPARK_CONF_DIR
WARN: spark-defaults.conf already exists in /var/run/cloudera-scm-agent/process/1546333106-hive-HIVESERVER2. Skipping symlink creation.

 

 

I try to search the messages on Internet but i can't find something to help me.

 

Can help me? 

 

So many thanks.

2 ACCEPTED SOLUTIONS

avatar
New Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Moderator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Explorer

Hi again...

 

I continue researching... i was tryed to find this config into the service configuration on cloudera manager -> hive -> configuration but i can't find:

 

hive.query.redaction.rules

 

After that i tryed to add another instance of hiveserver2 because i only have one.. the new instance present the same behaviour.

 

Finally i went to the /var/run/cloudera-scm-agent/process/ directory and found this in the hive-site.xml:

 

[user@host 1546333361-hive-HIVESERVER2]# cat hive-site.xml | grep redaction
    <name>hive.query.redaction.rules</name>
    <value>/var/run/cloudera-scm-agent/process/1546333361-hive-HIVESERVER2/redaction-rules.json</value>

 

And the file:

 

 

[user@host 1546333361-hive-HIVESERVER2]# cat redaction-rules.json
{
  "version": 1,
  "rules": [
    {
      "description": "Redact passwords from json files",
      "trigger": "password",
      "search": "\"password\"[ ]*:[ ]*\"[^\"]+\"",
      "caseSensitive": false,
      "replace": "\"password\": \"LOG-REDACTED\""
    },
    {
      "description": "Redact password= and password:",
      "trigger": "password",
      "search": "password[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "password=LOG-REDACTED"
    },
    {
      "description": "Redact passwd= and passwd:",
      "trigger": "passwd",
      "search": "passwd[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "passwd=LOG-REDACTED"
    },
    {
      "description": "Redact pass= and pass:",
      "trigger": "pass",
      "search": "pass[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "pass=LOG-REDACTED"
    },
    {
      "description": "Redact PASSWORD,",
      "trigger": "PASSWORD,",
      "search": "PASSWORD,[^\"\\\\]+",
      "caseSensitive": false,
      "replace": "PASSWORD, LOG-REDACTED"
    },
    {
      "description": "Redact secret= and secret:",
      "trigger": "secret",
      "search": "secret[:=][^ \"\\\\]+",
      "caseSensitive": false,
      "replace": "secret=LOG-REDACTED"
    },
    {
      "description": "Credit Card numbers (with separator)",
      "search": "\\d{4}[^\\w:]\\d{4}[^\\w:]\\d{4}[^\\w:]\\d{4}",
      "caseSensitive": true,
      "replace": "XXXX-XXXX-XXXX-XXXX"
    },
    {
      "description": "Social Security numbers (with separator)",
      "search": "\\d{3}[^\\w:]\\d{2}[^\\w:]\\d{4}",
      "caseSensitive": true,
      "replace": "XXX-XX-XXXX"
    }
  ]
}

 

Any suggest?

avatar
New Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

Hi..

 

So many thanks Silvan... i think that i found the "issue" is in the traslation to Spanish of the interface:

In English the interface inform about hiveserver2 on Hive and Hive on Tez

ING.PNG

 

But in Spanish not...

ESP.PNG

 

Any way i'll read the refered documentation... like i said is my first CDP... i intalled so many CDH in the pass... but is my first time with CDP.

 

Regards.

avatar
Explorer

Hello again Silvan

 

Like you said the problem was the engine, after install Hive on Tez, the issue persist, finally i was changed the engine from MR to Spark and HIVESERVER2 of Hive start propertly

 

So many thanks for your help!

avatar
Rising Star

@silvan 

 

I did a fresh install of Tez followed by HMS followed by Hive on Tez.

 

But Hive on TEZ not coming up and throws below error, Kindly suggest?

 

- Is there any configuration property needs to updated in Tez or HMS or at some place?

 

Error:

 

org.apache.hive.service.ServiceException: org.apache.hive.service.ServiceException: Unable to setup tez session pool
at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:690) ~[hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
Caused by: org.apache.hive.service.ServiceException: Unable to setup tez session pool

 

 

 

2020-04-16 15:24:13,671 ERROR org.apache.hive.service.server.HiveServer2: [main]: Error starting Tez sessions:
org.apache.hive.service.ServiceException: Unable to setup tez session pool
at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:793) ~[hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763) ~[hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138) [hive-service-3.1.2000.7.0.3.0-79.jar:3.1.2000.7.0.3.0-79]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.0.3.0-79.jar:?]
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1587049942856_0006 failed 2 times due to AM Container for appattempt_1587049942856_0006_000002 exited with exitCode: 1

 

- VM

 

 

avatar
Moderator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login