hive upgrade tool failed at hdp 3 upgrade

[#1]

I'm upgrading from hdp 2.6 to 3 The ambari upgrade from 2.6.2 to 2.7 went fine, I'm following the documentation, I'm at the step 4 on the page https://docs.hortonworks.com.s3.amazonaws.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-upgrade/content/... I'm getting:

$JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar:/usr/hdp/2.6.0.3-8/hive2/lib/*:/usr/hdp/2.6.0.3-8/hadoop/*:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar: org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute                 SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2018-07-26T13:34:30,051 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Starting with execute=true, location=.
2018-07-26T13:34:30,056 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Using Hive Version: 2.1.0.2.6.0.3-8 build: 2.1.0.2.6.0.3-8 from 21903e6a60f3f4f3de63731c0c7cbec2cb2e265e by jenkins source checksum 7af682f1761a67fa9e6f01d987cd6003
2018-07-26T13:34:30,094 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file null
2018-07-26T13:34:31,415 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-07-26T13:34:31,523 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool
2018-07-26T13:34:31,567 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) [hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
        at java.lang.Class.getConstructor0(Class.java:3082) ~[?:1.8.0_111]
        at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[?:1.8.0_111]
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        ... 6 more
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)


        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148)
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650)
        ... 6 more

I kinited as hive, running this as root. After some googling I found that having more than 1 version of org.apache.hadoop.hive.metastore.HiveMetaStoreClient might be an issue, so I did this:

# grep -r 'org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>' /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar  /usr/hdp/2.6.0.3-8/hive2/* /usr/hdp/2.6.0.3-8/hadoop/* /usr/hdp/2.6.0.3-8/hadoop-mapreduce/* /usr/hdp/2.6.0.3-8/hadoop-hdfs/* /usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/* /tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar /usr/hdp/2.6.0.3-8/hive/conf/*
[root@lsp7148c tmp]# grep -r 'org.apache.hadoop.hive.conf.HiveConf' /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar  /usr/hdp/2.6.0.3-8/hive2/* /usr/hdp/2.6.0.3-8/hadoop/* /usr/hdp/2.6.0.3-8/hadoop-mapreduce/* /usr/hdp/2.6.0.3-8/hadoop-hdfs/* /usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/* /tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar /usr/hdp/2.6.0.3-8/hive/conf/*
Binary file /usr/hdp/2.6.0.3-8/hive2/jdbc/hive-jdbc-2.1.0.2.6.0.3-8-standalone.jar matches
Binary file /usr/hdp/2.6.0.3-8/hive2/lib/hive-exec-2.1.0.2.6.0.3-8.jar matches
Binary file /usr/hdp/2.6.0.3-8/hive2/lib/hive-common-2.1.0.2.6.0.3-8.jar matches

I tried remove 2 out of 3 of these files but that didn't change anything.

Any idea on how to continue would be appreciated.

thanks in advance!

[#2]

Hi, I just had myself quite a few issues with this preupgrade tool, mainly from the fact that within the documentation there was the wrong classpath and I had to correct myself all of the wrong occurrences.

The preupgradetool needs the conf.server folder passed (/usr/hdp/current/hive-client/conf/conf.server), other than the other jars needed for execution. It looks like you haven't passed that argument in your classpath. Could you check if passing all of the correct folders/files in your execution command works?

My complete corrected classpath was the following:

$JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/current/hive-server2-hive2/lib/derby-10.10.2.0.jar:/usr/hdp/current/hive-server2-hive2/lib/*:/usr/hdp/current/hadoop/*:/usr/hdp/current/hadoop/lib/*:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/hadoop-mapreduce-client/lib/*:/usr/hdp/current/hadoop-hdfs/*:/usr/hdp/current/hadoop-hdfs/lib/*:/usr/hdp/current/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:/usr/hdp/current/hive-client/conf/conf.server:/usr/hdp/current/hive-metastore/lib/hive-metastore.jar:/usr/hdp/current/hive-metastore/lib/libthrift-0.9.3.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hive-client/lib/hive-common.jar:/usr/hdp/current/hive-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/* org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute

[#3]

Hi @Federico D'Ambrosio, Thanks for that. I actually believe that I did include the conf.server when I ran this but somehow didn't paste the right thing when creating this post. Sorry for the confusion. I also had issues with the libs but verified that all the ones I'm using exist on my side. Now, I also added what I saw in your command that I hadn't included earlier, but I'm still getting the same error

[root@host ~]# export JAVA_HOME=/opt/app/jdk1.8.0_111
[root@host ~]# $JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar:/usr/hdp/2.6.0.3-8/hive2/lib/*:/usr/hdp/2.6.0.3-8/hadoop/*:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:/usr/hdp/2.6.0.3-8/hive/conf/conf.server/*:/usr/hdp/2.6.0.3-8/hive/lib/hive-metastore.jar:/usr/hdp/2.6.0.3-8/hive/lib/libthrift-0.9.3.jar:/usr/hdp/2.6.0.3-8/hadoop/hadoop-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/hive-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/commons-cli-1.2.jar:/usr/hdp/2.6.0.3-8/hadoop* org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2018-07-31T12:33:48,044 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Starting with execute=true, location=.
2018-07-31T12:33:48,048 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Using Hive Version: 2.1.0.2.6.0.3-8 build: 2.1.0.2.6.0.3-8 from 21903e6a60f3f4f3de63731c0c7cbec2cb2e265e by jenkins source checksum 7af682f1761a67fa9e6f01d987cd6003
2018-07-31T12:33:48,074 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file null
2018-07-31T12:33:48,630 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-07-31T12:33:48,654 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool
2018-07-31T12:33:48,670 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) [hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
        at java.lang.Class.getConstructor0(Class.java:3082) ~[?:1.8.0_111]
        at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[?:1.8.0_111]
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        ... 6 more
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148)
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650)
        ... 6 more

If you have any additional tip please let me know

[#4]

Hi @yves dormont, I believe that if you write in your classpath:

/usr/hdp/2.6.0.3-8/hive/conf/conf.server/*

with the wildcard '\*' you are actually including all of the files in the folder which have the extension *.jar. (Source)

So if you use

/usr/hdp/2.6.0.3-8/hive/conf/conf.server

you would be actually including all of the config files (properties and xml, not really sure which are used by the preupgrade tool, presumably hivemetastore-site.xml) within the conf.server folder, where all the information needed for the instantiation of the HiveMetaStoreClient.

Could you please change that part of your classpath and see if it changes anything?

[#5]

I'm facing the exact same issue as you but nothing is working. Tried many things today. If I find a resolution I will post it here.

[#6]

Hi @Federico D'Ambrosio,

I just tried that too, but still the same problem:

[root@lsp7148c ~]# export JAVA_HOME=/opt/app/jdk1.8.0_111
[root@lsp7148c ~]# kinit -kt /opt/app/hadoop-security/keytabs/hive.service.keytab hive/lsp7148c.sebank.se
[root@lsp7148c ~]# $JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar:/usr/hdp/2.6.0.3-8/hive2/lib/*:/usr/hdp/2.6.0.3-8/hadoop/*:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:/usr/hdp/2.6.0.3-8/hive/conf/conf.server:/usr/hdp/2.6.0.3-8/hive/lib/hive-metastore.jar:/usr/hdp/2.6.0.3-8/hive/lib/libthrift-0.9.3.jar:/usr/hdp/2.6.0.3-8/hadoop/hadoop-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/hive-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/commons-cli-1.2.jar:/usr/hdp/2.6.0.3-8/hadoop* org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2018-08-01T09:20:08,278 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Starting with execute=true, location=.
2018-08-01T09:20:08,282 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Using Hive Version: 2.1.0.2.6.0.3-8 build: 2.1.0.2.6.0.3-8 from 21903e6a60f3f4f3de63731c0c7cbec2cb2e265e by jenkins source checksum 7af682f1761a67fa9e6f01d987cd6003
2018-08-01T09:20:08,309 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file file:/etc/hive/2.6.0.3-8/0/conf.server/hive-site.xml
2018-08-01T09:20:08,748 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.enforce.sorting does not exist
2018-08-01T09:20:08,748 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.enforce.bucketing does not exist
2018-08-01T09:20:08,869 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-08-01T09:20:08,893 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool
2018-08-01T09:20:08,909 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) [hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
        at java.lang.Class.getConstructor0(Class.java:3082) ~[?:1.8.0_111]
        at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[?:1.8.0_111]
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
        ... 6 more
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204)
        at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148)
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650)
        ... 6 more

On my side I'm looking more into the metastore now.

Will let you know if I get anywhere...

Thanks again!

[#7]

I have the same issue. it seems to be ralated to HIVE-15081 (https://issues.apache.org/jira/browse/HIVE-15081) which was fixed in HDP 2.6.2. I am using HDP 2.6.1. Is there a way to apply this patch?

[#8 · Accepted solution]

I have the same issue. it seems to be ralated to HIVE-15081 (https://issues.apache.org/jira/browse/HIVE-15081) which was fixed in HDP 2.6.2. I am using HDP 2.6.1. Is there a way to apply this patch?

[#9]

I have made some test with the new Hive-libraries including HIVE-15081 and was able to connect to the metastore. To me it seems that everyone using Hive Version: 2.1.0.2.6.2.0-205 and above will be able to run the Hive-upgrade tool. for all users below this version it will fail.

[#10]

@Alexandre Lopes, can you confirm that you also started from a version below HDP 2.6.2?

@Federico D'Ambrosio, can you confirm that you started from HDP 2.6.2 or above?

Thanks a lot for that SG, good find!