- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
hive upgrade tool failed at hdp 3 upgrade
- Labels:
-
Apache Hive
Created 07-26-2018 02:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm upgrading from hdp 2.6 to 3 The ambari upgrade from 2.6.2 to 2.7 went fine, I'm following the documentation, I'm at the step 4 on the page https://docs.hortonworks.com.s3.amazonaws.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-upgrade/content/... I'm getting:
$JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar:/usr/hdp/2.6.0.3-8/hive2/lib/*:/usr/hdp/2.6.0.3-8/hadoop/*:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar: org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2018-07-26T13:34:30,051 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Starting with execute=true, location=. 2018-07-26T13:34:30,056 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Using Hive Version: 2.1.0.2.6.0.3-8 build: 2.1.0.2.6.0.3-8 from 21903e6a60f3f4f3de63731c0c7cbec2cb2e265e by jenkins source checksum 7af682f1761a67fa9e6f01d987cd6003 2018-07-26T13:34:30,094 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file null 2018-07-26T13:34:31,415 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-07-26T13:34:31,523 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool 2018-07-26T13:34:31,567 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) [hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean) at java.lang.Class.getConstructor0(Class.java:3082) ~[?:1.8.0_111] at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[?:1.8.0_111] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 6 more Exception in thread "main" java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean) at java.lang.Class.getConstructor0(Class.java:3082) at java.lang.Class.getDeclaredConstructor(Class.java:2178) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ... 6 more
I kinited as hive, running this as root. After some googling I found that having more than 1 version of org.apache.hadoop.hive.metastore.HiveMetaStoreClient might be an issue, so I did this:
# grep -r 'org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>' /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar /usr/hdp/2.6.0.3-8/hive2/* /usr/hdp/2.6.0.3-8/hadoop/* /usr/hdp/2.6.0.3-8/hadoop-mapreduce/* /usr/hdp/2.6.0.3-8/hadoop-hdfs/* /usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/* /tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar /usr/hdp/2.6.0.3-8/hive/conf/* [root@lsp7148c tmp]# grep -r 'org.apache.hadoop.hive.conf.HiveConf' /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar /usr/hdp/2.6.0.3-8/hive2/* /usr/hdp/2.6.0.3-8/hadoop/* /usr/hdp/2.6.0.3-8/hadoop-mapreduce/* /usr/hdp/2.6.0.3-8/hadoop-hdfs/* /usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/* /tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar /usr/hdp/2.6.0.3-8/hive/conf/* Binary file /usr/hdp/2.6.0.3-8/hive2/jdbc/hive-jdbc-2.1.0.2.6.0.3-8-standalone.jar matches Binary file /usr/hdp/2.6.0.3-8/hive2/lib/hive-exec-2.1.0.2.6.0.3-8.jar matches Binary file /usr/hdp/2.6.0.3-8/hive2/lib/hive-common-2.1.0.2.6.0.3-8.jar matches
I tried remove 2 out of 3 of these files but that didn't change anything.
Any idea on how to continue would be appreciated.
thanks in advance!
Created 08-02-2018 02:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have the same issue. it seems to be ralated to HIVE-15081 (https://issues.apache.org/jira/browse/HIVE-15081) which was fixed in HDP 2.6.2. I am using HDP 2.6.1. Is there a way to apply this patch?
Created 07-26-2018 03:10 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, I just had myself quite a few issues with this preupgrade tool, mainly from the fact that within the documentation there was the wrong classpath and I had to correct myself all of the wrong occurrences.
The preupgradetool needs the conf.server folder passed (/usr/hdp/current/hive-client/conf/conf.server), other than the other jars needed for execution. It looks like you haven't passed that argument in your classpath. Could you check if passing all of the correct folders/files in your execution command works?
My complete corrected classpath was the following:
$JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/current/hive-server2-hive2/lib/derby-10.10.2.0.jar:/usr/hdp/current/hive-server2-hive2/lib/*:/usr/hdp/current/hadoop/*:/usr/hdp/current/hadoop/lib/*:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/hadoop-mapreduce-client/lib/*:/usr/hdp/current/hadoop-hdfs/*:/usr/hdp/current/hadoop-hdfs/lib/*:/usr/hdp/current/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:/usr/hdp/current/hive-client/conf/conf.server:/usr/hdp/current/hive-metastore/lib/hive-metastore.jar:/usr/hdp/current/hive-metastore/lib/libthrift-0.9.3.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hive-client/lib/hive-common.jar:/usr/hdp/current/hive-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/* org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute
Created 07-31-2018 10:35 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Federico D'Ambrosio, Thanks for that. I actually believe that I did include the conf.server when I ran this but somehow didn't paste the right thing when creating this post. Sorry for the confusion. I also had issues with the libs but verified that all the ones I'm using exist on my side. Now, I also added what I saw in your command that I hadn't included earlier, but I'm still getting the same error
[root@host ~]# export JAVA_HOME=/opt/app/jdk1.8.0_111 [root@host ~]# $JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar:/usr/hdp/2.6.0.3-8/hive2/lib/*:/usr/hdp/2.6.0.3-8/hadoop/*:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:/usr/hdp/2.6.0.3-8/hive/conf/conf.server/*:/usr/hdp/2.6.0.3-8/hive/lib/hive-metastore.jar:/usr/hdp/2.6.0.3-8/hive/lib/libthrift-0.9.3.jar:/usr/hdp/2.6.0.3-8/hadoop/hadoop-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/hive-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/commons-cli-1.2.jar:/usr/hdp/2.6.0.3-8/hadoop* org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2018-07-31T12:33:48,044 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Starting with execute=true, location=. 2018-07-31T12:33:48,048 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Using Hive Version: 2.1.0.2.6.0.3-8 build: 2.1.0.2.6.0.3-8 from 21903e6a60f3f4f3de63731c0c7cbec2cb2e265e by jenkins source checksum 7af682f1761a67fa9e6f01d987cd6003 2018-07-31T12:33:48,074 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file null 2018-07-31T12:33:48,630 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-07-31T12:33:48,654 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool 2018-07-31T12:33:48,670 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) [hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean) at java.lang.Class.getConstructor0(Class.java:3082) ~[?:1.8.0_111] at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[?:1.8.0_111] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 6 more Exception in thread "main" java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean) at java.lang.Class.getConstructor0(Class.java:3082) at java.lang.Class.getDeclaredConstructor(Class.java:2178) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ... 6 more
If you have any additional tip please let me know
Created 07-31-2018 03:15 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @yves dormont, I believe that if you write in your classpath:
/usr/hdp/2.6.0.3-8/hive/conf/conf.server/*
with the wildcard '\*' you are actually including all of the files in the folder which have the extension *.jar. (Source)
So if you use
/usr/hdp/2.6.0.3-8/hive/conf/conf.server
you would be actually including all of the config files (properties and xml, not really sure which are used by the preupgrade tool, presumably hivemetastore-site.xml) within the conf.server folder, where all the information needed for the instantiation of the HiveMetaStoreClient.
Could you please change that part of your classpath and see if it changes anything?
Created 07-31-2018 08:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm facing the exact same issue as you but nothing is working. Tried many things today. If I find a resolution I will post it here.
Created 08-01-2018 07:27 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I just tried that too, but still the same problem:
[root@lsp7148c ~]# export JAVA_HOME=/opt/app/jdk1.8.0_111 [root@lsp7148c ~]# kinit -kt /opt/app/hadoop-security/keytabs/hive.service.keytab hive/lsp7148c.sebank.se [root@lsp7148c ~]# $JAVA_HOME/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -cp /usr/hdp/2.6.0.3-8/hive2/lib/derby-10.10.2.0.jar:/usr/hdp/2.6.0.3-8/hive2/lib/*:/usr/hdp/2.6.0.3-8/hadoop/*:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:/usr/hdp/2.6.0.3-8/hive/conf/conf.server:/usr/hdp/2.6.0.3-8/hive/lib/hive-metastore.jar:/usr/hdp/2.6.0.3-8/hive/lib/libthrift-0.9.3.jar:/usr/hdp/2.6.0.3-8/hadoop/hadoop-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/hive-common.jar:/usr/hdp/2.6.0.3-8/hive/lib/commons-cli-1.2.jar:/usr/hdp/2.6.0.3-8/hadoop* org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool -execute SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2018-08-01T09:20:08,278 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Starting with execute=true, location=. 2018-08-01T09:20:08,282 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Using Hive Version: 2.1.0.2.6.0.3-8 build: 2.1.0.2.6.0.3-8 from 21903e6a60f3f4f3de63731c0c7cbec2cb2e265e by jenkins source checksum 7af682f1761a67fa9e6f01d987cd6003 2018-08-01T09:20:08,309 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file file:/etc/hive/2.6.0.3-8/0/conf.server/hive-site.xml 2018-08-01T09:20:08,748 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.enforce.sorting does not exist 2018-08-01T09:20:08,748 WARN [main] org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.enforce.bucketing does not exist 2018-08-01T09:20:08,869 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-08-01T09:20:08,893 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool 2018-08-01T09:20:08,909 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) ~[hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) [hive-pre-upgrade-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean) at java.lang.Class.getConstructor0(Class.java:3082) ~[?:1.8.0_111] at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[?:1.8.0_111] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 6 more Exception in thread "main" java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getHMS(PreUpgradeTool.java:190) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:204) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:148) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean) at java.lang.Class.getConstructor0(Class.java:3082) at java.lang.Class.getDeclaredConstructor(Class.java:2178) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1650) ... 6 more
On my side I'm looking more into the metastore now.
Will let you know if I get anywhere...
Thanks again!
Created 08-02-2018 02:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have the same issue. it seems to be ralated to HIVE-15081 (https://issues.apache.org/jira/browse/HIVE-15081) which was fixed in HDP 2.6.2. I am using HDP 2.6.1. Is there a way to apply this patch?
Created 08-02-2018 02:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have the same issue. it seems to be ralated to HIVE-15081 (https://issues.apache.org/jira/browse/HIVE-15081) which was fixed in HDP 2.6.2. I am using HDP 2.6.1. Is there a way to apply this patch?
Created 08-02-2018 04:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have made some test with the new Hive-libraries including HIVE-15081 and was able to connect to the metastore. To me it seems that everyone using Hive Version: 2.1.0.2.6.2.0-205 and above will be able to run the Hive-upgrade tool. for all users below this version it will fail.
Created 08-03-2018 07:52 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Alexandre Lopes, can you confirm that you also started from a version below HDP 2.6.2?
@Federico D'Ambrosio, can you confirm that you started from HDP 2.6.2 or above?
Thanks a lot for that SG, good find!