Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

Highlighted

hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

New Contributor

Urgently looking for help. We tried upgrading hortonworks HDP from 2.6.5 to 3.0.0 and it's tuck since last 4-5 days. I am new to hortonworks working since last 7-8 months..

Can someone recommend how to fix it up and where to look for.PFA for reference..

..screen-shot-2018-08-13-at-121206-pm.png

10 REPLIES 10

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

Rising Star

can you please give us more information like exactly at what task upgrade is stuck Click in upgrade in progress link and provide screenshot of the task it have executed and task it is stuck at so we can troubleshoot this further accordingly.

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

New Contributor

PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working.

it's just stuck. Is there something can be tried out from the back end? How and where to check logs can u guide me

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

Hi @pankaj singh,

You can click on Upgrade on process, SEe which task its stuck and see its error/output logs correspondingly .

If everything is smooth it might be pending at manual steps required stage. in that case you can perform the manual steps and proceed with the installation.

Hope Upgrade goes smooth for you.

if you face some issues.please feel free to post the error logs in

I am code format.

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

New Contributor

PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working.

it's just stuck. Is there something can be tried out from the back end? How and where to check logs. can you guide me?

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

New Contributor

I restarted hive metastore by running tool, post command line start (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/valida...

but ambari admin ui is still stuck.. is their a way we can restart upgrade or rollback or do something.. thr is no button to restart upgrade even..screen-shot-2018-08-14-at-115110-am.png

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

New Contributor

screen-shot-2018-08-14-at-101042-am.pngscreen-shot-2018-08-14-at-101055-am.pngscreen-shot-2018-08-14-at-101107-am.png

PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working.

it's just stuck. Is there something can be tried out from the back end?

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

New Contributor

screen-shot-2018-08-14-at-101741-am.png

uploading few more pictures for guidance how to troubleshoot..

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

New Contributor

on further debugging i could see this issue in hive metastore log:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre}

018-08-14T10:26:55,365 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(8986)) - Metastore Thrift Server threw an exception...

org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]

at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?]

at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?]

Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt

at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]

at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) ~[?:?]

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]

... 11 more

2018-08-14T10:26:55,379 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$main$0(8951)) - Shutting down hive metastore.

2018-08-14T10:26:55,393 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping hivemetastore metrics system...

2018-08-14T10:26:55,393 INFO[timeline]: impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.

2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - hivemetastore metrics system stopped.

2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - hivemetastore metrics system shutdown complete.

2018-08-14T10:26:55,395 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$startupShutdownMessage$1(9389)) - SHUTDOWN_MSG:

Re: hortonworks HDP 3.0 update stuck in progress for more than 4-5 days..

Hi @Pankaj Singh,

It looks like your Hive metastore upgrade is failing and another upgrade is going smoothly.

can you try to pause the upgrade and resume once see if there is any luck on same.

Additionally, if you want to proceed with installation of hive metastore you can try disabling the hive metastore verification but

By configuring the property hive.metastore.schema.verification to false and metastore to implicitly write the schema version if it's not matching. To enable the strict schema verification, you need to set this property to true in hive-site.xml
reference : https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

Please note: you need to pause the upgrade while changing the configurations through ambari.

Additionally i would like to tell you go through each subtasks of hive metastore installation and see why those one's failed. if possible try to dig root cause of same.

Hope this helps.